Probe Blog

I worked for 8 years at a successful startup called Base CRM. It was acquired by Zendesk 2 years ago. During this time I saw the company take different approaches to many problems startups have, but one remained unchanged: unparalleled, obsessive focus on shipping new functionality. Lately, I’ve been thinking a bit more about a company as a whole because I co-started a gig called Probe, so here’s some things I think should have been different at Base.

Disclaimer I have to make: although the things I write here may sound negative it wasn’t all that bad. We still were a really successful company, did many things right, we had a great team, a product used by thousands of sales people. I’m just focusing on where we could’ve done better (myself included).

Beginnings at Base

Base’s mentality was always oriented towards product and shipping features. Even our motto said it: “It’s null till you ship it” – what better way to say that all your R&D efforts should go to releasing new stuff to production? This mentality was brought by our CEO, the person behind the idea of building a new CRM – Uzi Shmilovici. He was the most experienced person in our very young team (I joined when I was 24). His experience was definitely something that helped to instill the value of shipping new product features in our minds.

Quickly the whole organization adopted and started living by this motto. To be perfectly fair: I think there was a lot of merit in it. Being a hardcore developer at that time I still believed that what we sell is the shipped product, not the technical solutions behind it, nor the automated tests, code coverage etc. At least this is how I projected what this motto meant, and I’m quite convinced that others looked at it this way as well. I wasn’t 100% right.

Maturity and Growth at Base

Over time the company grew. More customers, bigger funding rounds, higher ambitions. Our R&D suddenly became 40, then 100 people. All of them with the “it’s null till you ship it” mindset.

Things got more and more complicated as well. CRM is a broad topic and can mean many things. It tries to embrace the basics of many different sales processes that companies have, together with a broad set of other features like email or calendar integration. It took an enormous amount of work to ship new features that fitted nicely into the existing ecosystem. Suddenly we had 1-year projects on our roadmap (nicely hidden under 3-month projects which didn’t end on time).

Additionally we landed a few huge deals, the famous customers you can’t afford to lose. It meant custom work embedded into already complicated functionality, and complexity grew fast. Big customers certainly sounded good on paper, but they came with a cost (complexity and a very hands-on relationship) that we weren’t really prepared for, and that we didn’t manage to put boundaries on fast enough.

… the difficulties started piling up fast, and before we knew it we had to deal with a lot.

We also added a lot of complication on the backend of the product. Moving to microservices meant getting rid of some problems of our legacy architecture, but it introduced us to the world of (arguably) harder ones, like keeping data consistent across services. Some we managed to cover up with money by just buying more processing power. Others we desperately needed to fix, especially those basic building blocks of our infrastructure which we knew didn’t work. One example would be a communication channel between our microservices without the at-least-once guarantee – some messages would never reach the destination endpoints, which caused problems with data-consistency.

Of course, all of this was the effect of our success. We had new customers coming in, the metrics looked good, and we quickly moved up in the market. It was all because we were solving real user problems.

But the difficulties started piling up fast, and before we knew it we had to deal with a lot.

The Difficulties

On the product side the complexity was quite overwhelming. We had a huge system with many moving parts which interacted with each other, and these interactions were sometimes hard to understand. With the added factor of custom solutions for Big Customers™, more and more flaws in the interactions started showing up.

No Decision Paper-trail

There was little documentation about product decisions. If you’d been around for a long time (like myself), then you had a fairly good understanding of how the product worked. Anyone else had to ask around, but even the original authors of ideas didn’t have all the answers, because the situation was often complicated and involved multiple people on the team.

Insufficient Iteration

We didn’t revisit features often, either. The product specs were well thought through, sometimes crafted for months, but the truth is that almost nothing you build will be perfect the first time. (and if it is, there’s almost certainly a generous amount of luck involved). It’s normal, and there’s nothing to be ashamed of – no matter how hard we plan and research, we build with imperfect information, and we do the best we can. Sadly, we had a lot more confidence than was wise in how well our features were initially designed – some might say hubris – and we didn’t revisit or iterate on existing features.

Technology Built With More Enthusiasm Than Experience

The backend suffered as well. We moved into a world of microservices very early on, and that’s not to say that microservices aren’t a power tool (I’m still a fan of them), but you have to have expertise on how to do it well. Back then, we were inexperienced and just thinking about the outcomes and upsides, so we still pushed very hard to separate our architecture into microservices, and in many cases we did it badly. We pushed it too fast, too early, and without really learning along the way – so, in effect, we just repeated the same mistakes multiple times, rather than iteratively improving our architecture and processes as we moved forward.

We moved too slow with technical documentation as well. With a system as complex as this, it’s crucial to document your tech stack, code, and APIs Yet with the pace of shipping new things, and a pressure to deliver working solutions, we neglected documenting how things work and focused on just getting things out the door. In hindsight, we also should have standardized many more things as we moved forwards. Without coding or operating standards, the cost to introduce new solutions grew.

Lessons Learned

So far, I’ve just talked about the challenges we faced, and the pain we felt. There’s plenty you can take away just from that, but I’d like to make sure I’m also sharing some of the actual lessons I learned in the process, so that you don’t make the same mistakes.

Culture Shapes Everything

One thing it’s worth noting as a background to all of these lessons is that it’s important to recognise the huge and subtle impact of organisational culture. While some of our challenges were a result of us being a young, enthusiastic startup team, many could be traced back to the culture of “it’s null till you ship it”. That wasn’t just a saying, it was an idea that became embedded in how we thought about bugs, customers, growth and technology.

I’m certainly not saying it was a bad saying, or a toxic culture – it was instrumental in our growth and success! But I am saying that we didn’t pay enough attention to how it was shaping our organisation and our product practices. Be ready to adapt your organisation when your situation changes.

Iterate More

First and foremost: no matter how much time you put into your product design, iterate again. Your best, educated guess is not going to be enough, even if you spend 6 months perfecting it. The urge to have new functionality is huge – believe me, I know! – but make sure you spend time taking care of existing solutions. There are many improvements waiting, just look at your existing customers and what they tell you(via your support system or sales team), trust your team, and give them time to improve on solutions they previously crafted which could be delivering more value (or less friction).

Recognise – and Control – your Technical Debt

While you’re at it, gather quantitative data from your backend about errors & bugs. The complexity and friction you ignore in your software is inevitably going to haunt you when you want to try and build something new, so pay attention to your technical debt and take steps to control it. Bear in mind that some of your technical debt will be in the code, and some of it will be in the documentation or processes (or lack thereof). You need to address both, one way or another.

Beware of Money With Strings Attached

Hopefully you’ll be successful and will win big customers. However, beware of huge deals, because although the money is great, they often come with strings attached. Most likely you’re going to “pollute” your product with custom solutions which don’t really fit in, and which possibly shouldn’t even be a part of the product at all. It’s also going to force you to redesign your team structure somehow – perhaps create a Professional Services team – or else your roadmap is going to be a constant battlefield between what you want to ship and what your Big Customer™ wants. We thought these kinds of big customers would be a segway into a world of Enterprise sales, but it turned out they weren’t a guaranteed path. In fact, we lost our very first one without a clear benefit – no product improvements, and no references or stepping-stones for future Enterprise customers.

Speak up Sooner

If you are an individual contributor or a team leader of a small group of people, make sure you voice your concerns clearly. Founders are (hopefully) focused on the most important metrics and setting up strategy (and for good reasons!), but it is your responsibility to make sure they know about the things you’re responsible for. Product experience getting worse? More and more errors piling up on the backend side? Be open about it and ask for time to fix things, otherwise it’s you who’s going to have to deal with these issues all the time.

by Mike, co-founder of Probe

Originally published on www.mindtheproduct.com, 25th August 2020 Illustration by Ivan Haidutski from Icons8