Interesting! I especially enjoyed the insight into the evolution of the company. Though with this insight I have slightly different conclusions than the author:
1. A bias to ship and a bias to ship new things are not one and the same. A lot of the problems, such as a failure to iterate on existing products/feature, sound very much a product of the later, not the former. If anything, the issue "Insufficient Iteration" is probably not correctable without a bias to shipping.
2. A bias to ship and a bias to ship things that impact your customers are not one and the same. There is a note about an early shift to micro-services. I can't speak to this organization, but, generally speaking, spending time on internal engineering work to the detriment obvious missing features is a common issue with early stage companies.
3. A bias to ship and a bias to ship to the right customers. Specifically in regard to the high value, high demand customers.
4. A bias to ship and... idk what to call this? "The product specs were well thought through, sometimes crafted for months." TBH, the problem with this one feels like a lack of a bias to ship.
As described, I think the real culprit was a lack of or poor prioritization. FWIW, I suspect the author and I may actually be in violent agreement as I did find my self nodding with most of his lessons learned. Though I'd be careful about letting too much hindsight bleed in (eg - do situations that'd be improved by more decision documentation justify the effort of documenting all product decisions, especially in the early phase when the product is rapidly evolving)?
I think the problem is that the leaders in the company (technical or otherwise) didn't recognize that software systems are like children: They have phases.
The ship it constantly is probably ok when you're small, but at some point you need to become more mature than that, both as an organization, and as an approach to the software itself.
I work at a company that grew from a few hundred to a few thousand people in a few years' time, and there was an identical mindset and identical problems over that growth period. I started during that growth period, and 10 years later, we're still digging ourselves out of those mistakes.
One reason for this was how heavily we prioritized first mover advantage. We pushed extremely hard to get into new markets, and while strategically this worked out extremely well for us, it meant that later development was hamstrung by earlier decisions. Any iteration on old features almost always meant rebuilding it entirely, and we've only just started to get to a place where previous efforts to do this are now paying off in increased velocity.
I think there's an argument that this could be good, since you're only improving features when there's some other value you can ship. However, I think it's short-sighted, because it means prioritizing small iterations has an unnecessarily large cost, simply because that cost wasn't paid down slowly over time. While I do think prioritizing new markets early on was a good call, that mindset bit us in the ass in a big way when we tried to ship a very flashy new feature later in our growth cycle that crashed and burned because of tech debt, and it took us over a year to fix everything to release it. If we had taken a mindset of building for the long term five years ago, which was already years after we'd gotten to a comfortable place in revenue and marketshare, we would probably have paid off most of the major tech debt by now.
I honestly don't know how a company can get past the addiction to shipping, at least not during its growth period. It can be strategically necessary, and I imagine in only a two-year period at the early stages of a company, like what's in this article, it's impossible to avoid. But leaders must have in the back of their heads the idea that they will need to start tipping the scales toward standardization, building to last, and building infrastructure as well as features sooner than they'd probably like to avoid larger problems in the future. And I suspect the only way leaders will really take that seriously is if they've gone through it before.
Apart from the one catastrophic failure you mention it sounds like, from a business perspective this worked? You were able to extend your tech debt long enough to start generating money and are now in a position to pay it back. From an engineering perspective I agree it's a grind to pay all this back and lose so much velocity. Are their business strategies that favour slower more careful engineering practises? mission-critical systems come to mind but maybe also nth-movers could be put in there? Where you know what you're building and competing with but want to make the best version of X.
I worked at a small company that was always chasing the next potential sale. Potential customer mentions feature A and all of a sudden all development shifts towards developing that feature. The vast majority of the time the potential customer never became a customer. In the meantime competition focused on their own plan of features and our product wound up with features no one ever wanted and was missing critical functionality everybody demanded.
The core issue, from what I've seen, is one of incentives. Many leaders at many companies are compensated on growth and short term factors. It's hard to find a leader that is willing to take a 5 year look, or even find a leader that is planning on being around 5+ years.
The leadership at my publicly traded company is a revolving door of short term hacks for promos followed by a quick pivot to the competition or another company. It often feels like the C-Suite at most companies thinks at most, 2 years out.
I just can't see many leaders thinking 5 years out.
My previous employer prioritized shipping, won the market while having both more features and more technical debt than competitors, bought those competitors out as they started failing, and then migrated their clients over and threw their beautiful (but less featured) systems right into the garbage. To this day, I sometimes think back to how a particular competitor had the right schema for solving current problems, but they didn't even survive to the current day, which takes precedence.
> One reason for this was how heavily we prioritized first mover advantage. We pushed extremely hard to get into new markets, and while strategically this worked out extremely well for us, it meant that later development was hamstrung by earlier decisions.
I've seen so many developers over the years who view having to rewrite that module because it can't scale well enough anymore as a failure, but I call it a success. Software is meant to be tumultuous, things SHOULD be getting replaced.
In another post in this thread I mentioned that software systems are like children in that they have phases. Assuming that your company and software team is dedicated to fixing those issues (slowly, beside new features), then I think you're probably in a healthy spot despite the pain.
An observation is that business interests must ALWAYS trump technical interests when those business interests are critical. No business = no technical. But at some point when the company becomes stable that balance has to start shifting back towards the technical or the company itself becomes ridiculously inefficient.
This is kind of interesting but it sounds like the perpetual worry of the perfectionist developer. Of course you end up with tech tech, you develop new things when you could improve existing things, you deploy something that could be iterated but you don't.
The truth is that the world is a complex place and you don't always know whether you can keep your existing customers by improving what you already have or get new customers to decrease the risk of declining income. It is hard to measure what is acceptable tech debt and what is worth addressing.
In one sense, the proof is in the income. If you can help customers with an 80% OK product then you just need to live with the dodgy 20% bit.
I agree with the danger of big customers though! If you ever think something is a way to avoid hard decisions, pivots and annoying some of your customers, then it is too good to be true ;-)
I'm far from being a developer perfectionist, I wouldn't spend 10 years in this company if I was one, trust me (btw we fired a lot of them along the way).
The point of the article is not to convince anyone to slow down as much as possible and work on bugs/etc. It's just that at some point (3, 4 years in?) there comes a time that you just have to put more effort into the things I described, otherwise it gets complicated fast.
Obviously there is a tradeoff, my opinion is that this tradeoff wasn't correctly balanced (especially further down).
I worked at a large web development company. The one thing which continually hamstrung the company was we had several large customers who were paying us a good chunk of money to maintain their web properties. They sucked up so much of our resources to keep them happy that other smaller customers suffered as a result.
The amount of churn we had was always the smaller clients who were still paying good money but felt neglected and would be open to changing vendors.
We went out of our way to mind the big clients and thought if we lost these few big clients it would so dramatically adverse the companies bottom line, we couldn't afford to lose them. This resulted in a situation where they were able to hold our company hostage, make unreasonable demands on time and energy and hamstrung our ability to keep our more loyal clients (who only bothered us on occasion) happy.
It was a good lesson there is a danger when you have a small company servicing a much larger company that you get into a position where your fear of losing them is a detriment to your other, more faithful clients.
There's a lot of insight packed into this post, and really it transcends way beyond the mindset of constantly shipping new things. This particular line resonated with me:
Big customers certainly sounded good on paper, but they came with a cost (complexity and a very hands-on relationship) that we weren’t really prepared for, and that we didn’t manage to put boundaries on fast enough.
This is an issue I've dealt with more than once in my career. It seems to occur especially when an organization values the sale more than it values the business. It's virtually impossible to successfully apply boundaries after the customer has been trained that there are none. Irrespective of how carefully you do it or how much you try to reassure your customer, it always feels like you are taking value away. I've encountered this recently in an organization that values customer happiness, which it calculates as an absence of customer complaints, more than customer success. The squeaky wheels get all the grease at the expense of just about everything else.
And the software inevitably has become a maze of settings and IF-THEN-ELSE blocks and SWITCH statements, increasing the cognitive load required to make what should be simple changes. When new features are added, the impulse is to code defensively and build inevitable customization into everything up front. And further down the spiral of technical debt and increasing complexity we go...
I always wonder why nobody appears to try a hybrid model between product and project development when it comes to satisfying large customers which need very specific customizations that often don't fit well into a product development roadmap aiming at satisfying a more generic audience.
Like: don't just add weird features that make sense for just one customer, but are eventually delivered to all customers, but instead maintain a "core product" which provides a lot of generic, core functionality, but which by itself isn't actually usable and just serves as basic building blocks to construct the actual end products which are then built and packaged individually for each (big) customer. Each of these "end products" is a development project in itself, has its own team, its own codebase, but they all draw from a core set of base functionality developed and maintained by another dedicated team that is not directly in contact with any of the customers (or at least not permanently).
This - at least in theory - prevents custom functionality for specific customers from polluting the common core product that aims to please a general audience, while still allowing for the necessary freedom to satisfy big customers demanding custom functionality, but also offering big checks in return.
When I was running a product team, I often talked about "nuance" being the enemy. Every difference (or arrow in a flowchart) was a new piece of product debt to have to think about later.
This article is in that direction.
> … the difficulties started piling up fast, and before we knew it we had to deal with a lot.
I think this is a solid critical reflection that at least in enterprise relates well.
When it comes to your customers, I believe that you want to understand as much as you can, as early as you can, about what they are actually trying to do and about how they are actually faring with what you have sold them. I'd argue that the success of your initial customers is more important than the features for your next deal, and that feature investments should reflect that.
The problem, as the OP said, is in estimating the difference between the incremental feature requirements of your initial customers and what your strategic feature set will be. There may be no difference, and your strategy is to ship features in order satisfy each new customer as you get it. Perhaps you can design for that. Chances are though that your first customers are not wholly repsentative of your strategic, target market.
So, in the early stage, you have to make choices about allocating dev for tactical "now" or for strategic "later". In the early stage, if there is not enough "now", there is no "later". The question is when to begin budget for later. I'll argue that you are attempting to build a business, not a suspense movie, and that you budget a fixed amount for "later" from day one. Plan to succeed.
I agree with the recommendation of a strong Professional Services function as an buffer for dev, but will note that culturally you have to work hard to not silo. Your Support and ProServe teams become a primary channel of learning for dev. As a business, though, you may have to give away ProServ hours.
All in all my biggest takeaway here is that you attempt to "hire" customers strategically. As you journey together, will this customer want to go where you will want to go? What time is it when your "Big Customer" sits on your roadmap? ...
"Hire" customers strategically is easier said than done when there's salaries on the line -- accordingly look at how you compensate Sales and budget away from early stage big deals.
What an awful article. Expensive conclusions are drawn with no apparent consideration for their cost. The experience of a company which appears to have successfully balanced complicated tradeoffs in choosing features is discounted because there were bugs. And almost no examples or useful stories about any detail, or suggestions on how to do it better other than to go (much) slower and to spend (much) more or to not take rich customers money.
I suspect good arguments could be made for these things, but they need to acknowledge and justify their cost.
It's tough to suggest detailed solutions to a problem which is in general very vague. Btw I hope the article shows my appreciation for what the company achieved.
One small insight from Uzi (CEO + founder) when I discussed this with him (after publishing article): Base should've focused on much less features but with greater detail. It somehow confirms my guess that we didn't work on existing features as much as we should, but we "spread too thin".
Oh good, I was afraid this was going to be an intervention for me regarding my steady stream of almost daily deliveries from Amazon/HD/Walmart/Ebay etc.
digitallogic|5 years ago
1. A bias to ship and a bias to ship new things are not one and the same. A lot of the problems, such as a failure to iterate on existing products/feature, sound very much a product of the later, not the former. If anything, the issue "Insufficient Iteration" is probably not correctable without a bias to shipping.
2. A bias to ship and a bias to ship things that impact your customers are not one and the same. There is a note about an early shift to micro-services. I can't speak to this organization, but, generally speaking, spending time on internal engineering work to the detriment obvious missing features is a common issue with early stage companies.
3. A bias to ship and a bias to ship to the right customers. Specifically in regard to the high value, high demand customers.
4. A bias to ship and... idk what to call this? "The product specs were well thought through, sometimes crafted for months." TBH, the problem with this one feels like a lack of a bias to ship.
As described, I think the real culprit was a lack of or poor prioritization. FWIW, I suspect the author and I may actually be in violent agreement as I did find my self nodding with most of his lessons learned. Though I'd be careful about letting too much hindsight bleed in (eg - do situations that'd be improved by more decision documentation justify the effort of documenting all product decisions, especially in the early phase when the product is rapidly evolving)?
RandoHolmes|5 years ago
The ship it constantly is probably ok when you're small, but at some point you need to become more mature than that, both as an organization, and as an approach to the software itself.
montagg|5 years ago
One reason for this was how heavily we prioritized first mover advantage. We pushed extremely hard to get into new markets, and while strategically this worked out extremely well for us, it meant that later development was hamstrung by earlier decisions. Any iteration on old features almost always meant rebuilding it entirely, and we've only just started to get to a place where previous efforts to do this are now paying off in increased velocity.
I think there's an argument that this could be good, since you're only improving features when there's some other value you can ship. However, I think it's short-sighted, because it means prioritizing small iterations has an unnecessarily large cost, simply because that cost wasn't paid down slowly over time. While I do think prioritizing new markets early on was a good call, that mindset bit us in the ass in a big way when we tried to ship a very flashy new feature later in our growth cycle that crashed and burned because of tech debt, and it took us over a year to fix everything to release it. If we had taken a mindset of building for the long term five years ago, which was already years after we'd gotten to a comfortable place in revenue and marketshare, we would probably have paid off most of the major tech debt by now.
I honestly don't know how a company can get past the addiction to shipping, at least not during its growth period. It can be strategically necessary, and I imagine in only a two-year period at the early stages of a company, like what's in this article, it's impossible to avoid. But leaders must have in the back of their heads the idea that they will need to start tipping the scales toward standardization, building to last, and building infrastructure as well as features sooner than they'd probably like to avoid larger problems in the future. And I suspect the only way leaders will really take that seriously is if they've gone through it before.
EDIT: Clarified a few points.
jamil7|5 years ago
macjohnmcc|5 years ago
jorblumesea|5 years ago
The leadership at my publicly traded company is a revolving door of short term hacks for promos followed by a quick pivot to the competition or another company. It often feels like the C-Suite at most companies thinks at most, 2 years out.
I just can't see many leaders thinking 5 years out.
satyrnein|5 years ago
RandoHolmes|5 years ago
I've seen so many developers over the years who view having to rewrite that module because it can't scale well enough anymore as a failure, but I call it a success. Software is meant to be tumultuous, things SHOULD be getting replaced.
In another post in this thread I mentioned that software systems are like children in that they have phases. Assuming that your company and software team is dedicated to fixing those issues (slowly, beside new features), then I think you're probably in a healthy spot despite the pain.
An observation is that business interests must ALWAYS trump technical interests when those business interests are critical. No business = no technical. But at some point when the company becomes stable that balance has to start shifting back towards the technical or the company itself becomes ridiculously inefficient.
lbriner|5 years ago
The truth is that the world is a complex place and you don't always know whether you can keep your existing customers by improving what you already have or get new customers to decrease the risk of declining income. It is hard to measure what is acceptable tech debt and what is worth addressing.
In one sense, the proof is in the income. If you can help customers with an 80% OK product then you just need to live with the dodgy 20% bit.
I agree with the danger of big customers though! If you ever think something is a way to avoid hard decisions, pivots and annoying some of your customers, then it is too good to be true ;-)
michalbugno|5 years ago
The point of the article is not to convince anyone to slow down as much as possible and work on bugs/etc. It's just that at some point (3, 4 years in?) there comes a time that you just have to put more effort into the things I described, otherwise it gets complicated fast.
Obviously there is a tradeoff, my opinion is that this tradeoff wasn't correctly balanced (especially further down).
at-fates-hands|5 years ago
The amount of churn we had was always the smaller clients who were still paying good money but felt neglected and would be open to changing vendors.
We went out of our way to mind the big clients and thought if we lost these few big clients it would so dramatically adverse the companies bottom line, we couldn't afford to lose them. This resulted in a situation where they were able to hold our company hostage, make unreasonable demands on time and energy and hamstrung our ability to keep our more loyal clients (who only bothered us on occasion) happy.
It was a good lesson there is a danger when you have a small company servicing a much larger company that you get into a position where your fear of losing them is a detriment to your other, more faithful clients.
renewiltord|5 years ago
trentnix|5 years ago
Big customers certainly sounded good on paper, but they came with a cost (complexity and a very hands-on relationship) that we weren’t really prepared for, and that we didn’t manage to put boundaries on fast enough.
This is an issue I've dealt with more than once in my career. It seems to occur especially when an organization values the sale more than it values the business. It's virtually impossible to successfully apply boundaries after the customer has been trained that there are none. Irrespective of how carefully you do it or how much you try to reassure your customer, it always feels like you are taking value away. I've encountered this recently in an organization that values customer happiness, which it calculates as an absence of customer complaints, more than customer success. The squeaky wheels get all the grease at the expense of just about everything else.
And the software inevitably has become a maze of settings and IF-THEN-ELSE blocks and SWITCH statements, increasing the cognitive load required to make what should be simple changes. When new features are added, the impulse is to code defensively and build inevitable customization into everything up front. And further down the spiral of technical debt and increasing complexity we go...
Slartie|5 years ago
Like: don't just add weird features that make sense for just one customer, but are eventually delivered to all customers, but instead maintain a "core product" which provides a lot of generic, core functionality, but which by itself isn't actually usable and just serves as basic building blocks to construct the actual end products which are then built and packaged individually for each (big) customer. Each of these "end products" is a development project in itself, has its own team, its own codebase, but they all draw from a core set of base functionality developed and maintained by another dedicated team that is not directly in contact with any of the customers (or at least not permanently).
This - at least in theory - prevents custom functionality for specific customers from polluting the common core product that aims to please a general audience, while still allowing for the necessary freedom to satisfy big customers demanding custom functionality, but also offering big checks in return.
pyromine|5 years ago
bleonard|5 years ago
This article is in that direction. > … the difficulties started piling up fast, and before we knew it we had to deal with a lot.
troelsSteegin|5 years ago
When it comes to your customers, I believe that you want to understand as much as you can, as early as you can, about what they are actually trying to do and about how they are actually faring with what you have sold them. I'd argue that the success of your initial customers is more important than the features for your next deal, and that feature investments should reflect that.
The problem, as the OP said, is in estimating the difference between the incremental feature requirements of your initial customers and what your strategic feature set will be. There may be no difference, and your strategy is to ship features in order satisfy each new customer as you get it. Perhaps you can design for that. Chances are though that your first customers are not wholly repsentative of your strategic, target market.
So, in the early stage, you have to make choices about allocating dev for tactical "now" or for strategic "later". In the early stage, if there is not enough "now", there is no "later". The question is when to begin budget for later. I'll argue that you are attempting to build a business, not a suspense movie, and that you budget a fixed amount for "later" from day one. Plan to succeed.
I agree with the recommendation of a strong Professional Services function as an buffer for dev, but will note that culturally you have to work hard to not silo. Your Support and ProServe teams become a primary channel of learning for dev. As a business, though, you may have to give away ProServ hours.
All in all my biggest takeaway here is that you attempt to "hire" customers strategically. As you journey together, will this customer want to go where you will want to go? What time is it when your "Big Customer" sits on your roadmap? ...
"Hire" customers strategically is easier said than done when there's salaries on the line -- accordingly look at how you compensate Sales and budget away from early stage big deals.
hn_throwaway_99|5 years ago
Very glad (most of) the cool kids have re-evaluated their stance on microservices.
michalbugno|5 years ago
unknown|5 years ago
[deleted]
pkteison|5 years ago
I suspect good arguments could be made for these things, but they need to acknowledge and justify their cost.
michalbugno|5 years ago
It's tough to suggest detailed solutions to a problem which is in general very vague. Btw I hope the article shows my appreciation for what the company achieved.
One small insight from Uzi (CEO + founder) when I discussed this with him (after publishing article): Base should've focused on much less features but with greater detail. It somehow confirms my guess that we didn't work on existing features as much as we should, but we "spread too thin".
unknown|5 years ago
[deleted]
Covzire|5 years ago
jmull|5 years ago
coldcode|5 years ago
29athrowaway|5 years ago
https://en.wikipedia.org/wiki/Scope_creep
arbitrage|5 years ago
What is it specifically about Feature & Scope creep that you would like to say?