Ask HN: What's a build vs. buy decision that you got wrong?

[kiernanmcgowan]

What are services/products that you built and wished you had bought?

Have you bought something that you had to scrap and build yourself anyways?

[nodoodles]

> Have you bought something that you had to scrap and build yourself anyways?

Yes, but that hasn't meant it was a mistake -- 'buy-then-build' can be a great strategy. Often the 'then-build' never happens, but going into a decision with the mindset readiness for 'then-build', you can learn from existing products, hit their limits and understand what is the custom version of it you'll need in your context. Recent examples are on smaller scale, though - using a library that speeds work up early, hitting its limits, and replacing or extending with DIY that does less things but goes deeper for my use case.

> What are services/products that you built and wished you had bought?

The most annoying recurring version of this has been being just a little too early - building something, then discovering a few months or years later a public product that does the same, but better. At that stage, rebuilding to use has low ROI, and one ends up maintaining a legacy monster. There was a period when public offering of supporting backend infra was maturing, ie things like secrets/configuration management, logging, observability, monitoring, a/b tests, a bit earlier even basic web frameworks (ie building anything on PHP before Laravel came out meant you built your framework first; iirc worse than the frontend framework landscape in 2022).

[jeremy_k]

I 100% agree with your second point. Case in point, my previous employer built an in house development platform to create ephemeral environments and run deployment pipelines for production. At the time, around 2016, there weren't many (or any? I wasn't part of the research time) options to buy this type of software. Today, there are more than a handful in this space.

I wonder at what point companies would finally bite the bullet and swap to buying? Maybe cataloging the feature set of the internal software, seeing what is a necessity, and seeing which vendors can cover those requirements?

[ctvo]

Workflow engines. Every company develops one base off of queues or some form of async messaging. Works great when prototyping and your initial customer base. Works less great as you grow, add more complicated features, and realize you didn't have the distributed systems expertise to write this thing to begin with. It doesn't handle any of the common edge cases, and is increasingly painful to operate, needing constant babysitting.

Use Temporal, StepFunctions, something and try to avoid this urge.

[mjr00]

"Workflow engines" and their close cousins "DSLs" are the ultimate newbie trap. In theory, it's awesome for everyone: programmers get to work on interesting, abstract problems like distributed systems, syntax parsing, event-based architectures, and "business users" get to make changes to "business rules" without bothering developers or impacting feature roadmaps. Win-win, right?

In reality you just end up making a shitty, nerfed version of a programming language, that business users can't understand, because you still have to understand conditional logic to model workflows, oh and your documentation is terrible because devs don't bother with the boring stuff. Most of the time the devs end up implementing the workflows anyway because they don't actually work properly.

If you really need a workflow engine definitely use something off-the-shelf, but I would go so far as to say that in the 95% case, you don't even need a workflow engine: you need a developer who is capable of writing some python scripts. Even if you pay a developer a full salary to do nothing but sit around and make changes to Python scripts on-demand, that's still going to be way cheaper than the complicated workflow engine solution, which will probably require a team (or multiple teams) to maintain.

[FpUser]

>"Workflow engines"

I actually had quite opposite about this particular area. At the time we were to develop particular product for a client company. The product would've greatly benefited from using workflow engine. I did some shopping around, talked to sales reps and have discovered that we would have to shell out at least $350K for our particular case. So I've proposed to the boss that I would quickly build one that would cover basics. The boss has agreed and I built it in about one month. It worked fine for what it was intended for.

After a while we have approached a vendor (the one we would have paid those 350K for their wares). We showed them what we have built, how it was used. They were impressed enough and we became sales and implementation partners. They have routed a gobbles of jobs, training and installations for us to do.

As for original home built engine - over the time we have replaced it with the one from our partner without much troubles.

Win win for everyone involved

[skrtskrt]

Temporal has really solved so many problems for us it is only opinionated about a few things that actually matter, and gives you complete flexibility otherwise.

The days of Airflow and similar seem like a stone age in comparison.

[debarshri]

Workflow engines are the most difficult thing to sell in organisations as the usecases are open ended. Organisations needs certain level of maturity to understand that they need one

[zaptheimpaler]

What exactly is a workflow engine?

At a previous job, we had a fair amount of celery tasks and logic around starting them based on user input or on a schedule, retrying on failures and marking progress or cleaning up state in various databases.

Is that a workflow engine?

[macrael]

What makes Temporal a "workflow engine" rather than a background job runner? I think I used Active Job or its like a career ago in rails. The docs on Temporal are showing me retries and storing results of the job. That does seem useful!

[barbarbar]

This comment is just too real and too precise.

[samwillis]

I built an entire custom e-commerce platform - product catalog, cart, checkout, CRM, order tracking, fulfilment management and back office. The site is a print on demand store, using contract printers. In hindsight I wish I had gone with either an off the shelf package or taken the punt on Shopify (this was 2012).

My assumptions around how much integration our custom product customiser/editor needed with the rest of the e-commerce platform ware wrong. I thought I need a user system and "saved designs" for the customers, but that's somewhat rarely used, and could have been bolted onto a standard system.

Maintaining and updating it is extra work over what the core business is, there is now a lot of custom code to fix old assumptions and implement features that we didn't previously expect. All of which come as standard with Shopify.

We also believe that customers are increasingly used to seeing the Shopify checkout, it is a reassuringly familiar experience. I suspect it has a measurable effect on dropouts.

If I was to start again now I would 100% just use Shopify, no question. We are considering a large project to move to it. It would be quite satisfying to delete all that code. But it would probably bring new problems, and thing we are used to being able to customise that we will be unable to.

Do I regret doing it? No not really, hindsight is 20:20. A lot of lessons were learnt, but that enabled us to build a successful business.

[doctor_eval]

ORMs. I didn’t “buy” Hibernate but we adopted it and the whole JavaEE thing wholesale. What a disaster. Learned a lot of lessons from that, #1 being, don’t use an abstraction layer (ORM) to abstract away another abstraction layer (SQL).

The performance of Hibernate relative to plain SQL was abominable, and this directly caused us to lose at least one contract. It turns out - on reflection, of course - that it’s not even theoretically possible to get into the same performance ballpark.

After years of doing battle with the tools, I eventually kicked them all out, decided to work hand-in-glove with the database, and suddenly things became both straightforward and performant. I now think that ORMs are a code smell.

[wccrawford]

I didn't get it wrong, but the company I worked for did. I spent 2.5 years creating a sales tool that they dropped the same day I released it. The company went with Salesforce instead.

It was really disheartening to have 2.5 years of hard work dropped like that, but it was absolutely the right choice. They had designed it themselves and it was barely functional. They had a lot of upgrades planned to make it do what they really wanted.

[hangonhn]

That's just bad management. It didn't take them until the release date to realize that SF was the right choice. They just procrastinated until they couldn't anymore when you released your project.

I had the same thing happen at a previous job. My team built and executed and hit every object and was ready to ship and that's when they decided to inform us that they don't want to ship what we built and wants us to integrate our features into a product they had acquired. Like that decision could have been made 6 months or even a year earlier. Instead they procrastinated until the last moment. I decided to leave the company after that. Seeing 2 years of effort and good execution flushed down the toilet was very disheartening.

[password4321]

The last three years of my work will be permanently abandoned

https://news.ycombinator.com/item?id=33804293 by Eric Lippert last month

[Nican]

Can you tell me more about the Salesforce decision? Salesforce is one of those companies that I hear the name a lot, but seldom do I get to read insights about them.

[yamtaddle]

I have frequently found myself directed to build something when it clearly would have been better for the company to buy (or use SaaS). It seems like a lot of business owners and managers have some mental block that makes it hard for them to pay for that kind of thing directly, but easy for them to burn shitloads of developer, ops, and support hours doing the same thing in-house even when there's no strong business reason to do so. Totally bizarre.

[vesinisa]

Did you continue with this company?

[gbro3n]

Not a decision I've got wrong, but I'd suggest authentication services are a common regret. Yes authentication is important to get right, but it's not _that_ complex. SaaS tools for auth are incredibly expensive for the feature set they offer, and are difficult to replace once embedded in your services. SSO in large scale business is a case where buy is the right option but I'd anticipate that for many simpler use cases teams have to work hard to unentangle themselves from Okta etc.

[tschellenbach]

Most common mistakes I see people make:

- They ignore overhead. If an engineer makes $100k they calculate with those number. The reality is a 2X to 10X overhead depending on the company. - Required return. You can't spend $100 to make $100, no investor will fund that. So you need to do activities that generate an adequate return. - Opportunity costs. So say that you have an engineer that costs 100k, the overhead takes that to $400k and they need to generate at least $600k a year. That still doesn't mean that you should do the project if you have something else that's even better. - Maintenance. People always underestimate the cost of maintenance. I'm skeptical of any estimate where maintenance is < build costs

One thing I regret building is an analytics pipeline. We should have relied on Segment for that. I've also once built an analytics platform from scratch which was bad.

On the flip side, one time we bought an ETL tool and it was terrible compared to in-house solutions

[deathanatos]

The common mistake I see people make is presuming that the "buy" will deliver on their end of the deal. When they don't, that engineer making $100k is spending a lot of time (i.e., and money) writing support tickets, desperately trying to get it escalated to someone on the vendor's side who can actually do something, as inevitably you have to first peel away the layers and layer of customer support, account managers, "solution" people who aren't actually capable of fixing anything. (Even for some inanely simple questions, sometimes, things that I think "wow, a support rep. should actually be able to answer this, it's not a highly technical problem for once" and then, nope, it has to get escalated to eng., inevitably.)

The amount of my own salary that has gone towards "Azure support ticket monkey" is frankly frightening. And I have never once seen that included in the estimate for "buy".

Many of the "build" solutions I have run require <1 engineer to maintain. When they require maintenance (whether due to something needing an upgrade, new feature request, etc.), yeah, there's maybe a two or three week piece of work, but then there are months long spans where it just sort of hums along in the background.

Even outside of "support", I still end up having to dedicate eng time to "buy" solutions, to fill in gaps in their implementation. (E.g., an artifact store having read-your-writes bugs. Heck, Github Actions ("buy" of CI) has so many bugs in it that we hit in the course of adopting it…)

[mritchie712]

what was the ETL tool? Fivetran is great at EL, but leaves so much work on T. They have some open source dbt models[0], but they need a lot of work. For example, the Hubspot model doesn't have a way to join most of the tables (e.g. you can't join Deals and Companies, which seems like an obvious thing you'd want).

If they dedicated one data engineer to building an amazing set of dbt models on top of their raw EL output they could drastically improve the lives of their customers.

0 - https://hub.getdbt.com/#:~:text=feature_store-,fivetran,-ad_...

[parsley27]

This is a great explanation of overhead and opportunity cost. I'm always trying to make that pitch for selling my product, but the biggest issue is that I think many companies are just jumping in on BUILD without even researching BUY.

[johnklos]

In the past I've built servers by carefully selecting parts, calculating maximum power usage versus airflow, calculating full time electrical load, building, testing with fans disabled in rooms with high ambient temperatures, looking for failure modes, researching failure rates of specific power supplies and drives, et cetera.

I'd build one or two, test them for several months while making adjustments and fixing various weaknesses, then I'd build ten, twenty, however many were needed.

One particular client decided they wanted to skip all that and just spend the money on Dell. They didn't have many servers, certainly not enough to justify a separate server room. Their offices weren't air conditioned at night, and they had an entire summer where one server or another would become unresponsive over the course of a weekend, and sometimes at night between weekdays. Accessing iDRAC was beyond what they wanted to do, so of course I had to do that.

They had Dell support, but Dell had no "fix" for unstable machines other than to tell them to build a server room. Mind you - the ambient temperatures were always below 100º - any reasonable person would say that while that's not ideal for servers, "premium" servers should still be able to handle warm rooms, particularly when they're idle, and not crash or lock up.

After that fiasco, they gradually replaced the Dells, one at a time, with machines I built. I wish they had tried harder to return the Dells, but they just wrote them off.

I've learned that any savings in time and money (mostly - the Dells cost more than the machines I build, even accounting for extra billable time) aren't worth the loss in time, productivity and reliability in the long run. Of course, the opposite would be true if I just wanted more billable work, but I can't do that, unlike many others in the field.

BTW - when I build a new generation of servers, I do months of testing, but the same general platform can last a good five years, like Ryzen 1000 through 5000 systems with ECC have lasted since 2017.

[flyinglizard]

I might have a consulting gig for you if you’re interested. Let me know how to get in touch if relevant!

[c_o_n_v_e_x]

Is there much business building custom rack mount servers for people? I would have thought that market would have been taken out by the big brands.

Do you have a favored brand for server motherboards?

[rsync]

Sometime around 2019 I went all in on the Ubiquiti ecosystem, specifically the UDM-PRO "Dream Machine" and all of the integration with cameras and wifi APs that could be done with that.

I made this decision vs. building out a new gateway/router/switching/monitoring/SSIDRoaming infrastructure from scratch.

This was a bad decision.

Even now, nearly 2023, the UDM-PRO is a beta product and I am a beta tester. Further, we have all learned that Ubiquiti is a dysfunctional organization not focused on anything at all resembling technical/engineering goals.

Ubiquiti wifi APs are still, probably, some of the best available so I will probably keep those ... but everything else - including PoE switches is getting ripped out and replaced.

[damascus]

I'm getting ready to do a build for a client with about 60 people and no expectations of growth. They are technical and want gear that one or two of their folks can be trained to 'power user level admin' vs calling me every time (hourly consultant). I was investigating Ubiquiti gear, but you've given me pause.

What gear did you go with and/or would recommend to check out?

[chucky_z]

I second this. I went all in on the dream machine and some of its parts and it failed in an unrecoverable way, I sent it in for repairs; and it failed again in a different unrecoverable way. I just use the crappy Comcast router with a ubiquiti AP now.

[moonchrome]

I've seen a project outsource a core component "to speed up development". Turns out integrating with a 3rd party service added similar level of completely, all the design decisions had to be made around 3rd party system which constrained the project pointlessly. Project eventually got scrapped - part of the reason was that their core component wasn't even their own and they would have to renegotiate the license to pivot.

[dclusin]

Definitely had this happen in recent memory. Also the thing to be careful about here from experience is you still need to qa the contractors deliverables. Team had a contractor delivering features and it turns out what was “delivered” many times didn’t actually work. Wasted several months of my engineering time babysitting and validating said contractor. Net result was wasted time and money as we just ended up reimplementing ourselves.

[levelforge]

Magento Cloud... a.k.a. platform.sh was exactly that.

[bnert]

cough azure...

[wheybags]

A commercial Nas (qnap, this time), vs just a simple Linux box with hard drives. Simple Linux box was the better option. We even wrote a blog post about it: https://www.factorio.com/blog/post/fff-330 (scroll down, it's topic #3. Also, I don't work there anymore)

[db48x]

ZFS is ridiculously good too. So is Factorio, in fact!

[gwbrooks]

Virtually everything in my self-hosted tech stack: Web, mail, specialty apps for CRM and project management... all of it.

I like the control and I like tinkering. But as I've become busier, I realize how much revenue-producing/move-the-ball-forward time I've lost and am still losing by doing all of this myself.

[VTimofeenko]

I am spending an inordinate amount of time on my homelab (and the todo list does not seem to get any smaller). But on the other hand this activity gives me a ton of experience that I could not get doing $DAYJOB, and this experience comes extremely handy from time to time.

[europeanguy]

There's people who use hammers because they have nails that need hammering, and there's people who use hammers because they enjoy hammering.

[c_o_n_v_e_x]

- A former CTO made the decision to fork thread mesh protocol for an IoT project. Thread (and our development) was totally inadequate tech for industrial usage. Months of development went by and no progress was made. We wound up licensing some proprietary mesh protocol.

- Same CTO somehow convinced investors there was no x86 machine available on the market with the right specs for what was needed… so they put together a team of hardware people to design and build a motherboard.

- Personal… bought a late 70’s / early 80’s Sol cat catamaran sail boat for 500 bucks while in college. Unbeknownst to me, the hulls were notorious for delaminating and I didn’t know what to look for at the time when I purchased it. Long story short - I spent months of effort fixing and painting it, a lot of money, and sailed it once before giving up on it (of course it partially sunk during the maiden voyage). It wound up blowing away in a hurricane.

[version_five]

My opinion is most companies that have built in-house "AI" or machine learning teams, or hired consultants to custom build AI "use cases" for them have made the wrong decision and are coming around to it. This is an area where buying a product is going to win, because most companies don't have the capability of successfully managing in-house ML teams (I've seen line departments trying to hire a PhD ML scientist) and because a company with a product has done the work to figure out a proper value proposition, as opposed to just trying to ham fist AI somewhere into operations.

[cdolan]

How can you buy a product in the AI/ML space when the data needed to generate the model (and the value) is proprietary?

You may be correct that buying is better than building in a lot of these areas, but I don’t think its practical given IP concerns

[starik36]

About a decade ago or more, I spent a ridiculous amount of time trying to build a rig to mine bitcoin.

Should have bought a $1000 worth and then just post this from my Caribbean island.

[unnouinceput]

1k USD a decade or more would've netted no more than 20 bitcoin. Even at all high time of, let's say 70k USD, that would've gave you a maximum of 1.5 M USD before taxes. Doubt any Caribbean island is anything less than 10 M USD, unless you buy one where you can only stand on one feet.

[takeda]

You could just build an exchange :)

[steveBK123]

Most buy vs build decisions that go wrong with buying is because there was insufficient due diligence on the vendors, comparison to other vendors, and general ignorance of the thing being bought-or-built.

The worst outcomes I've seen are when it's SOWs that are buy-to-build. That is - paying a vendor to build something that doesn't exist yet. It's like the blind leading the blind. You have to agree to these detailed specs and project plans and contract, and then at the end of the day hope they deliver because there's so much nuance in software it's not going to hold up in court or be with your time. If you did write specs and acceptance criteria detailed enough to be bullet proof, you'd probably have been better off just writing the software instead.

My company did two of these recently, one ended up overrunning by 100% with 50% of requirements not met, never going to production and half the people on vendor side quit/fired. The other one went to production and then the vendor went bankrupt, lol.

[dpkirchner]

This is sort of related but I've often wished that I hired someone to work on my house, instead of DIYing. I'm never going to be good at most house-related repair and upgrade tasks that I will do one or two times in my life. And I've been unhappy with the results of most of my work.

The main reason I don't immediately jump to hiring is it takes weeks to get people out for estimates, then half of them ghost you, and if you do manage to convince someone to take your money they are booked months out (and they also ghost you).

[takeda]

> Have you bought something that you had to scrap and build yourself anyways?

I'm very close to with another product.

HashiCorp is way overhyped. Yes, their open source products aren't bad if you don't pay for them, but then you get an enterprise license and realize how bad their support is. Seems like the company thinks that "enterprise" means just a product with additional features, but you're still on your own with any help.

Consul, especially when running on k8s is very complex, it feels like support barely knows more than you. They won't answer any questions explaining how given feature works, referring to solutions architects, which takes months to get access to them. Unlike Open Source, since you don't have the source, you can't just modify code yourself to add missing functionality, and if you ask your rep about it, they might tell you it could take a year (if it even happens) to implement it. WTF?

[throwawaaarrgh]

Have never regretted a buy. Sometimes you need to pivot if price gets too high, but there's always been mitigations and alternatives.

Anything I've seen built that was not contributing business value ended up being a distraction.

[Apreche]

At my previous job the company heavily relied on an industry standard application. This app basically had a monopoly on that particular industry. However, they were pushing the app beyond the limits of most other companies. It was also just generally an old and busted Windows 2000 era kind of deal.

When I first arrived, and knew nothing, they told me we were going to build a replacement. That sounded great to me at the time. Obviously the right thing to do is replace something old and busted.

Well, I and the company learned over time that actually that was not really the best business strategy. It may have been possible, but not with the resources we had available. I ended up doing a whole lot of work that went unused, through no fault of my own.

[habibur]

A better strategy is to first move a small part of it out of the system. And not go for a complete rewrite. And then make sure the old system can work as a proxy for the new system, or vice versa.

When both can be used simultaneously you can now gradually move from one to the other, or revert back as needed.

This is what I followed.