Nice, I laughed out loud when I read these two paragraphs:
> Initially, we tried messing with some garbage collector parameters we didn’t really understand, but to our surprise that didn’t magically solve our problems so instead we disabled garbage collection altogether. This increased our memory usage, but our automatic on-demand scaler handled this for us, as the graph below shows
> Today we are making some of the code that we can afford to open source available on our GitHub page. It is useless by itself and is heavily tied to our infrastructure, but you can star it to make us seem more relevant.
Sigh, it looks like the title got a bit mangled…oh well. As most of the commenters have gathered, this was mostly tongue-in-cheek; I kept a window with a few of these kinds of posts open as I wrote so I could refer back to them so if you might be able to make out some familiar companies or technologies in there. That being said, I love hearing about your employer's technical decision making process, so I don't want to discourage anyone from writing blog posts like these. Just try to keep in mind that your decisions not be the right ones for everyone else. On the flip side, if you're looking to make technical decisions yourself, sometimes what $FAMOUS_COMPANY did is not the right choice for you :)
Nice one. I'll cast my vote for $AN_ENGINEER_TOOK_A_MYTHOLOGY_CLASS as "funniest bit."
Obviously at some point you're gonna have to do the companion/followup piece, "Why $FAMOUS_COMPANY Ditched $HYPED_TECHNOLOGY for $BORING_DEPENDABLE_TECHNOLOGY and What We Learned"
Saagar - I haven't laughed this hard in months. Thank you for writing this - the bit about "Medium article we found that had something to do with multi-armed bandits" got me on the floor. Laughing during this time feels so so good, and I didn't realize how much I needed that!
Lost it at $AN_ENGINEER_TOOK_A_MYTHOLOGY_CLASS. I guess it's not as bad as $A_FOUNDER_WHO_TOOK_AYAHUASCA or $THE_VC_WHO_SPEAKS_ONLY_IN_RAP_QUOTES_AND_METAPHORS.
I can't believe anyone is still considering $HYPED_TECHNOLOGY in $CURRENT_YEAR. We at $NO_PROFIT_STARTUP gave it our best shot, but the bugs they mention aren't as small as they seem -- they completely sank the project. At this point it's clear that $HYPED_TECHNOLOGY is falling behind on both adoption and ergonomics to $TRENDY_TECHNOLOGY, which handles these issues in a much cleaner and more scalable way. We'd strongly suggest anyone considering $HYPED_TECHNOLOGY to take a closer look before following the advice in this post.
Right, and those cowboys at SFAMOUS_COMPANY obviously haven't read the latest blog posts that clearly show $HYPED_TECHNOLOGY doesn't support $HIPSTER_PARADIGM in an async I/O functional stackless copy on write environment, meaning $FLASHY_LANGUAGE can't be used with dependent types unless you don't mind your static analyser giving false positives every time you try to multiplex coroutines in your template meta-programming frameworks; what were they thinking?
We hear you. We at $SOON_TO_BE_FAMOUS_COMPANY spent $N_YEARS developing our own version of $HYPED_TECHNOLOGY, because $HYPED_TECHNOLOGY didn't quite fit our use case in that we couldn't understand how to operate it well on our environment.
Did you consider $EVEN_WORSE_HYPED_TECHNOLOGY? Based on a keyword match with some academic paper I didn't understand and my poor understanding of virtualization, containers, and latency, it could solve all your problems.
> Every metric that matters to us has increased substantially from the rewrite, and we even identified some that were no longer relevant to us, such as number of bugs, user frustration, and maintenance cost.
That has happened more than I'd like to admit: massive overhaul effort, $new_thing is an improvement in many ways but has all of the reliability and maturity problems of a new system that hasn't gotten years of patches. But the team feels like they have to show ROI on reinventing the wheel, so they massage the metrics to look like an unambiguous success, when everyone involved at the ground level knows it wasn't.
> We hope that you internalize our company’s anecdote as some sort of ground truth and show it to your company’s CTO so they too can consider redesigning their architecture like we have done.
The sad part is how often this is the case. Not just about technology choices, but everything else as well, HR, marketing, and so forth.
Someone reads something, usually very short - that's semi correct, but lacks a ton of context, and then it gets accepted as some sort of unquestionable truth.
This page really should just support get params in the url! Do you know how many people I could trick into believing this with ?FLASHY_LANGUAGE=rust&HYPED_TECHNOLOGY=kubernetes ??
Sums up the last 2 companies I've worked for and they weren't even large companies reallly.
Re-writing our entire platform in Go from .Net (at both places) solved zero of the problems we had as there is nothing in language syntax that could fundamentally fix: priority churn, lack of dev skills, no testing on the most critical platform components(!).
IMO it made it worse because 2 weeks of PluralSight doesn't turn a .NET (or any other) dev into a high performing Golang dev. They just write .NET apps using Golang syntax - a common Golang refrain I hear. So if you thought they weren't great in uncool-language-xyz then just wait until they 'learn' Go.
That excerpt about your experience almost felt like a post mortem of my last job. The only differences are that we went from Typescript > Go, and had absolutely no time to stop and learn the language.
“but as we seek to grow further it’s clear that a complete rewrite of our application is something which will somehow prevent us from losing two billion dollars a year on customer acquisition”
Used to work at a PaaS, which had a 17 year old codebase. It did something that none of our competitors could do, which was really important to our client base.
We were a second-tier player in the space, where the 1st-tier player would be exponentially more expensive to implement and of little benefit to any but the largest clients (think, SAP vs Mysql, for analogy's sake).
When courting new customers, we often encountered the question, "When are you going to re-write your product and get off [language platform]?"
We wrote a white paper to explain where we were, where we were going, and how [language platform] wasn't impeding our 1) security, 2) performance, 3) extensibility.
But we got that question every time. CEO asked tech team how long it would take to re-write the entire 17 year old code base, without losing any client facing features, in [any_new_platform].
They ended up selling to a big financial concern (for analogy's sake, think CA -- Computer Associates).
You know, I guess these kinds of articles are formulaic, but I enjoy seeing the rationale for switching. There definitely are reasons to sometimes migrate to another infrastructure or language (be it cost, ease of hiring, performance, etc), and I like hearing the rationale behind it, even if I don't agree with it.
Going onto the other end of the spectrum, I had a job in 2012 doing ColdFusion, ActionScript, and DOS FoxPro. ActionScript was still semi-relevant, but ColdFusion was barely supported anywhere anymore, and FoxPro (in any version AFAIK) hadn't been updated in at least six years.
Whenever I got annoyed by this, they had this strong attitude of "if it ain't broke, don't fix it", and told me that there's no reason to rewrite anything.
Coldfusion isn't the worst language I've used, but ColdFusion code can be extremely difficult to debug, especially before they introduced CFComponents, and after about six months, I quit. I haven't talked to anyone there in a number of years now, but my understanding is that they really haven't been able to replace anyone who left, since the number of people willing to do ActionScript and Coldfusion and FoxPro is shrinking every day.
Changing to $Hyped_technology has one really good side effect: it attracts talent. Is your site getting 2000 visitors a day really going to benefit from Kubernetes? Is your Android app really going to see an appreciable difference if you rewrite the server in Rust? Is your code really going to have less bugs if your new server it in Haskell? The answer to all these questions is "maybe", but all of those technologies have the advantage of being sexy to people who absolutely love compsci, and being able to attract that kind of talent has value.
I don't know how much of a success story this really is, but I took a job at Jet.com, when that was still relevant, because it was using F#, and that seemed pretty neat. I met some of the most incredibly smart and talented engineers that I've ever worked with there, and I think that's in no small part because a lot of people who are interested in F# do programming for more than "just the money".
> Today we are making some of the code that we can afford to open source available on our GitHub page. It is useless by itself and is heavily tied to our infrastructure, but you can star it to make us seem more relevant.
Our internal studies showed that gaslighting users by showing them a completely new interface once in a while and then switching back to the old one the next time they loaded a page increases user engagement, so we made sure to implement such a system based on a Medium article we found that had something to do with multi-armed bandits.
"Every metric that matters to us has increased substantially from the rewrite, and we even identified some that were no longer relevant to us, such as number of bugs, user frustration, and maintenance cost. Today we are making some of the code that we can afford to open source available on our GitHub page. It is useless by itself and is heavily tied to our infrastructure, but you can star it to make us seem more relevant."
Most people here seem to agree that $Hyped_technology is a bad choice vs. $Old_and_tested_tech ... but those same people piss on tech like PHP and/or MySQL and instead recommend $Hyped_technology
company i am consulting switched from php stack to client side react. killing search traffic and botching evey other metric they had. inlcuding revenue.
there is a german word for this fallacy: “Technologieglaeubigkeit”: the belief that with modern technology you automatically make a the right decision.
I'm waiting for that $startup_name to try (ab)using this $Hyped_technology everywhere and subsequently writing a blogpost titled: '$startup_name: Our incredible journey'.
$Famous_company to $other_startups: We can't wait to see what you'll come up with!
> We know you’ll ignore the fact that you’re not us and we have enough engineers and resources to do whatever we like, but the decision will ruin your startup so it’s not like we’ll see your blog posts about your experience with $HYPED_TECHNOLOGY anytime soon.
[+] [-] est31|5 years ago|reply
> Initially, we tried messing with some garbage collector parameters we didn’t really understand, but to our surprise that didn’t magically solve our problems so instead we disabled garbage collection altogether. This increased our memory usage, but our automatic on-demand scaler handled this for us, as the graph below shows
> Today we are making some of the code that we can afford to open source available on our GitHub page. It is useless by itself and is heavily tied to our infrastructure, but you can star it to make us seem more relevant.
[+] [-] rndgermandude|5 years ago|reply
[+] [-] segfaultbuserr|5 years ago|reply
Made my day.
[+] [-] JJMcJ|5 years ago|reply
[+] [-] saagarjha|5 years ago|reply
[+] [-] rdiddly|5 years ago|reply
Obviously at some point you're gonna have to do the companion/followup piece, "Why $FAMOUS_COMPANY Ditched $HYPED_TECHNOLOGY for $BORING_DEPENDABLE_TECHNOLOGY and What We Learned"
[+] [-] ceocoder|5 years ago|reply
[+] [-] roosterdawn|5 years ago|reply
[+] [-] mistersquid|5 years ago|reply
Some great writing here. Thank you so much.
[+] [-] momirlan|5 years ago|reply
[+] [-] panic|5 years ago|reply
[+] [-] shric|5 years ago|reply
[+] [-] gorbachev|5 years ago|reply
Look for our 25 part blog series at http://$SOON_TO_BE_FAMOUS_COMPANY.com/blog/#WEAREFUCKINGNINJ... on how to operate our version of $HYPED_TECHNOLOGY well on your environment.
[+] [-] julianeon|5 years ago|reply
[+] [-] bernawil|5 years ago|reply
[+] [-] JJMcJ|5 years ago|reply
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] munchbunny|5 years ago|reply
That has happened more than I'd like to admit: massive overhaul effort, $new_thing is an improvement in many ways but has all of the reliability and maturity problems of a new system that hasn't gotten years of patches. But the team feels like they have to show ROI on reinventing the wheel, so they massage the metrics to look like an unambiguous success, when everyone involved at the ground level knows it wasn't.
[+] [-] voidfunc|5 years ago|reply
[+] [-] raiyu|5 years ago|reply
The sad part is how often this is the case. Not just about technology choices, but everything else as well, HR, marketing, and so forth.
Someone reads something, usually very short - that's semi correct, but lacks a ton of context, and then it gets accepted as some sort of unquestionable truth.
[+] [-] ChrisSD|5 years ago|reply
[+] [-] mrosett|5 years ago|reply
[+] [-] troughway|5 years ago|reply
Example: https://rbt.asia/g/thread/41920845 / https://news.ycombinator.com/item?id=7745561
[+] [-] AnimalMuppet|5 years ago|reply
[+] [-] therealdrag0|5 years ago|reply
[+] [-] vatotemking|5 years ago|reply
[+] [-] xxxxxxxxxxxx11|5 years ago|reply
[deleted]
[+] [-] spiantino|5 years ago|reply
[+] [-] AegirLeet|5 years ago|reply
I didn't make it very pretty, but it should work.
[+] [-] saagarjha|5 years ago|reply
[+] [-] legitster|5 years ago|reply
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] aasasd|5 years ago|reply
[+] [-] pseudosavant|5 years ago|reply
Re-writing our entire platform in Go from .Net (at both places) solved zero of the problems we had as there is nothing in language syntax that could fundamentally fix: priority churn, lack of dev skills, no testing on the most critical platform components(!).
IMO it made it worse because 2 weeks of PluralSight doesn't turn a .NET (or any other) dev into a high performing Golang dev. They just write .NET apps using Golang syntax - a common Golang refrain I hear. So if you thought they weren't great in uncool-language-xyz then just wait until they 'learn' Go.
[+] [-] Kaze404|5 years ago|reply
[+] [-] albeebe1|5 years ago|reply
[+] [-] jpxw|5 years ago|reply
Lmao
[+] [-] nkingsy|5 years ago|reply
Bwahahahahaha
[+] [-] jasonv|5 years ago|reply
We were a second-tier player in the space, where the 1st-tier player would be exponentially more expensive to implement and of little benefit to any but the largest clients (think, SAP vs Mysql, for analogy's sake).
When courting new customers, we often encountered the question, "When are you going to re-write your product and get off [language platform]?"
We wrote a white paper to explain where we were, where we were going, and how [language platform] wasn't impeding our 1) security, 2) performance, 3) extensibility.
But we got that question every time. CEO asked tech team how long it would take to re-write the entire 17 year old code base, without losing any client facing features, in [any_new_platform].
They ended up selling to a big financial concern (for analogy's sake, think CA -- Computer Associates).
Was probably the right move.
[+] [-] tombert|5 years ago|reply
Going onto the other end of the spectrum, I had a job in 2012 doing ColdFusion, ActionScript, and DOS FoxPro. ActionScript was still semi-relevant, but ColdFusion was barely supported anywhere anymore, and FoxPro (in any version AFAIK) hadn't been updated in at least six years.
Whenever I got annoyed by this, they had this strong attitude of "if it ain't broke, don't fix it", and told me that there's no reason to rewrite anything.
Coldfusion isn't the worst language I've used, but ColdFusion code can be extremely difficult to debug, especially before they introduced CFComponents, and after about six months, I quit. I haven't talked to anyone there in a number of years now, but my understanding is that they really haven't been able to replace anyone who left, since the number of people willing to do ActionScript and Coldfusion and FoxPro is shrinking every day.
Changing to $Hyped_technology has one really good side effect: it attracts talent. Is your site getting 2000 visitors a day really going to benefit from Kubernetes? Is your Android app really going to see an appreciable difference if you rewrite the server in Rust? Is your code really going to have less bugs if your new server it in Haskell? The answer to all these questions is "maybe", but all of those technologies have the advantage of being sexy to people who absolutely love compsci, and being able to attract that kind of talent has value.
I don't know how much of a success story this really is, but I took a job at Jet.com, when that was still relevant, because it was using F#, and that seemed pretty neat. I met some of the most incredibly smart and talented engineers that I've ever worked with there, and I think that's in no small part because a lot of people who are interested in F# do programming for more than "just the money".
[+] [-] 0x262d|5 years ago|reply
Good, less satirical related article: https://logicmag.io/failure/freedom-isnt-free/
[+] [-] bassman9000|5 years ago|reply
Some darkness here.
[+] [-] gumby|5 years ago|reply
[+] [-] zadkey|5 years ago|reply
This is gold.
[+] [-] tiffanyh|5 years ago|reply
Most people here seem to agree that $Hyped_technology is a bad choice vs. $Old_and_tested_tech ... but those same people piss on tech like PHP and/or MySQL and instead recommend $Hyped_technology
[+] [-] franze|5 years ago|reply
there is a german word for this fallacy: “Technologieglaeubigkeit”: the belief that with modern technology you automatically make a the right decision.
[+] [-] snazz|5 years ago|reply
[+] [-] rvz|5 years ago|reply
$Famous_company to $other_startups: We can't wait to see what you'll come up with!
[+] [-] greendave|5 years ago|reply
Brilliant.