[lightcatcher]

I was an intern at Mixpanel at the time (but not the author of this article or involved in the discussed rewrite). For context, the company was then <10 people. (I'm surprised to see anything from my first professional software experience on top of HN!)

My recollection of events (of 8 years ago): The Erlang endpoint was written a year or two earlier as someone's first Erlang project, in a tiny org with no Erlang experience. The endpoint worked well enough and people moved onto other projects in the mostly Python codebase. Eventually, it became painful to debug or add features to this endpoint because no one was particularly proficient in Erlang. Ankur (author of the post, intern) then rewrote it in Python.

I wouldn't read this as a negative about Erlang or a positive article about Python. I'd instead read it as "use a language that makes sense for your organization and already existing codebase".

[erlangNewb]

Seems like an oversight that nobody thought to get proficient with erlang. As someone who has started learning it, this seems like the obvious choice.

[diminoten]

From the company's perspective, you'd need at least 2 people doing that, preferably everybody. Bus factor of 1 isn't great, and when the rest of the team already knows Python, the benefits of switching to a language everyone knows is huge.

You're also presupposing Python is inferior to Erlang, which I'm not really going to argue about here, but I think we can all agree that Python isn't so bad that the decision to continue to use Python substantially and negatively impacted the org's ability to succeed.

[macspoofing]

>Seems like an oversight that nobody thought to get proficient with erlang.

If it's a component that nobody owns, and nobody cares about, then I'm not very surprised.

>As someone who has started learning it, this seems like the obvious choice.

The other obvious choice is to rewrite it.

[bonif]

Thanks for this comment, gives context and explains "why we switched from X to Y", answer: I did not know X, or it didn't make sense at the time. It does not mean X is bad, or Y > X (as usually suggested in this kind of articles)

[bradleyjg]

In general language diversity at a company is a big negative. Obviously sometimes there are even bigger negatives that mandate overriding that rule--you aren't going to use Java on the front end or javascript on the backend, that would be madness--but you never want two languages in the same niche. That's a cost with no benefit.

If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy. Either he doesn't understand the downsides or he does, but doesn't care.

[cellularmitosis]

Some dev: "Hey, I noticed one of our more complex API endpoints had some pretty bad response latency, so I dug into it a bit. It turns out our PHP/Symfony/Doctrine stack was generating over a hundred SQL queries, all of which were being performed sequentially because we don't have a good per-request concurrency story in this tech stack. So I wrote an alternate implementation in Clojure which has a great concurrency story and cut the latency by 75%."

lol yeah that guy should totally be fired.

[nojvek]

Language diversity is a pragmatic balance. Sometimes a language offers superior abstraction or some guarantees that eliminate a whole host of bugs and manual effort.

It’s always worth trying a hypothesis in a small spike and reliably proving whether it will scale to the rest of the org and the impact will be worth the effort.

But you’re right, there should be one official supported way of doing some kind of thing at a company.

[Silhouette]

If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy. Either he doesn't understand the downsides or he does, but doesn't care.

What if it has the potential to be a much better tool for the job than the mainstream options, and your developer thinks it's worth trying it to find out?

I've seen this situation with adopting a relatively unusual language for a task that needed particularly high reliability. Yes, it was a concern that only one person currently working in the organisation knew much about it and that it might be harder to hire help if more developers were required later. On the other hand, that software was developed faster than the similar system it was due to supersede, and it had a much lower rate of reported bugs once deployed.

Of course you don't want to hire people who intend to pad their resumes with extra buzzwords on your dollar when it has no business benefit, but equally, if you never use tools that aren't chosen from among a small set of mainstream options, you'll never outperform what you can do with mainstream tools.

[tomjakubowski]

You could fire them or you could just help them understand something that they don't. Sheesh.

[detaro]

> If you have some developer that wants to write something in a new (for your company) language, [...] you probably want to fire that guy.

Now that's

> a cost with no benefit.

[michaelterryio]

Erlang excels at being used in mixed-language shops. Erlang is an incredible control plane language.

[jacquesm]

> or javascript on the backend, that would be madness

It seems madness is pretty common.

[teilo]

My company is an example of this, and in a way that is quite different from that laid out in the article. We actually began with a Python implementation for our integration / API server. But this was well before asyncio and eventlet and kestral. It didn't scale well, and we quickly outgrew it.

We implemented its replacement in Elixir, a solution we are still running. It is written by a developer we still have, is well organized, follows proper functional paradigms, and fully leverages OTP/Genserver. It is resilient, and so far as the one two developers we have working in it at present, easy to maintain. We love it.

But we also are switching back to Python. The reason is simple: We only have two Elixir developers (really only 1.5), and we have far more work than they can complete. And when something breaks (usually because something changed in the various systems being integrated), we have one guy who can fix it, and get our production queues flowing again.

We have looked for more Elixir developers. They are rare beasts. But we have a lot of Python developers. And now that we have asyncio/eventlet/etc., the landscape has drastically changed. In Python, the various teams can create their own worker modules, and do so more efficiently because they know their own workloads far better.

Language diversity IS a problem when it impedes development and puts production at risk.

[scarface74]

We have three approved languages - C#, JavaScript and Python.

Of course C# you get all of the advantages of statically typed languages, so for major programs where you wil have multiple developers, it’s the go to language. But for smaller scripts like event based lambdas and scripting type scenarios, C# is overkill. Besides, each piece of AWS functionality is in its own NuGet package as opposed to one package for everything for JavaScript (Node) and Python.

JavaScript - our product is web based. Every developer is expected to know JS. So if someone decides to do a simple Microservice or lambda in C# or JS, it’s okay. Whichever they feel like doing, it’s okay.

We have another team that does a lot of ETL type processing. I doubt that many people would argue that JS is a better language for it than Python. If you are on that team and need to write something, you can choose whichever of the three languages you wish.

[nathan_long]

> In general language diversity at a company is a big negative

I agree. It really increases the difficulty of understanding the whole system. But it's a negative that has to be considered alongside potential benefits.

> If you have some developer that wants to write something in a new (for your company) language, or even worse goes off and does so without asking, because he wants to try it out, you probably want to fire that guy.

I get what you're saying; resume-driven development can ruin your systems. But at some point between "we wrote this new banking system in COBOL" and "every developer who knows COBOL and could maintain our old system is now dead", there needs to be an accepted way to experiment.

[aiisjustanif]

Depends. If it's a internal, innersourced, tool for everyone. Have at it, use another language. If it's my critical application, explain why we use only one language for it and don't fire them.

We teach, not react.

[ummonk]

Why would it be madness to us javascript on the backend?

[oconnor663]

I think Facebook chat had a very similar experience. The first version was written by someone who liked Erlang. Later on, when some performance issues came up (probably related to GC), no one knew Erlang, and all the high performance backend code in the company was in C++. So it made sense to rewrite the thing.

[trab]

Then they acquired whatsapp that was written in Erlang.

[mst]

Your recollection is valuable, and thank you - but contra a large chunk of the rest of this thread, the article was very clear to me that it was "our erlang was crap erlang and we didn't know how to make it not crap so we switched" which is totally reasonable and didn't seem like a negative towards erlang at all. I've supported rewrites from a language I liked more to a language I liked less for similar reasons, and what Ankur did seems like an entirely sane decision.

But even if I didn't need the clarification, the additional data was awesome, so my thank you remain sincere :)

[rlander]

So, in a Python-only org, someone with no Erlang experience decided to ship his first ever server to production, thinking Erlang would be a magic bullet to “scalability issues”. Obviously, there were maintainability issues and the org, blaming the language, decided to switch back to python.

To me, the fact that a novice was able to deploy production code that went on unmaintained for months, running a crucial part of the operation, speaks positively about Erlang.

[robocat]

"the org, blaming the language".

I didn't see anywhere in the article blaming the language - can you find a quote to support your assertion?

From the article: "After two years of iteration, the code has become difficult to maintain. No one on our team is an Erlang expert, and we have had trouble debugging downtime and performance problems. So, we decided to rewrite it in Python, the de-facto language at Mixpanel."

[_RPM]

Is it common for an intern to get such an impactful project?

[exelius]

Yeah; it’s actually pretty easy work. Interns have a lot of uncommitted time, and they’re going to leave soon so you don’t want to integrate them into the team too much. It’s also way easier to build to a well-defined spec (which an in-production endpoint would likely have).

So you let them rewrite something that already works. If what they build is good, you‘ve just knocked out a good chunk of technical debt and found someone you probably want to hire. If not, they don’t get an offer. Either way you probably didn’t pay them much to begin with so there’s a lot of upside and not much downside.

[znpy]

No idea, but that seems like a reasonable choice in that case.

Yes the project was impactful, but there already was a reference implementation with a reference performance target. Worst case scenario, they would have been back to using the old implementation.

[macspoofing]

Depends on the intern, and the component. If the intern is talented, the component is well-tested and the mentor or supervisor can do an effective code-review - then why not?

[notdang]

lightcatcher did not say that the work was done by an intern