top | item 39612443

Leaving LinkedIn

473 points| wyclif | 2 years ago |corecursive.com | reply

314 comments

order
[+] bgirard|2 years ago|reply
> Okay. Well, I can keep butting my head against this wall. I had conversations with that manager where I was told in literally so many words, ‘You’re too idealistic. You don’t care enough about the bottom line. You should change your values.’ And I was like, no, nope, that’s not how this is going to work, man. Uh, outside I did the politic.

To me that's the most interesting quote of the podcast and that's the impression I had gotten before reading this quote. Sounds like they received valuable feedback along the way and ignored it on purpose.

My career lesson is that being right isn't the hard part of being a senior staff engineer. Being right isn't enough. It's building alignment across the entire org towards the right solution. That's the hard valuable problem that I'm paid to solve.

I worked on the facebook.com rewrite to React in 2019. I find this story particularly interesting.

[+] chriskrycho|2 years ago|reply
There's an element of truth to this! I alluded to this (and Adam totally reasonably cut even more of it for time) but I definitely was not as effective organizationally as I could have been; that is one of the major things I learned from my time at LinkedIn. The biggest challenge for DX teams remains figuring out how to (a) communicate but more importantly (b) align their work with the key priorities of the business. I got to be okay at (a) but never particularly succeeded at (b) in my time at LinkedIn. Some of that is on me! Some of it… is on LinkedIn.

That said, in this case, the person telling me “You’re too idealistic” literally meant it in the sense of “You should not care about anything unless it directly moves the bottom line,” and I reject that to my bones. The bottom line matters. So, though, do things like user experience, developer experience, and for that matter just basic ethics about what we build (though happily the latter wasn’t in view here).

[+] ejb999|2 years ago|reply
>>My career lesson is that being right isn't the hard part of being a senior staff engineer.

Not only that, but your own definition of 'right' may or may not align with what is 'right' for the people paying your salary - pretending otherwise is just silly.

In any organization, you advocate for what you believe is 'right' the best you can, someone else (or a consensus of others) will likely decide if they agree - you get to decide if you are willing to live with it/compromise, or move on - I have done both in my career.

[+] atomicnumber3|2 years ago|reply
I mean, yes and no. I find staff engineers, and especially the senior-staff-engineer-type positions, to be the ones most sensitive to the exact specifics of the org they've joined.

I've seen a senior staff engineer (at a popular unicorn you've all heard of) who was a very smart, but also friendly and reasonable person (how often are those traits in opposition of each other!). They were very good at the staff engineer game. They were vouching for us to upgrade a framework that we pumped 50M of spend through per year. From v2 to v3. It wasn't a huge change, not like python 2 to 3 at all. Positively trivial compared to py2 to 3. But management just were not up for it at all. Even with research done showing we could expect literally a baseline of 10% improvement in performance (so you do the math on that vs the 50M of spend), they just did not want to "waste time on version upgrades." The staff engineer was told they were being unreasonable in pushing so hard. Their manager tried to ditch them, but again, they're good at the staff eng game, so they switched teams due to being in the good graces (due to an excellent track record) with a middle manager above them. They decided, hey fuck you, I'm going to one man army this effort. They did, they had preview versions ready in less than a month and had migrated some of the bigger-win jobs within 2, saving their annual comp (which was prodigious) several times over just in that. Suddenly everyone was keen on moving their stuff over now that the startup cost (both political and eng-time) was paid.

Anyway fast forward a year, the rollout is complete and nearly the entire fucking management chain above him has turned over, a 50/50ish mix of firings vs quittings. But he's still there and the upgrade migration is complete.

So sure, sometimes staff engineers have a stick up their bum and can't dislodge it enough to get work done on the bottom line. But sometimes they're the only sane ones in a crazy world, and you can only do so much to try to turn a balky mule around.

[+] skeeter2020|2 years ago|reply
you gotta a do what you gotta do, but if you have the luxury, and conceptual integrity, there's going to be sacrifices in the name of "alignment" that you're just not prepared to make. I agree with your positioning on lots of "right" solutions, IME it's recognizing and understanding and still not accepting when it violates your foundational values. I've been there and had the ability to not participate, but this isn't always the case.
[+] wlll|2 years ago|reply
> To me that's the most interesting quote of the podcast and that's the impression I had gotten before reading this quote. Sounds like they received valuable feedback along the way and ignored it on purpose.

We don't really have enough context to make a judgement here IMO. I've been in places large enough to have fairly complex politics where people would manouver themselves into better jobs as much as they could, including performing a hostile takeover of another department. They might not have had the best ideas or the best plans, but they did have the right connections, go to lunch with the right people and say the right words.

I've seen bad decisions made because more persuasive and better connected people were attempting to climb the ladder and managed to talk management into it.

"You’re too idealistic. You don’t care enough about the bottom line. You should change your values." could just be something someone says when they're trying to smear someone in the way, especially if that's how they've sold themselves and their ideas to upper management.

Personally I've seen and done a number of large scale changes/upgrades (mostly Ruby/Rails/Postgres things) in orgs smaller than Facebook but still with hundreds of engineers and large codebases, both stop the world, and where the work was streamed seamlessly into regular feature etc. work and the methodology outlined by Chris seems incredibly sensible and aligns with what I have found to be the most successful way of doing these things.

As the host says, we ony have one side of the story here, but at least a lot of the things Chris is saying seem to make sense.

> My career lesson is that being right isn't the hard part of being a senior staff engineer. Being right isn't enough. It's building alignment across the entire org towards the right solution. That's the hard valuable problem that I'm paid to solve.

I agree, leadership positions require trust and respect to be effective. You still have to be right for that to be useful of course. Progress in the wrong direction isn't really progress.

[+] choppaface|2 years ago|reply
A great choice quote, and add to that LinkedIn as an engineering culture already heavily skews towards “craftsmanship” (as they call it) and refinement versus the larger industry. The speaker appears ridiculously unaware of how outlier his position is.

Do you want to wax the wheels of a title-winning race car or of a 1990 Toyota Camry that happens to be in mint condition? A lot of LinkedIn engineers I’ve met are the latter, and Microsoft is trying to push them into being the former. If you are The One True Master Wheel Polisher then sure maybe you get to choose your car, but don’t soapbox it. The audience is smarter than that (I hope).

[+] samstave|2 years ago|reply
Have you ever worked with a BOFH?

BOFHs suck.

Your career lesson is correct.

[+] willvarfar|2 years ago|reply
Its interesting because, although I don't know the linkedin codebases nor worked at linkedin, I've seen a few codebases that sound scarily similar and am familiar with a few companies that sound organisationally and politically scarily similar.

And I generally advocate 'finger gun' approaches.

Finger gun rewrites can be implemented in a good way. Often when you have n clients trying to do the same thing, one of those can be used as a basis for serving the other platforms. Or if you start over, you can do it in a good clean quick concise way.

The general success ingredient is putting a small team of veterans on making the new system. Success always comes from a small team of engineers who are both domain experts and technical experts rolled into one. (Controversially, I think that generalises; all success comes from this, even for mundane keep-the-lights-on problems. And everyone else is just slowing them down.)

The big problem most tech management everywhere keeps repeating is that the next big system will be built the least-experienced.

Would be really nice to hear the symmetrical interview from a finger-gunner.

[+] ztratar|2 years ago|reply
Sounds to me like he made a couple unfortunate moves:

- Propose a 5 year plan (lol) - Don't lead the incident with leadership, but with blame - Speaking about problems more than solving problems (hard to do) - Lack of relationship building

On one hand, I empathize with Chris. On the other hand, this sounds to me like he just didn't know how to perform in this environment. And that's totally fine! Not everything should learn how to perform in the bureaucratic knots -- startups are simpler in this way. And there's a reason the big guys lose their edge over time, and then some exec in a board room is faced with -10% YoY loss without any truthful VPs around the table.

[+] mjburgess|2 years ago|reply
I go back and forth on how to interpret people like Chris (and likewise, myself).

I've been in this situation, and it's psychologically disorienting: I seem to be right, but am I? Are the people around me really as incompetent as they seem, as totally disinterested in learning from their colleagues, etc. etc.

Then a few years after leaving, you look back and: oh, all those people have been fired or left; they still cannot do X as an org; the agility and competence of teams I then joined really does exist etc.

So, on the one hand, it's true that there's a sort of arrogance, political incompetence, and inability to cope with environments with a pathological practice culture --- but, on the other hand, maybe that's the right reaction?

Maybe the institution is undergoing a pathological cultural period, and maybe talented, considered, passionate people should be driven mad by it. Maybe the people who arent driven mad by it are either irrelevant to productivity/growth, or worse, net deadweights.

Who knows? That's what makes being in this environment such a psychodrama -- is the situation really this bad, or am I over-reacting? "Everyone tells me...."

[+] chriskrycho|2 years ago|reply
A bit of clarification from some of the nuance that (reasonably) had to get cut from the episode for time:

- Propose a 5-year-plan: yeah, “lol” is about right. It was never going to win any hearts or minds. It was also our least favorite plan. But it was also the only one we felt we could actually bring to our executive leadership given what they were telling us prior to that, which was basically “Even for a migration we are asking you for, do not slow down product iteration velocity at all.” How do you do that? Welllll…

- I’m not really sure what you’re referring to about leading the incident with blame instead of leadership. It’s possible something got lost in translation with the amount we cut, but I actually aimed very much to do the opposite. We didn’t blame the people who lowered the memory thresholds, the people who typo’d a bad value in YAML, or anyone else. I just insisted that we actually solve the root issues instead of leaving them to fester for the next poor person who happened to be around when it inevitably blew up again.

- “Speaking about problems more than solving problems”: really not sure what you mean here. I didn’t spend a ton of time bragging about what I had pulled off on the show because wow would that ever have been in poor taste, but… I did pretty well in the problems I solved there.

- Lack of relationship building: yeah, I called that out on the episode! It was my weakest area. I had a really good rapport with engineers, and failed pretty miserably at building cachet with management, especially with management above me.

I wouldn’t say in the end that it was just down to not knowing how to perform in the environment, though. Some of it was also a choice, in the end, not to perform in ways that I could see would be successful in the environment but which I simply did not believe in. I think a lot of the engineers I respect most are like this: can and will do the political dance for something they believe in… but not for things they don’t believe in.

[+] cbb330|2 years ago|reply
I work at LinkedIn now. Chris’s role and the podcast describes ember and front end web dev. I think the LoC and build he’s referring to could be voyager-web. Our monolith flagship web app. And, there’s more systems at LinkedIn with MM lines of code and long builds example is the mid tiers, the offline data stacks, the metrics systems, KafkaKafkaKafka.

Unfortunately, 17min to build is pretty good. 17min without a transient infra failure is very good.

Anyways, AMA

[+] compacct27|2 years ago|reply
The big rewrite. Risky even on manageable codebases, and the leftovers never seem to fully go away. Who wants to score points on re-writing a tucked-away settings page several years into these?

I've seen so many attempts at these you'd think we'd have a framework for rewriting codebases. We don't. Automated codemods require a consistency that few adhered to. The code patterns evolved so much over time, it's like looking at tree rings.

We're basically putting code in boxes, re-arranging the boxes, and rightfully saying some arrangements are more efficient. How come we haven't found a better way? Automations operate at the code level, but not at the box level.

[+] LightFog|2 years ago|reply
Oh boy, been there. This is Conway’s law in action - they are about to go and create the same code soup again - because the org won’t have changed. Having been in the same boat my lesson is positive engineering initiatives can only come top down - through some very senior champion. You can’t change an organisation from the bottom up - and it’s the organisation that produces the code base.
[+] joshhart|2 years ago|reply
I spent 12 years at LinkedIn. Sadly, it's not even close to the engineering org it used to be. The era where Kevin Scott led engineering was a really good one in comparison.
[+] MisterPea|2 years ago|reply
Every engineer of a company of this size says the same exact thing.

IMO more about growth of the engineering teams that lead to any specific culture deteriorating.

[+] hasty_pudding|2 years ago|reply
Ryan Rolansky said LinkedIn is essentially feature complete.
[+] jll29|2 years ago|reply
MM lines of JavaScript! That is bloat incarnate.

I've been thinking of re-implementing something like LI, or rather implement my own contact database (without the "FB" features, please).

The only problem would be how to make my contacts migrate over in bulk.

Apart from the bloat, the main problem of Microsoft LinkedIn is that it does not let you export your contacts' infos, which really is a must-have feature of a contact platform.

[+] soarerz|2 years ago|reply
> Apart from the bloat, the main problem of Microsoft LinkedIn is that it does not let you export your contacts' infos, which really is a must-have feature of a contact platform.

Platform lock in is certainly intended (even though it sucks for users)

[+] game_the0ry|2 years ago|reply
Call me crazy but I did not think that was a surprise. I worked at big bank that had consumer-facing JS web app with 6MM lines of code, though now I am starting to think that that number is false, given the feedback in this thread.
[+] rm_-rf_slash|2 years ago|reply
Have you tried grabbing JSON directly from the web server responses to the browser to get your contacts?

I haven’t tried this on LI but it’s my dirty trick for exporting conference attendee lists when they’re made public on websites. YMMV.

[+] bobterryo|2 years ago|reply
I was really impressed with how open Chris Krycho was about his struggles without playing the blame game. CoRecursive is one of my favorite podcasts because it explores the complex context behind the code.
[+] Lyngbakr|2 years ago|reply
Agreed. And I also think Adam is a great host. He asks good questions and lets the guest talk.
[+] yellow_lead|2 years ago|reply
He seems like the kind of guy I'd like to work with, personally.
[+] gedy|2 years ago|reply
This sounds like one of those "soft leadership" roles, that almost always suck because you're "responsible" for something, but have little or no authority over the rest of the org. The real tech leadership (if any) are MIA or no longer in touch with the actual problems, even if they've been their forever or are the "system experts"

Been there, done that, no thanks.

[+] willcipriano|2 years ago|reply
> My previous company had what I thought at the time was a pretty decent sized app, like 150,000 lines of code. And the LinkedIn front end, when I got there had 2 million, it’s just like, I’m, did you say million? Uh, that’s, that’s a lot.

I'd love to see what makes it so large. That's like 1/15th the size of Linux.

[+] dilyevsky|2 years ago|reply
Have you ever worked on projects (particularly java suffers from this) where there is generational bloat because most people have very narrow understanding of the codebase so they can only add things and not remove/refactor without breaking it? That’s how
[+] diarrhea|2 years ago|reply
The site makes me sad whenever I visit.

On my beefy desktop, load times for the home page are at around three seconds. On my feeble (but still enormously expensive: a Microsoft Surface) laptop, it’s well in excess of 10 seconds. Both tests run on the very same network, so my conclusion is that load times are CPU bound, aka there’s huge amounts of JavaScript running, for one reason or another.

[+] apimade|2 years ago|reply
Including boilerplate code? ASP and JS is far more verbose in those areas.
[+] tayo42|2 years ago|reply
That quote is why you can't take anything seriously on the internet about running software or software engineering seriously.

There are to many people that just don't know what they don't know and act they have things solved.

I think the software world has like two bubbles of people around the scale they work with and they don't understand each other or run into each other, except in interviews and on the internet.

[+] li_engineer|2 years ago|reply
I am an engineer at LinkedIn and have been here for a long time. Haven't worked with CK and work on areas unrelated to Finger Gun project. But I agree with the general sentiment of feeling frustrated. Everything is now happening top-down. Execs want AI thrown everywhere even where things don't make sense. Engineers don't have the room to give feedback that they used to. Leaders want everything delivered asap. We are creating lot of tech debt every day and not maintaining any balance between speed and quality. Features are being implemented with missing edge cases and they say we will fix it later but no one has time or motivation to fix it. The execs only know we launched a feature, nobody tells them that the feature is too broken to actually be usable. There's a lot of fear of being managed out for performance so people are sucking it up and implementing what is asked with urgency. No one has time or motivation to fix bugs or clean up the code.
[+] callamdelaney|2 years ago|reply
It's no surprise that LinkedIn's codebase is a mess if you try to use the website, it's a complete nightmare. Say I see an interesting post, I click on the person who wrote it to learn more about them, then I press back to go back to the post and continue reading it - now the feed refreshed and I've lost the post.

Say I want to scroll back through messages I've received, I do that, the entire webpage starts lagging and takes 10-15 seconds to register inputs.

Why do I get 30 fake notifications? They are literally not real, made up rubbish to force interactions from people - it's disingenuous. The recommendation algorithm is also completely terrible.

[+] mkl95|2 years ago|reply
> That's 20 times almost the size of my, how, how do you even build that? How long to build? Oh, 17 minutes for a new build. That's, that's not bad. We should probably make that better, but that’s not bad.

Am I spoiled for thinking 17 minutes is too much, even if it's 2 million LOC? I guess LinkedIn can afford it

[+] ryanjshaw|2 years ago|reply
It depends on what those LOC are. Is it HTML? Is it JS? It it a compiled language? How much of it is generated?

Aside: I don't understand the obsession people have with throwing out LOC. It's a completely meaningless number unless you're familiar with the codebase, in which case you don't need to be told the stats.

[+] liampulles|2 years ago|reply
It is not too much or too little. The real question is whether the value of X% speedup worth Y hours of developer time to make the improvement. Relative to other work that could be done.

Answer could well be yes since build time impacts everyone, but also have to factor in complexity of the CI system they use.

[+] rkangel|2 years ago|reply
The more important number for a codebase that size is almost always "what's the incremental build time for whatever I work on". It's nice to have a "build from scratch" be fast, but almost always you can leave it for two hours to do the first build while you're in a meeting or something. More important is that when you make a change, the incremental build is fast.

This is why build systems like buck/pants/bazel are good in larger codebases. You can trust your incremental builds in a way that you can't in any build system that isn't hermetic.

[+] esprehn|2 years ago|reply
Yeah, they could definitely optimize it quite a lot further. With investment in code structure and caching I bet they could get below 5min at that scale. But also for edit/refresh that should be on the order of a second or two, never minutes.
[+] padjo|2 years ago|reply
No shade on ember as a framework but LinkedIn’s decision to use Ember always baffled me. As far as I remember they switched to it long after it was clear react had “won”.
[+] queuebert|2 years ago|reply
Since when is LinkedIn is fast paced? It hasn't substantially changed in ten years.
[+] maximinus_thrax|2 years ago|reply
Although I do listen to Adam Gordon Bell's podcasts from time to time, I'm not sure if I should invest 40something minutes into hearing about why some rando left BigCorp. This may be an unpopular opinion, but isn't unprofessional to spill the beans about your past employment like this? As an external observer and hiring manager, I wouldn't touch this person with a 10-foot pole if they're in the hiring pool.

I personally find this type of content to be mundane, basic and useless. This is the type of stuff most people run into if their engineering career spans more than a couple of years, touches large organizations, they can't handle conflicts and have the luxury of being more affluent than other employees. I don't understand why someone would go out of their way to share something like this.

[+] uptownfunk|2 years ago|reply
In my career I have learned to listen more carefully to engineers. Sometimes it is not about saying yes and no to them but listening to what they are trying to say. Sometimes if you ask a few questions they learned something about the business that makes their case come apart. Sometimes if you listen to them you find a good idea and it opens up a whole new area of the product. So I think it’s a false dichotomy and you have to be good at listening and asking the right questions to get your point across.
[+] swader999|2 years ago|reply
Really feels like mission impossible to do this kind of role remotely.
[+] elbear|2 years ago|reply
The migration from Ember to React sounds like a good case for a technology a previous client of mine created.

He created a language called Unhack which allowed you to do a kind of search-and-replace but at the level of the AST. You would use this language to specify a pattern in the original code, then another pattern that would be the replacement. Unfortunately, I don't know if it still exists, as I stopped working with that client a couple of years ago.