top | item 32462725

Chrome was delivered without any sprints at all (2021)

355 points| luu | 3 years ago |twitter.com | reply

250 comments

order
[+] _gabe_|3 years ago|reply
> I mean even at Google (on a different team) I was a "technical lead" in my 20s, and let me tell you, I had noooo business leading anything technical of any importance. But this is very common! We would never accept this in other fields. Would you live in a house built entirely by junior carpenters in their late 20s who built one or two houses that barely stood up? Would you drive cars designed and built by junior engineers?

I find this kind of funny, because this is what happens right? I was under the assumption that architects typically design the building plans and do all the engineering, and a construction crew (which can consist of people mainly in their 20s) will build those plans under the supervision of the lead engineers/architects.

So, in the same way that many senior software engineers don't write much code, don't architects/civil engineers typically refrain from using power tools to build the actual building? If this is the case, then software engineering is very akin to other engineering disciplines in this regard.

I feel like the author of this tweet is conflating craftsmen with senior leads. A craftsmen is somebody I would expect to have been working with the medium for 10+ years, and continues honing their craft throughout the years. Whereas engineers and architects are typically more concerned with the abstract ideas and overall outcome. An engineer/architect can be a craftsman, but I don't believe they need to be synonymous.

[+] suzzer99|3 years ago|reply
A software architect who doesn't code anymore is worse than useless in all of my experience. They're flimflam salespeople who build PowerPoints full of buzzwords that aren't even necessary, and then delegate everything to developers who actually code to try to build their Frankenstein contraptions.

"But we have to have websockets, it's in the presentation!"

[+] throwaway892238|3 years ago|reply
> I was under the assumption that architects typically design the building plans and do all the engineering, and a construction crew (which can consist of people mainly in their 20s) will build those plans under the supervision of the lead engineers/architects.

Draftsmen can also draw up the plans rather than an architect. There are different educational requirements as well as trade requirements before each can begin practicing said trade. (You can't just call yourself an architect just because your last employer gave you that title! both require professional licenses too.)

Plans are approved by a city official and permits are given. A contractor is hired to build based on the plans, and they hire subcontractors. The result often has to change due to unforeseen circumstances, and usually those changes don't result in a redesign. If the contractor sucks balls, the result will be a tire fire unfit for disaster relief housing. If the contractor is good, nearly everything will go as expected, it will be delivered on time and on budget. In between those, a lot of shit gets swept under the rug.

[+] seadan83|3 years ago|reply
I think the issue is that a lot of those "high level" decisions are also actually made at the code level. It is more analogous to an architect being a city planner, and coders are creating entire buildings (and not just installing the dry wall, but electrical, layout, elevators, plumbing, earthquake stability, etc)
[+] nso95|3 years ago|reply
Well an early career civil engineer isn't building the actual structures either
[+] aboodman|3 years ago|reply
Not at all. Every construction project has a site lead, usually in their 30s or 40s who has decades of experience.
[+] majormajor|3 years ago|reply
This thread really seems to be burying the lede which isn't just "we didn't have crunch" but the more specific claim that it was engineers with at least a decade of experience having deep technical involvement that made the difference.
[+] mbrodersen|3 years ago|reply
I have delivered successful software for 30+ years without sprints or any other ceremonies. I simply keep a prioritised “to do” list and only work on what is the #1 priority. I do it when managing development teams and when working as part of a team. It’s simple. It works. And when asked to change priorities, it is clear from the priority list what the consequences will be. And usually people will change their minds when they can easily see the consequences. It also means that you never need to say no. You simply add the new task to the priority list and negotiate whether it moves up/down relative to other tasks.
[+] pas|3 years ago|reply
when asked do you give estimates? how many people can ask you to move something up/down? is your list public to others?

does it happen that the #1 thing is blocked by some dependency? (do you simply move to #2?)

how do you manage a team? (do they work on your list or they have theirs?)

[+] United857|3 years ago|reply
Unlike IE, Chrome initially built off WebKit, so a lot of the work in writing a renderer was already done. Obviously a lot of work with V8, multiprocess IPC, etc. still went into the effort but still easier than starting from scratch like what IE did.
[+] olliej|3 years ago|reply
While this is true, and I'm generally fairly anti-chrome, this does disregard a huge amount of very real, and very hard engineering work.

I think V8 was not actually that complicated - a lot of the "deep" computer science they originally wrote and talked about in marketing material was from the 80s, and things like the hidden classes were picked up almost immediately by JSC at least (as in just a few weeks of non-crunch time). The original JITs for V8 and subsequently JSC were essentially template JITs, i.e not particularly complicated (for JSC the big issue was having to support x86, x86_64, and Thumb2, and managing the security implications on iOS). Even the GC imo wasn't super impressive.

The primary reason that Spidermonkey and JSC didn't have the JIT or hidden shape concepts wasn't engineering complexity, but simply that at the time the apparently important perf problems were with other parts of JS and the DOM - so huge numbers of objects (the argument for generational GC), and non-dom property access (which recall was not optimized in V8 at the time) were just not considered that important compared to other parts of JS and the browser.

But when you get to WebCore and the browser the Chrome folk did a lot of very hard work, the process separation was also conceptually simple, but practically a huge amount of very complex work. You can contrast the difference in time it took to get JS engines performance up to the same order as V8 vs the time it took any degree of process separation in WebKit and Gecko. The first multiprocess Safari was just two processes, the app, and the render process - just getting WebKit going with a single separate render process was a lot of work, it was at least two full releases before true multiprocess was a thing in WebKit (WebKit's multiprocess model makes the process separation part of the engine, vs. the blink model of having the app be responsible).

Then for sandboxing, WebKit obviously got to leverage the Mac+iOS sandboxing that was built into the system. The chrome folk had to build an entire sandboxing system for windows, without kernel support, or in fact any support. Which was another gigantic amount of work.

So kudos to the chrome team, they did a lot of work, and a lot of it was very hard, and shouldn't be reduced or dismissed.

[+] ubercore|3 years ago|reply
IE started with Mosaic, no?
[+] throwfh80h82|3 years ago|reply
I'm not advocating for death marches, but a small project being built in a company that has a money machine printing billions of dollars a quarter isn't reflective of most environments.
[+] nojvek|3 years ago|reply
Microsoft had a machine printing billions of dollars but their environment was quite different.

I was in IE team on dev tools. We shipped every 6 months, while chrome shipped every month. In the last 2 months of 6 months release, we had feature code freeze. Even small bug fixes weren’t allowed. First month was planning. Our dev pace was slower than Chrome. Chrome shipped a lot more in 6 months than IE did.

Chrome had an insane testing suite that tested pixel perfect rendering and Perf benchmarks. Doing this on every pull request is just amazing.

IE had some work to do. I was surprised to learn that the E2E test results were hosted on some engineer’s dev machine in his office. We used to run tests on our machine before committing to source depot. IE CI/CD had a lot to desire.

My takeaway from IE vs Chrome was that Chrome had good leadership, they highly invested in all kinds of tooking to encode their security, UX, performance bars.

IE leadership was sometimes far away from the user experience. They didn’t care about the little big things.

[+] ahahahahah|3 years ago|reply
The twitter thread was literally a reply to a story of a company that had a money printing machine and still required a terrible death march to ship a browser.
[+] jmyeet|3 years ago|reply
> I mean even at Google (on a different team) I was a "technical lead" in my 20s, and let me tell you, I had noooo business leading anything technical of any importance. But this is very common!

So these big tech companies have a caste system. And no I don't mean the Indian caste system, which obviously has its own controversies. The caste system is really a form of social proof.

Did you go to MIT, Stanford, UW, Waterloo or CMU? Ok, you're in the club. You can join TI (Technical Infrastructure). Out of college you'll be L5 in 2-3 years (the same level an external hire with 10 years of experience will have). You will find yourself on the better projects with more promotion prospects.

This kind of premature promotion is to find the 1 in 20 of these people who are truly talented enough to continue getting promoted to L6-8+.

[+] google234123|3 years ago|reply
Is TI really regarded better than search? Do they typically promote faster?
[+] shadowgovt|3 years ago|reply
Sprints are only really necessary when there's a deadline to hit.

For the longest time, one of the advantages Google has had as a software house is that when you're an industry leader, deadlines are soft because all you're worried about is somebody playing catch up, not your need to catch up to somebody else. As the company has grown in size and scale, calcified a bit, and branched out into spaces where they aren't the leader, the culture around this sort of thing has changed. They are not, for example, the leader in Cloud, and the life of a Cloud SWE is markedly different than the life of an ads or artificial intelligence research SWE.

[+] vlovich123|3 years ago|reply
Sprints are just a formal way for management to check in that the team isn’t down a weird rabbit hole / getting them to do regular check ins with the team for visibility.

It’s independent of deadlines.

[+] telltruth|3 years ago|reply
Sprints were actually invented for team and culture that is below average. The goal is to keep putting pressure while pretending not to do so. It works well on boring projects or subpar teams. If you did your recruiting right and didn’t end up in pointless projects then you don’t need sprint bullshit. A high powered team is typically self-motivated and can get things done without scrum master asking them what they were upto.
[+] seadan83|3 years ago|reply
If you replan at each sprint iteration, how does that help ensure a multi-sprint deadline is going to be achieved?

My impression was the sprint is the deadline, so it emphasizes to not have deadlines in favor of small incrementalism. Hence, if there is an actual deadline, sprints I think are exactly the wrong tool to help

[+] thrwy_918|3 years ago|reply
The author is using the word a bit differently, but the fact that "sprint" has been normalized as a unit of work for software development, and developers are expected to be in a "sprint" more or less at all times, has always been a source of the deepest absurdity to me.

A "sprint" is, almost by definition, a pace that's sustainable only for short periods. The fact that developers are expected to perform sprint after sprint endlessly, to view "sprint" as the default baseline pace, seems a ludicrous abuse of language.

[+] crazygringo|3 years ago|reply
I think you're being overly literal with "sprint".

And no, developers are not expected to be "in a sprint more or less at all times".

To the contrary -- in between each sprint, there's a review, a retrospective, often a weekend, then regrouping and feature analysis, planning poker, decision on which features to impement, and getting it up on the board.

That's the whole point. Instead of a long marathon without revision/reassessment/regrouping, it's a series of short sprints interspersed with revision/reassessment/regrouping. The metaphor here is the distance traveled, not the speed. Sprints are shorter distances, that's all.

Nobody has ever intended "sprint" to mean that engineers are going faster than in a marathon, anymore than "waterfall" means engineers are all tumbling down white water to their deaths at the bottom. :)

[+] throwaway821909|3 years ago|reply
I think the language is accurate and that's the problem (or not depending on your perspective) - sprints are all or nothing, and a way to create an artificial sense of urgency. "It's the last day of the sprint, better work late/cut corners because 99% done is failure". And then if you do your reward is one more work item next sprint, to ensure everyone is working as hard as possible all the time.
[+] benreesman|3 years ago|reply
I've always chuckled at the nomenclature that's crept in around software development. You've got "agile", "sprint", "scrum", "extreme programming", "sustainable pace", and probably other athletic metaphors I'm forgetting. There are various kinds of "stories", often combined into "epics", "retrospectives", "personas" and other literary metaphors that I'm likely forgetting.

"Track Star" and "Literature Major" are not stereotypes that I recall from my peers getting started in software, back then I didn't know many other computer nerds who had done more sports than my half-hearted JV swim and football, which is pretty weak. We for the most part had all been spending way too much time in front of the computer to be truly serious about athletics and literature.

But I'm old, and times change (which is a good thing).

Of course it would also be darkly funny if a bunch of literate athletes invaded the software business when it became lucrative/high-status and immediately started fogging up the windshield with a bunch of vague metaphors to obscure their lack of technicality. Who knows.

[+] throwaway892238|3 years ago|reply
It is a poor choice of word. But what's interesting to me is nobody notices the end and beginning of sprinting requires rest, regrouping, planning, getting ready. You can't just stop sprinting and start sprinting the next day like you're ready for a brand new race. Sprints should require reflection, a re-set, getting prepared. Instead it's just "the next 2 weeks of the same bullshit", without proper preparation, without clearing the schedules to allow for streamlined work.

When sprinting, any bump in the road can make you stumble, losing the race. We need more focus on clearing the bumps out of the road before a sprint.

[+] slim|3 years ago|reply
if memory does not fail me, original sprint was 3 weeks long followed by "spare" time till the end of the month (mostly meetings, planning, review, etc..)
[+] mdemare|3 years ago|reply
Maybe “laps” would have been a better word than “sprints”?
[+] butlerm|3 years ago|reply
It certainly implies developers recovering for several days between sprints.
[+] paulryanrogers|3 years ago|reply
Was the word 'sprint' chosen to describe a never ending crunch? Or just to roughly imply quick forward motion. Unlike waterfall that often slows for large portions of time until the next sudden and irreversible release?

Perhaps it would've been more accurate to use something like 'unit' in education. Which is just a subdivision of time and effort roughly equal to a week or two.

[+] dboreham|3 years ago|reply
Absurdity abounds in the field, in my lengthy experience.
[+] gravypod|3 years ago|reply
To me it is clear that Age != Experience. Early in my career I have worked with many people, while being much older, were not as emotionally intelligent as I was. I have also worked with many people who were younger than I was who were more emotionally intelligent. That is what leadership mainly boils down to.

Also, technical decisions should not come from the top. A leader should look at what something puts together and say "yea you covered your bases, go for it."

My current TL is older than me by a big gap but my experience on many technical things out-ways theirs and so they defer to me. Their experience on the social side of things is far better than mine and I always defer to them for planning and comms advice.

Basically: Age is a very poor proxy for how competent a leader is at running a team, making technical decisions, and many other factors that come to building large products. Selecting by age and putting the oldest person in change of a project is a horrible idea. Case in point: most governments are gerontocracy. How confident in the wise decision making powers of our political leadership?

[+] koala_man|3 years ago|reply
Sprint here refers to crunch time, and not agile development.
[+] Barrin92|3 years ago|reply
>"I mean even at Google (on a different team) I was a "technical lead" in my 20s, and let me tell you, I had noooo business leading anything technical of any importance. But this is very common! We would never accept this in other fields. Would you live in a house built entirely by junior carpenters in their late 20s who built one or two houses that barely stood up? Would you drive cars designed and built by junior engineers?"

one of the strangest and most baffling things about the entire industry tbh. Like, would you ever expect a 25 year old guy to command a spaceship? Yet in software you have these weekly "I'm 40, is my life over" posts. In most disciplines people correctly acknowledge that there's a sweet spot of skill and experience that overlaps somewhere in your late 30s, 40s or even 50s, yet in software very often we recreate Lord of the Flies, leading to chaotic project management.

[+] colechristensen|3 years ago|reply
Circa Apollo 11 the average age of a NASA engineer was 28. In 2009 it was 47.

There is this constant infantilization of people where once you’re an adult you don’t think anyone younger than you is capable of anything. It’s a big problem that this has been creeping into how we treat people like children at continually older ages.

Nobody knows how to do things when they start doing them, and they won’t until they do start. Delaying this to an older age makes this harder not easier.

[+] endtime|3 years ago|reply
Being a "tech lead" at Google is nothing like commanding a spaceship. It's more like being partially responsible for a team of 3-5 mid-20s engineers building the dashboard and reporting for space shuttle wind tunnel test results (or whatever they do with space shuttles).

Personally, I was a tech lead at Google pretty consistently from the ages of 26-35. I got better at it, and responsible for more, over time. It was a good learning experience for me and even when I was inexperienced at it, I was saving someone else some time.

[+] Uptrenda|3 years ago|reply
I interviewed at a YC startup recently and the founder was making ageist jokes to me because I was older than 25. Tech is literally insane regarding age. The assumption seems to be that once you get a single grey hair you can't use a computer any more? But I can tell you when I was writing code at like 18 or 20 everything I did was unsalvageable garbage.

It actually amazes me how much goes into writing quality code that is usable in production. I would be very surprised if any junior engineers could do it without messing up a bunch of stuff (which I certainly wouldn't fault them for as they're still learning!) You would need a good mentor if you wanted to avoid that I guess.

[+] YZF|3 years ago|reply
In the IDF (Israel's military) due to mandatory conscription there are plenty of young people in various leadership capacities. So yeah, you can be 19+, an officer, leading people into combat. Or flying a jet/cargo plane/helicopter. Or leading a software project.

That said, there are older people around in more senior roles and the way leaders act there is very structured but leaders are also expected to improvise when necessary.

It's somewhat amusing when you're dealing with those guys on some sort of technical collaboration. You might have a team from some big tech and then a bunch of "kids" in the room.

[+] vlovich123|3 years ago|reply
There are 20 year olds who demonstrate fine leadership skills and maturity. There are plenty of 40 year olds who do not. Find the best people you can regardless of age.

Also, often times the only way to get that experience in the first place is to be put into the positions of leadership to develop your skills.

[+] boxfire|3 years ago|reply
Just wanted to point out at JPL there are many 25-year-old people commanding spaceships, rovers, and landers. Sometimes room-fulls of them. Not a great example.
[+] zerr|3 years ago|reply
Off-topic: does 9-to-5 mean that 1-hour lunch break is considered as working time? How common is that?
[+] olliej|3 years ago|reply
I really don't understand the "sprints" approach to development - I've never worked on any project where it makes sense, either the things I've been working on take less time than a "sprint" or they take longer, and that's for any sprint length you can produce.

Real software requires some degree of planning, and sprints seem to be more an attempt to avoid that planning. I don't mean to the level of gantt chart hell (I've experienced that as well).

Sprints, at least as they actually occur in the real world, seem to actively harm any large scale projects, and increase the overhead for long term projects if you can get them to fit.

[+] rr808|3 years ago|reply
Before Scrum we had 3x 1-hour team meetings a week to talk about what we're on and what we might need help with. After we went "agile" we moved to having 3x 1-hour team meetings a week with the same. Power to the developers. :)
[+] michelb|3 years ago|reply
I think they wanted the best people on it and accept whatever they wanted/needed? I mean it is one of their most important products to dominate the web.
[+] babyshake|3 years ago|reply
A sprint means to go as fast as you possibly can, and is associated with exertion to the point of exhaustion. There's a reason managers love the word sprint.
[+] throwaway892238|3 years ago|reply
> Would you live in a house built entirely by junior carpenters in their late 20s who built one or two houses that barely stood up?

Well carpenters don't build houses by themselves. You might be talking about a framer, whose job is definitely important, but not really more important than most other of the trades needed to build a house. But actually by their late 20s a framer can have a decade of experience, more than enough to become a journeyman and frame with one or two assistants. It depends on when they apprenticed and became journeymen, where they learned the trade, and what their contracting and business ethics are.

> Would you drive cars designed and built by junior engineers?

Designers don't really build cars, but junior engineers are involved. Typically not leading themselves, though, as the handful of big car manufacturers can hire senior engineers to oversee them, and screwing up the launch of a car won't make it easy for you to work for one of the few competitors. People who make custom cars have probably been doing it for years as a hobby, and often aren't engineers at all.

In the non-software world, if you build something, there is a direct tangible result of that thing coming into existence. Like, if you build a cabinet, the worst case that happens is the shelves fall apart, so you don't need a whole lot of guarantees as a consumer. If you're building a car, there's a shit-ton of regulations (today, anyway) and potential lawsuits. If you're building a house, there's the code, there's 30 different specialized trades, all kinds of restrictions on who can do what and when and how, and a hundred government inspections.

But like in any trade, you can pass a test and still be a lying cheating piece of shit. (If you're a government contractor it's hard to tell if you're the former or incompetent) There is no magic spell or development lifecycle that stops shitty things from being shitty, or makes things automatically good. But sometimes there are regulations that enforce due diligence, and sometimes a contractor earns a good reputation through their results and word of mouth.

Sadly, in the software world, there are practically no regulations, no [serious] trade groups, no apprenticeships, no unions, no threat of class-actions, and very very rarely any substantive real-world consequence to shame a company into hiring competent workers. Bottom line: if you work somewhere and you are not satisfied or don't feel the work is challenging enough, get better and move on. The only way to escape monotony is to raise yourself up.