This may be an egregious question, but what does the Uber app actually do that makes it so big? It displays a map with some moving dots on them, allows you to pick a location, and asks the server for a route and price. Plus a bit of workflow for signup, credit card, reviews; not a huge number of screens.
I appreciate the difficulties of getting an app to fit in a fixed space having worked on a point of sale system that had to run on a MIPS device with 32MB of Flash, of which half was occupied by Windows CE. It's not really clear to me why Uber's app needs to be so large and complicated other than the availability of a large amount of money, high-powered engineers, and shiny new unproven technologies.
(The compact app fit in about eight megs in the end, plus another meg or so for the file which defined all the screens; it was written in the least trendy, unsexiest framework known to man, Windows MFC in C++)
I was part of the team in Amsterdam on this. Insane days. Had some great engineers we hired in AMS. We were moving the whole source code into a monorepo as well which was another crazy project. I think if we had not had some of the engineers we did at those exact times, we would have not gotten out of it.
Problem here with Swift came down to
- Apple not dogfooding its own tech . Hell they did not even help us or other major companies like AirBnB and LihnkedIn (We formed a workforce on this problem with them which then forced the download limit to be upped by apple)
- Engineers deciding to adopt a language which they thought was great (and turned out not to be at the time) because they did not do the right analysis upfront for such a significant project. 'Wow look at this shiny new cool tool, lets use it' (Oversimpiification, swift was more pleasant to write with and you could be more productive which was no doubt a factor here)
Also, while management at the time didnt care for the remote offices much, I think the AMS team played a critical role in the success here. With the recent layoffs that occurred, I heard that they were fired.
Another fun story, we were trying to bring down our build times on our CI builds which were 1hr+ while battling macstadium and their crappy infrastructure with tens/hundreds of patches incoming each day.
I was an iOS engineer at Airbnb when we were battling binary size increases caused by Swift. I rewrote the watch app twice; once to migrate to watchOS 2 and once to convert it all to ObjC (which saved ~9mb). While Swift does have its problems (and had a lot more back when we worked on this), there are many factors to why the Uber app (and Airbnb) is getting so large.
Airbnb is 75% executables, 9.5% assets and 8% localizations. This is pretty different from Uber which is 60.7% executables, 26% localizations and only 3.8% assets. While the Uber executables are large, ~10mb more than Airbnb (note: this is only the Uber rider app, while Airbnb has host+guest in one app), their localizations seem to be driving a lot of the app’s 300+mb install size.
Looking at how the localizations are laid out in the app bundle, there are thousands of *.strings files in separate bundles, seems to be one for each feature. Many of these files have comments to provide context for the strings. An example from the app is: "Various screens show this as a title in a snackbar message when there is no network connectivity available". Just stripping these comments out of the localization files would save about 17.6mb of the install size. Another side effect of splitting localizations into so many files is there are over 23k duplicated strings throughout these bundles.
While the Swift code size is part of the problem with Uber’s app size, it's not the only way to free up space. There's a lot of work that goes into bringing a large scale iOS app’s size down, and not many tools to help. I left my job at Airbnb recently to work on making it easier to visualize, monitor and reduce app size, please email me (noah [at] emergetools.com) if you might be interested!
I was the person that actually managed the binary size task force and proved that it was a business problem in the first place. Not all management didn’t care about AMS :)
The Amsterdam team 100% played a critical role in solving the issue (at least until apple bumped the limit). No way we would have stayed under it without you all.
Appreciate your insight, and I'm sorry to hear about the AMS team.
The OP indicates that a rewrite was necessary regardless. Even if Swift was not yet mature, do you think it was more worthwhile than rewriting in ObjC? Given that the goal was to "sustain mobile development at Uber for the next 5 years".
Secondly, I'm not a mobile developer so I don't have much context, and I'm wondering about your outlook on iOS development as a whole. Swift obviously had major problems but has supposedly gotten much better. You mention MacStadium's issues, but Apple build servers seem to be making progress with the recent AWS EC2 Mac reveal.
Is the iOS landscape turning a corner, or do you expect a new generation of bad developer experience?
> So said brilliant engineer in Amsterdam, built an annealing algorithm in the release build to reorder the optimization passes in such a way to as minimize size. This shaved a whooping 11 mbs off the total machine code size and bought us enough runway to keep development going.
> This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs
This would scare me too.
> I had privately had the “we need to stop” conversation with my director. He told me that if this project fails he might as well pack his bags. The same was true for his boss all the way up to the VP. There was no way out.
This brings up an interesting point. The author attributes some decisions in the thread to the "sunk cost fallacy". The business costs were sunk, but if the employees and management are too afraid for their job to make the "cut your losses" decision, the costs are not really sunk, at least not in the minds of the right people.
When you write code like shit, the language doesn't matter. What Uber did was write a monstrosity with too many engineers and too many features without considering that was beyond their ability to deliver reliably. The product decision making was also clearly deficient and didn't help.
Rewriting is neither a panacea nor a problem if you completely understand what you started with and what you will wind up with and be brutally honest. Trying to combine a rewrite with major feature changes is extremely hard to pull off (I've done it a couple times with iOS apps, not fun, but ours came out well) and adding a new language at the same time is probably beyond any team's ability. A rewrite for good reason can be done, but not with 100's of engineers and major features changes, all done in a massive hurry; that's a recipe for disaster.
Couple things always to mind whenever this article is referenced.
Joel's example is Mozilla, but Mozilla ultimately became more successful than Netscape (in terms of userbase with Firefox). Was the problem really the rewrite, or that a browser company was doomed anyway and the rewrite exacerbated it?
Also the article was published just a year before Microsoft pulled off what's maybe the most successful large rewrite and customer migration in software history: Windows/DOS to Windows NT (XP was where it became complete).
I've always wished for Joel's take on NT, especially since he was at Microsoft previously and it happened just after this article was posted. What's his opinion on its success when so many weren't?
He's right on, but maybe people don't actually know what to do on those cases
A full rewrite is not a Ship of Theseus thing. You're just going to reverse-engineer your own thing in another thing. Unless it is unusable (quality-wise or maybe the only current version was written in APL for Solaris, who knows) there's no need for that.
Can Swift talk to ObjC? Great. Then do that. Attack the critical areas first
Yes sometimes a whole new architecture is needed, but then maybe you needed to reevaluate your earlier procedures and/or hires.
Great article, thank you for sharing this. Makes a good case that the right thing to do, from a business sense, is usually not to do the full rewrite. Rather, you should do slow, un-sexy work of refactoring and cleaning up while adding new and maintaining current features.
The "curmudgeon" ending line in this story rubs me the wrong way. The ObjC people were right, basically, and it seems like the problem was essentially no one in the right place was going to drop Swift at that point.
Seems less like they weren't offering answers, so much as the answer wasn't "but we'll do some stuff and fix Swift!"
Though the 6 library limit and how that wasn't a gigantic red flag on the language is baffling.
I'm reminded of the story of Windows NT and how David Cutler commented at one point after the release that divorces and nervous breakdowns were tracked in the program metrics. I expect that's hyperbole, but there is a real cost to these death marches.
I couldn't find that specific quote, though, and being 25 years back it could be a false memory of who said it (I did read it somewhere, if you were on the team at the time, please speak up!). Here are a couple of illustrations[0][1].
> I expect that's hyperbole, but there is a real cost to these death marches.
I used to be a game developer at EA. In the middle of one particularly nightmarish 60-hour-weeks-for-months crunch, a twenty-something artist with no known pre-existing conditions spontaneously died in his sleep.
I find it fascinating that so much engineers (arguably « smart » people) are actively hurting themselves over a freaking job and acting as if it were just a fact of life.
If I could read a book of these stories I'd stop everything and read it now. Absolutely incredible. I love stories like these, but I also understand these hells are probably reserved for the special world of startups. The interesting takeaway from this for me is the question "Was the rewrite successful?".
As an engineer, I understand I'm a vehicle for success at a startup. Even if I ultimately get burnt out and quit. If I make important changes, that maybe only I could have thought of before I quit, it probably becomes a good investment to hire me.
Uber managed to hire enough of the "right" vehicles for success to get to the point where they shipped a platform rewrite at massive scale. By all high level objective metrics, this is probably success story.
The one metric I think could derail the projects success isn't really documented--what was the human cost to this effort? I bet you looking at a graph of turnover at Uber you could probably identify exactly when this project happened. I'm curious if the engineering turnover from such a massive effort was enough to offset the benefit.
The engineering org had doubled in sized almost every year prior ... Teams began shifting all their focus to bringing their features to the new app ... Apple’s recommend maximum number of libraries in a single binary was 6. We had 92 and counting ... But as the app kept growing. Soon we hit the cellar [sic] download limit (100 mb) for our universal binaries ... All this time the Swift code growth continued ... discovered that our compiled code size was growing a rate of 1.3 mb a week ... But the real problem was the growth curve. It was always eating our winnings back.
This is one of the main reasons why I have no intention of going back to BigTech any time soon. I am sure the author is an extremely capable engineer but he is obviously missing the elephant in the room: that the root problem is app growth and feature creep caused by uncontrolled growth of the eng. workforce (who are all incentivized to ship their code because it's good for promotion). As if app growth was some kind of god-given fate that can't be changed! But instead of fighting the real (organizational, political) issues within the company, they blame it on technical issues (Swift is not mature! 100 MB app size limit is too low! etc.). Incredible.
The larger community benefited from our learnings.
No. The larger community would have benefited if you had written a story about how someone in the org. stood up to management, fought the feature creep, the crazy app growth and doubling eng. each year, and brought the whole project back to sanity. But that didn't happen.
So my advice. Everything in Computer Science is a trade off. There is no universally superior language. Whatever you do, understand what the tradeoff are why you are making them. Don’t let it descend into a political war between opinionated factions.
No disrespect, but no choice / tradeoff in the world would have led to a satisfactory outcome if you have hundreds of engineers burying this app under an avalanche of code day after day. The tools / tech are alright. The company is not.
What you're ignoring is that some significant percentage of that "bloat" is what leads to more rides and more money. All those tweaks and a/b tests and features that solve problems for specific segments that most users won't ever see, getting shipped week after week, increasing the new user conversion rate by 0.5% here, reducing the rate of canceled trips by 0.3% there, it all adds up over time. Many of the features obviously fail too and don't improve anything they just add bloat, but you can't ever know for sure which features those will be in advance.
In aggregate, the goal is obviously that it leads to the company making more money than was lost by paying all those engineers to move mountains to get around all the issues they hit up against. In the case of Uber, now a $96 billion public company and worth nearly double their 2016 valuation, it's a little ridiculous to call it a failure.
You read this story and see an unmitigated disaster, but it's mostly just a picture of things working the way the company wants them to. The goal is to grow the company, not to have a nice, conceptually clean, tech-debt free codebase just for the sake of it.
While I think your tone is a bit more critical than it needs to be, as many very intelligent people have made the same mistake, I think the sentiment is correct. But 'standing up to management' is literally not possible if they don't want it to be. They get to make the decisions if they want to, more or less by definition of 'management'. He did say 'we need to bail' to his boss at one point.
But, essentially, your final paragraph's point is the bottom line. This was not fundamentally a tech problem, it was a too-many-cooks-in-the-kitchen problem. That's a business decision.
It's also interesting to look back at the internal politics of Uber at this time. [1]
There's also a bunch of big articles on VPs quitting during this time. It was peak Uber controversy with the sexual harassment scandals, Waymo stolen IP, TK resigning, Fowler, spying, backdoors, etc. Reading the headlines in the algolia search is like a telenovela.
It’s interesting to me that you came away from this thinking that I didn’t realize the eng size growth was the problem. I specifically said in the third tweet that hypergrowth was the problem. I even went on to say why it was a problem. The entire point of including the details about the org growth was specifically to draw you to that conclusion.
> So my advice. Everything in Computer Science is a trade off. There is no universally superior language. Whatever you do, understand what the tradeoff are why you are making them. Don’t let it descend into a political war between opinionated factions.
It was pretty obvious when Swift 1.0 launched that it should’ve been called Swift 1.0 beta 1. Not a knock against anyone who worked on the project, other than senior management at Apple.
Speaking for myself, I didn’t add any Swift to any codebase I worked on until after 3.0 shipped because it was pretty clear that the language just wasn’t quite ready yet.
I’m genuinely surprised no one ever thought to just email Chris Lattner or invite him out for a drink to say ‘hey, confidentially between you and me we’re thinking about rewriting our app in your new language. Are there any other apps with this order of magnitude of LOC written in Swift yet?’
Large Swift apps are easy to find; companies would invariably use it as a marketing point on their engineering blog. Even if not, people open up apps all the time, any large app that uses Swift would be found and discussed.
Making a huge decision and sticking with it -- blowing through all obstacles through workarounds and hacks -- because otherwise would be deemed to be a failure is the core issue here.
Uber's culture of "letting builders build" is fine, as long as teams have the ability to change things early and quickly. I think top-down decision making and monumental bloating of eng teams made things very difficult over time.
Yeah, I imagine this pressure from the top had something to do with the "90 hour weeks" cited in the thread...I've never worked in an org as large as Uber, but personally I have made some very bad engineering choices when in crunch mode, especially on the doubling-down-on-bad-decisions front.
When there's no room to come up for air and look at a project from a perspective other than "this is super broken and way behind schedule" my brain just fixates on the micro-steps that need to be done to get it across the finish line (whether or not the finish line is realistically anywhere within sight). It's work being done, but none of the exhausted people working on a project like that have any energy left to consider whether it's the right work.
Management had very different incentives while engineers had different motives. The two were not aligned, and TK culture was still very prominent back then (work hard, dont go home early, we are going to be a 1 trillion dollar company etc etc) and Thuan definitely helped him spread that culture. So definitely not great days but I think the engineers took away a lot from their time there. Lots of lessons learned
I started to write a whole big thing here because I found this whole thing pretty disgusting but it's not worth it.
> I had privately had the “we need to stop” conversation with my
director. He told me that if this project fails he might as well pack
his bags. The same was true for his boss all the way up to the VP. There
was no way out.
> With only a week left we decided eat the 8 figures and drop support for iOS 8.
I'll say this: If I were an Uber investor I would be pretty angry. And the story
just gets worse from there.
This is the story of some hackers who got in good at a big company with
lots of money and sturm und drang so they had no adult supervision
while they partied while pretending to be actual engineers. (That they
were pretending to themselves too doesn't make it any better.)
The writing was on the wall, but rather than own up and take
responsibility they double down and keep messing things up for months
and years.
> This is when the real brilliant engineers started to shine.
This is when they get pulled from other tasks to clean up your mess!
(cf. Broken window fallacy.) Those real brilliant engineers should be
doing other better things than dark rituals to appease the Swift compiler
gods.
Look, we all make mistakes. There's a blog post from an engineer at a
company I used to work for complaining about the pointless middleware I
wrote years earlier. (I feel ya bro, I told them at the time that it was
pointless.) Here's the thing: I got out of that place when I realized
that the folks running it weren't really competent. (And, sure enough,
they got pushed out by the investors and now the company is a "Wallstreet
darling". So... Yay?)
I've burned enough investor money in my career that I don't think it's
funny anymore. (I never really did think it was funny, I don't like
waste, but I was young. REDACTED was only my second official job, and in
hindsight it was really crazy. I mean, they built a skating half-pipe in
the office! At the time, everybody thought that was real cool.)
One company I worked for a few years ago, I joined right as a just such a ground-up rewrite of their core system was coming into place. Everyone in orientation was telling me "You'll be spending most of your working life dealing with <new project codename>" but they were actually training me on the existing system. About a month out of orientation there is a company wide. They canceled <new project codename> because it wasn't meeting it's technical specs.
It was also, coincidentally, the company where I most respected the key technical decision-makers, because they had the will to discontinue a product that a few customers already were using, because it wasn't meeting its criteria.
In some other thread someone who regularly attended meetups hosted by Uber in NY said this (paraphrasing): "You could rely on Uber coming up with the most complicated solution to problems they create for themselves. But the food was good" Pretty much sums up anything I have ever read about uber engineering.
Many engineering horror stories involve an entire system rewrite. My recommendation is to never rewrite a large system. Literally never. If the people proposing the rewrite aren’t capable of adding the features incrementally, they aren’t capable of rewriting the entire system to “easily” support those new features.
This 100%. From the tweets it sounds like the problem they were trying to solve was to make the app more modular so it would scale better to more devs. There were/are well established patterns for doing so in ObjectiveC. The Swift rewrite was not necessary and probably took a very difficult project and turned it into the death march that the OP describes.
They should have solved the architectural problem incrementally without adding a 1.0 language rewrite to the mix and then slowly rebuilt the app using Swift.
This is so obvious to anyone with experience it seems pretty evident that the decision making process there at the time was deeply flawed.
I shudder at the thought of hundreds or thousands of Uber engineers banging out a rewrite together. I've been through a number of rewrites, and every single time it always meant the same thing: get your resume ready and look for the exit. It's going to be a multi-year shitshow.
Binary size should have been treated like a limited resource: set a hard limit in the test suite, and let the engineers who commit code in the organization fight it out how to delete unused code when they create new one for a new feature.
I'm following George Hotz's tinygrad that is a CPU+GPU deep learning framework under 1000 lines of code with great interest where all engineers are trying to shave lines of code while maintaining readable code (it's like a game when you set rules):
[+] [-] pjc50|5 years ago|reply
I appreciate the difficulties of getting an app to fit in a fixed space having worked on a point of sale system that had to run on a MIPS device with 32MB of Flash, of which half was occupied by Windows CE. It's not really clear to me why Uber's app needs to be so large and complicated other than the availability of a large amount of money, high-powered engineers, and shiny new unproven technologies.
(The compact app fit in about eight megs in the end, plus another meg or so for the file which defined all the screens; it was written in the least trendy, unsexiest framework known to man, Windows MFC in C++)
[+] [-] nobunaga|5 years ago|reply
Problem here with Swift came down to
- Apple not dogfooding its own tech . Hell they did not even help us or other major companies like AirBnB and LihnkedIn (We formed a workforce on this problem with them which then forced the download limit to be upped by apple)
- Engineers deciding to adopt a language which they thought was great (and turned out not to be at the time) because they did not do the right analysis upfront for such a significant project. 'Wow look at this shiny new cool tool, lets use it' (Oversimpiification, swift was more pleasant to write with and you could be more productive which was no doubt a factor here)
Also, while management at the time didnt care for the remote offices much, I think the AMS team played a critical role in the success here. With the recent layoffs that occurred, I heard that they were fired.
Another fun story, we were trying to bring down our build times on our CI builds which were 1hr+ while battling macstadium and their crappy infrastructure with tens/hundreds of patches incoming each day.
[+] [-] sond813|5 years ago|reply
Airbnb is 75% executables, 9.5% assets and 8% localizations. This is pretty different from Uber which is 60.7% executables, 26% localizations and only 3.8% assets. While the Uber executables are large, ~10mb more than Airbnb (note: this is only the Uber rider app, while Airbnb has host+guest in one app), their localizations seem to be driving a lot of the app’s 300+mb install size.
Looking at how the localizations are laid out in the app bundle, there are thousands of *.strings files in separate bundles, seems to be one for each feature. Many of these files have comments to provide context for the strings. An example from the app is: "Various screens show this as a title in a snackbar message when there is no network connectivity available". Just stripping these comments out of the localization files would save about 17.6mb of the install size. Another side effect of splitting localizations into so many files is there are over 23k duplicated strings throughout these bundles.
While the Swift code size is part of the problem with Uber’s app size, it's not the only way to free up space. There's a lot of work that goes into bringing a large scale iOS app’s size down, and not many tools to help. I left my job at Airbnb recently to work on making it easier to visualize, monitor and reduce app size, please email me (noah [at] emergetools.com) if you might be interested!
[+] [-] cbrauchli|5 years ago|reply
The Amsterdam team 100% played a critical role in solving the issue (at least until apple bumped the limit). No way we would have stayed under it without you all.
[+] [-] nwsm|5 years ago|reply
The OP indicates that a rewrite was necessary regardless. Even if Swift was not yet mature, do you think it was more worthwhile than rewriting in ObjC? Given that the goal was to "sustain mobile development at Uber for the next 5 years".
Secondly, I'm not a mobile developer so I don't have much context, and I'm wondering about your outlook on iOS development as a whole. Swift obviously had major problems but has supposedly gotten much better. You mention MacStadium's issues, but Apple build servers seem to be making progress with the recent AWS EC2 Mac reveal.
Is the iOS landscape turning a corner, or do you expect a new generation of bad developer experience?
[+] [-] snarkypixel|5 years ago|reply
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] offtop5|5 years ago|reply
I hate objective C with a passion. It's like some archaic language of a bygone era forced to run on a cell phone
[+] [-] nwsm|5 years ago|reply
> So said brilliant engineer in Amsterdam, built an annealing algorithm in the release build to reorder the optimization passes in such a way to as minimize size. This shaved a whooping 11 mbs off the total machine code size and bought us enough runway to keep development going.
> This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs
This would scare me too.
> I had privately had the “we need to stop” conversation with my director. He told me that if this project fails he might as well pack his bags. The same was true for his boss all the way up to the VP. There was no way out.
This brings up an interesting point. The author attributes some decisions in the thread to the "sunk cost fallacy". The business costs were sunk, but if the employees and management are too afraid for their job to make the "cut your losses" decision, the costs are not really sunk, at least not in the minds of the right people.
[+] [-] staplung|5 years ago|reply
The story arc is almost always the same: starts out very promising and moves quickly, then at some point the wheels come off but everyone keeps going.
Anyway, Joel Spolsky nailed it 20 years ago: https://www.joelonsoftware.com/2000/04/06/things-you-should-...
[+] [-] coldcode|5 years ago|reply
Rewriting is neither a panacea nor a problem if you completely understand what you started with and what you will wind up with and be brutally honest. Trying to combine a rewrite with major feature changes is extremely hard to pull off (I've done it a couple times with iOS apps, not fun, but ours came out well) and adding a new language at the same time is probably beyond any team's ability. A rewrite for good reason can be done, but not with 100's of engineers and major features changes, all done in a massive hurry; that's a recipe for disaster.
[+] [-] trimbo|5 years ago|reply
Joel's example is Mozilla, but Mozilla ultimately became more successful than Netscape (in terms of userbase with Firefox). Was the problem really the rewrite, or that a browser company was doomed anyway and the rewrite exacerbated it?
Also the article was published just a year before Microsoft pulled off what's maybe the most successful large rewrite and customer migration in software history: Windows/DOS to Windows NT (XP was where it became complete).
I've always wished for Joel's take on NT, especially since he was at Microsoft previously and it happened just after this article was posted. What's his opinion on its success when so many weren't?
[+] [-] raverbashing|5 years ago|reply
A full rewrite is not a Ship of Theseus thing. You're just going to reverse-engineer your own thing in another thing. Unless it is unusable (quality-wise or maybe the only current version was written in APL for Solaris, who knows) there's no need for that.
Can Swift talk to ObjC? Great. Then do that. Attack the critical areas first
Yes sometimes a whole new architecture is needed, but then maybe you needed to reevaluate your earlier procedures and/or hires.
[+] [-] apozem|5 years ago|reply
[+] [-] XorNot|5 years ago|reply
Seems less like they weren't offering answers, so much as the answer wasn't "but we'll do some stuff and fix Swift!"
Though the 6 library limit and how that wasn't a gigantic red flag on the language is baffling.
[+] [-] 11thEarlOfMar|5 years ago|reply
I couldn't find that specific quote, though, and being 25 years back it could be a false memory of who said it (I did read it somewhere, if you were on the team at the time, please speak up!). Here are a couple of illustrations[0][1].
[0] http://read.pudn.com/downloads196/ebook/923902/ShowStopper!-...
[1] http://www.proudlyserving.com/archives/2007/09/dave_cutler_w...
[+] [-] munificent|5 years ago|reply
I used to be a game developer at EA. In the middle of one particularly nightmarish 60-hour-weeks-for-months crunch, a twenty-something artist with no known pre-existing conditions spontaneously died in his sleep.
[+] [-] Bayart|5 years ago|reply
[+] [-] saagarjha|5 years ago|reply
[+] [-] sixstringtheory|5 years ago|reply
[+] [-] georgiecasey|5 years ago|reply
[+] [-] alexggordon|5 years ago|reply
As an engineer, I understand I'm a vehicle for success at a startup. Even if I ultimately get burnt out and quit. If I make important changes, that maybe only I could have thought of before I quit, it probably becomes a good investment to hire me.
Uber managed to hire enough of the "right" vehicles for success to get to the point where they shipped a platform rewrite at massive scale. By all high level objective metrics, this is probably success story.
The one metric I think could derail the projects success isn't really documented--what was the human cost to this effort? I bet you looking at a graph of turnover at Uber you could probably identify exactly when this project happened. I'm curious if the engineering turnover from such a massive effort was enough to offset the benefit.
[+] [-] rsp1984|5 years ago|reply
This is one of the main reasons why I have no intention of going back to BigTech any time soon. I am sure the author is an extremely capable engineer but he is obviously missing the elephant in the room: that the root problem is app growth and feature creep caused by uncontrolled growth of the eng. workforce (who are all incentivized to ship their code because it's good for promotion). As if app growth was some kind of god-given fate that can't be changed! But instead of fighting the real (organizational, political) issues within the company, they blame it on technical issues (Swift is not mature! 100 MB app size limit is too low! etc.). Incredible.
The larger community benefited from our learnings.
No. The larger community would have benefited if you had written a story about how someone in the org. stood up to management, fought the feature creep, the crazy app growth and doubling eng. each year, and brought the whole project back to sanity. But that didn't happen.
So my advice. Everything in Computer Science is a trade off. There is no universally superior language. Whatever you do, understand what the tradeoff are why you are making them. Don’t let it descend into a political war between opinionated factions.
No disrespect, but no choice / tradeoff in the world would have led to a satisfactory outcome if you have hundreds of engineers burying this app under an avalanche of code day after day. The tools / tech are alright. The company is not.
[+] [-] cactus2093|5 years ago|reply
In aggregate, the goal is obviously that it leads to the company making more money than was lost by paying all those engineers to move mountains to get around all the issues they hit up against. In the case of Uber, now a $96 billion public company and worth nearly double their 2016 valuation, it's a little ridiculous to call it a failure.
You read this story and see an unmitigated disaster, but it's mostly just a picture of things working the way the company wants them to. The goal is to grow the company, not to have a nice, conceptually clean, tech-debt free codebase just for the sake of it.
[+] [-] rossdavidh|5 years ago|reply
But, essentially, your final paragraph's point is the bottom line. This was not fundamentally a tech problem, it was a too-many-cooks-in-the-kitchen problem. That's a business decision.
[+] [-] notional|5 years ago|reply
There's also a bunch of big articles on VPs quitting during this time. It was peak Uber controversy with the sexual harassment scandals, Waymo stolen IP, TK resigning, Fowler, spying, backdoors, etc. Reading the headlines in the algolia search is like a telenovela.
Reflecting on one very, very strange year at Uber (Susan Fowler) - https://news.ycombinator.com/item?id=13682022
I am an Uber survivor - https://medium.com/@amyvertino/my-name-is-not-amy-i-am-an-ub...
Uber Founder Travis Kalanick Resigns as C.E.O. - https://news.ycombinator.com/item?id=14600873
How Uber Used Secret “Greyball” Tool to Deceive Authorities Worldwide - https://news.ycombinator.com/item?id=13785564
Suicide of an Uber engineer: Widow blames job stress - https://news.ycombinator.com/item?id=14200486
Uber's Design Meltdown - https://news.ycombinator.com/item?id=11109272
Uber employees used the platform to stalk celebrities and their exes - https://news.ycombinator.com/item?id=13163895
[1] https://hn.algolia.com/?dateEnd=1512691200&dateRange=custom&...
[+] [-] StanTwinB|5 years ago|reply
[+] [-] tantalor|5 years ago|reply
[+] [-] epaga|5 years ago|reply
[+] [-] andrewprock|5 years ago|reply
[+] [-] pauljurczak|5 years ago|reply
[+] [-] flareback|5 years ago|reply
I really like this point.
[+] [-] aaronbrethorst|5 years ago|reply
Speaking for myself, I didn’t add any Swift to any codebase I worked on until after 3.0 shipped because it was pretty clear that the language just wasn’t quite ready yet.
I’m genuinely surprised no one ever thought to just email Chris Lattner or invite him out for a drink to say ‘hey, confidentially between you and me we’re thinking about rewriting our app in your new language. Are there any other apps with this order of magnitude of LOC written in Swift yet?’
[+] [-] saagarjha|5 years ago|reply
[+] [-] lxe|5 years ago|reply
Uber's culture of "letting builders build" is fine, as long as teams have the ability to change things early and quickly. I think top-down decision making and monumental bloating of eng teams made things very difficult over time.
[+] [-] macNchz|5 years ago|reply
When there's no room to come up for air and look at a project from a perspective other than "this is super broken and way behind schedule" my brain just fixates on the micro-steps that need to be done to get it across the finish line (whether or not the finish line is realistically anywhere within sight). It's work being done, but none of the exhausted people working on a project like that have any energy left to consider whether it's the right work.
[+] [-] nobunaga|5 years ago|reply
[+] [-] carapace|5 years ago|reply
> I had privately had the “we need to stop” conversation with my director. He told me that if this project fails he might as well pack his bags. The same was true for his boss all the way up to the VP. There was no way out.
> With only a week left we decided eat the 8 figures and drop support for iOS 8.
I'll say this: If I were an Uber investor I would be pretty angry. And the story just gets worse from there.
This is a story of gross incompetence. It's not an engineering disaster, it's a management disaster. This is classic "inmates running the asylum". ( https://www.goodreads.com/book/show/44098.The_Inmates_Are_Ru... )
This is the story of some hackers who got in good at a big company with lots of money and sturm und drang so they had no adult supervision while they partied while pretending to be actual engineers. (That they were pretending to themselves too doesn't make it any better.)
The writing was on the wall, but rather than own up and take responsibility they double down and keep messing things up for months and years.
> This is when the real brilliant engineers started to shine.
This is when they get pulled from other tasks to clean up your mess! (cf. Broken window fallacy.) Those real brilliant engineers should be doing other better things than dark rituals to appease the Swift compiler gods.
Look, we all make mistakes. There's a blog post from an engineer at a company I used to work for complaining about the pointless middleware I wrote years earlier. (I feel ya bro, I told them at the time that it was pointless.) Here's the thing: I got out of that place when I realized that the folks running it weren't really competent. (And, sure enough, they got pushed out by the investors and now the company is a "Wallstreet darling". So... Yay?)
I've burned enough investor money in my career that I don't think it's funny anymore. (I never really did think it was funny, I don't like waste, but I was young. REDACTED was only my second official job, and in hindsight it was really crazy. I mean, they built a skating half-pipe in the office! At the time, everybody thought that was real cool.)
[+] [-] Apocryphon|5 years ago|reply
https://www.reddit.com/r/iOSProgramming/comments/6upeu6/how_...
https://news.ycombinator.com/item?id=14903228
And their Android app patching Dalvik at runtime to deal with the huge number of Java methods:
https://www.facebook.com/notes/facebook-engineering/under-th...
And they're not the only ones known for bloated apps, either. Take Amazon's apps for instance:
https://news.ycombinator.com/item?id=14902875
[+] [-] mandevil|5 years ago|reply
It was also, coincidentally, the company where I most respected the key technical decision-makers, because they had the will to discontinue a product that a few customers already were using, because it wasn't meeting its criteria.
[+] [-] bfrog|5 years ago|reply
[+] [-] pavlov|5 years ago|reply
[+] [-] valzam|5 years ago|reply
[+] [-] Apocryphon|5 years ago|reply
LinkedIn adopted Swift back in 1.2. I would love to read some war stories of how things there have progressed over the years.
Source: https://www.youtube.com/watch?v=X9waDi787uo (14:15-15:30, 39:26-40:37)
At an interview with them a few years ago, I was told they had been moving towards rewriting the app in Objective-C.
[+] [-] georgiecasey|5 years ago|reply
[+] [-] _dh54|5 years ago|reply
[+] [-] wankerrific|5 years ago|reply
They should have solved the architectural problem incrementally without adding a 1.0 language rewrite to the mix and then slowly rebuilt the app using Swift.
This is so obvious to anyone with experience it seems pretty evident that the decision making process there at the time was deeply flawed.
[+] [-] deckard1|5 years ago|reply
[+] [-] mpweiher|5 years ago|reply
So easy with the absolutes :-)
[+] [-] xiphias2|5 years ago|reply
I'm following George Hotz's tinygrad that is a CPU+GPU deep learning framework under 1000 lines of code with great interest where all engineers are trying to shave lines of code while maintaining readable code (it's like a game when you set rules):
https://github.com/geohot/tinygrad
Here's the GPU ops part:
https://github.com/geohot/tinygrad/blob/master/tinygrad/ops_...