The summary of that comment is "we have to include a ton of stuff that will never be relevant to most users like payments APIs that only work in India."
This is why some global apps have different apps for different countries. It's a trade off. Would you rather have a single fat Uber app, or have to download Uber India when you arrive there?
It strikes me that this would be the perfect use case for loadable modules. The Uber app could download the payment module you need on the first use and leave the dozens of other APIs off your device. This could also significantly cut down on the number of updates (300MB downloads each time!) that the app needs, since NA or European users won't have to re-download the app because some Indian payment API changed.
Unfortunately the way Apple and Google set up their walled gardens makes this impossible. I guess Apple would prefer if the Uber app dropped all of that and just made everybody use ApplePay instead.
> And then you have the binary size bloat with Swift that OP takes about.
Hopefully, swift ABI stability will reduce that. The new bytecode stuff will help to reduce bloat, as well, but he notes that a lot of SDKs are used. In my experience, SDKs and dependencies often won’t work, compiled with bytecode. Hopefully, that’s changing.
That code repetition thing also happens when a lot of dependencies are used; which often reinvent the wheel. That just comes with the territory. It can be addressed by using highly granular dependencies, but that sort of flies in the face of why we use dependencies. One advantage that Uber has, is they are an 800-lb gorilla. They could contract for specific configurations of dependencies.
I’m not a big Uber user (but I’ve used it a few times). I think it’s a fairly well-done app, as a user.
Yeah that’s a great read, super interesting detail. You wonder what would have happened if Apple didn’t up the limit.
This post is next level though, deeep optimization. All of it is just increasing the ceiling though, there is some limit on number of features Uber can offer in one app.. and eventually that limit will be reached, doesn’t sound like they are willing to accept being over the limit either. Wonder what space saving techniques are left in the box?
I’m the founder of a YC company in the current batch focused on solving this exact problem! https://www.emergetools.com
We parse Obj-C and Swift runtime metadata to determine size contributions of individual types and functions in your app. We use this analysis to post PR comments with granular size diffs to help devs write smaller, better code.
I tried it out on the Uber app and immediately noticed a disproportionate impact from their code-gen dependency injection framework, Needle. The codegen is responsible for over 30k classes in the app binary, and contributes over 10mb! In general codegen is a common problem with Swift binary sizes, and the fewer reference types generated the better, it even helps with startup time!
"The Lyft app has hundreds of duplicate files, the largest consumer of space is a single asset catalog copied 73 times in separate bundles. Another asset catalog that is virtually identical except for the timestamp at which it was created is copied 67 times. Each of these contain nothing but 482 colors (colors can be stored in asset catalogs to simplify management of dark mode). With each one taking ~250kb these quickly eat up 35mb."
I read this as: Lyft installed Dark mode for 35mb. I can only imagine what my JavaScript modules are doing behind the scenes.
I'd like to see Apple expose more control over their app thinning technologies.
Currently they only deliver the binary for the device's CPU, and only the assets for the device's asset class. There's then some tech targeted at game devs for on-demand assets for things like game levels that you don't need all of on device at one time.
I suspect the limitations of this are around the binary not being subject to this, but maybe it could be. I can see a couple of options, one is some way of extending the asset classes to code features, so that the App Store doesn't have to download iPad screens for iPhones, etc. Perhaps this could be extended with either App Store account region or locale so that, Uber in this example could not include the Venmo SDK outside of the US where no one has heard of Venmo.
Or perhaps Apple could extend the on-demand assets to allow for some sort of plugin system, perhaps backed by Swift Packages, such that apps can on-demand decide they need the Venmo SDK because they're in the US, and download just that. I don't think we want a generalised package manager here, I don't envision that SDK coming from Venmo directly, but allowing an app author to upload all their separate packages if they want to.
With feature heavy, international apps such as Uber I'd expect this to dramatically improve things. I'm not sure whether this benefit would translate to that much demand across the whole App Store though as I think this matters more to a very few big apps. Apple is at that optimisation point in the iOS lifecycle though so perhaps it's worth it to them.
Well since on demand resources were made for games (lua is specifically called out as 'ok'), I could imagine many games also making their initial size far smaller if they used binary code to make new levels or regions. Games are very large chunk of the app store and it's revenue.
Can someone help me understand this? They blame the source of the large bundle size on:
> The choice of Swift as our primary programming language, our fast-paced development environment and feature additions, layered software and its dependencies, and statically linked platform libraries result in large app binaries
but can somebody familiar with iOS development explain what makes app bundles so big? Actual CPU instructions or config can't contribute this significantly. The entire Bible is about 4.5mb. If you're writing an app by yourself you almost certainly didn't write that much text in the source code. A sibling comment links to https://news.ycombinator.com/item?id=25376346 which says that they have a lot of screens but even something like "PayTM (15+ screens)" is still just textual source code and config that I don't follow how it gets beyond kilobytes. The App Store places them at 309mb, so ~68 bibles.
I understand when games are large because they typically ship with images and videos included in the binary for game assets. But for a normal application where does the size come from?
Is it dependencies? (And how did _they_ get so big?) That weird intro video they have on the loading screen? Are they shipping bitmaps of the cities they have markets in?
Uber's article focuses on binary size, but the App Store 309mb number is app bundle size. 120mb of this is not coming from the binary. I have a breakdown of this here: https://news.ycombinator.com/item?id=25380198
The same equivalent code in objective-c is significantly smaller than the same code with swift. Swift has a lot of implicit specializing templates which really bloats code size, like it would in C++. If you compare binary sizes of apps from the pre-swift era, you'll notice many are far smaller. Like I remember tweetbot being 4.5mb in the pre ios 6 days.
I'm not very familiar with iOS dev, but I'd suspect a lot of dependencies, yes.
Also, the Uber app has a LOT more features than you would expect at a glance, due to extensive customization of the experience (i.e. feature flagging) along many vectors, and so it wouldn't surprise me at all if this ends up adding to a lot of code.
Edit: Linked post from sibling commenter bhupy outlines this in detail.
> The app has a couple of millions of lines of code
I wonder if Uber is planning to do anything about that? The technique described in the article (whole program instructions outlining optimisation) is a band aid style solution, merely delaying the inevitable: the code produced by numerous teams independent of each other will inevitably cross first the download size limit threshold, and later maintainability threshold.
I also find it unbelievable to have so much code for a single app, this is approaching the level of magnitude of OS code bases: if you think that the entire Linux kernel is around 28M lines so roughly 15x the Uber app.
The binary size is also from the same ballpark as the entire Windows 98 needed for installation.
I'm glad Uber is doing something about this, but in my opinion Apple should tackle this across their entire ecosystem at the toolchain level, devices with less than 64GB of storage can quickly run out of space with just a handful of applications installed.
Unfortunately it's in Apple's interest that people buy devices with more storage, so I don't expect them to invest much effort in this.
The more the can make the app a "thin-client" (effectively just taking configuration from the server on how to display components w/in a Screen), the more product code they can pull from the app.
I mean I imagine no one person or team completely understands the entire app. Different people/teams are responsible for different portions of the app. Each team only needs to understand their modules and the few other modules they interact with.
Includes, among other things: forcing Apple to increase cellular download limits, 45 seconds for letters to appear in XCode, 12 seconds to call main, rewriting the linker and so on.
This is certainly an great read, and working on it must have very interesting. That being said, in my experience things like these are invariably technical band-aids over social problems. Whenever I see things like this, often paired with statements like “there’s so many screens and feature flags”, usually the problem is not there but actually in many other processes: for example, the design team adds assets in a way that is not enforced by the usual tooling that checks binary size, or the build process adds duplicate files into the bundle that nobody notices. Sometimes the underlying issue is hard to fix, like if it’s code size explosion due to a custom templating engine, but they really should get addressed at some point. Changes like these don’t actually solve the underlying issues, which can be a benefit for a while, but eventually they become so complex that it is hard to maintain them and they start impacting productivity in harder to measure ways by doing things like increasing build times and reducing the quality of debugging information.
> While power law and fractal patterns have revealed themselves in several physical, biological, and man-made phenomena, to our knowledge we are the first to identify their presence in machine-code sequences in computer executable code. Presumably, machine code is a human expression of instructions to a computer and it is well established that all human languages show a power-law in the frequency of the words.
Made me chuckle. Maybe the authors should look at getting an ACM subscription.
It is an over claim to attribute the findings in the blog here to the "Power Laws in Software" paper published in 2008 at ACM Transactions on Software Engineering and Methodology. The ACM paper is casting a wide net on many places where distributions show power law in software and is focused on software modules/libraries/classes and their dependencies. There is a mention of CPU ISA using instruction frequency in CISC architectures but no in-depth treatment of the subject. This blog focuses on the machine instructions and analyzes just not instruction frequency but a whole sequence of instructions and their frequencies plus their lengths in an exhaustive manner.
Machine-code outlining sounds kind of like the opposite of function inlining. Right down to the name! I am amazed I've never heard of this optimization technique being used in compilers before -- it sounds like it could improve performance in many cases by making code smaller (or hurt performance for the same reason that inlining can help performance)...
I've observed that lines of code are measured differently based on whether the writer is trying to convince the audience that the subject matter is big and complicated and the reader should respect the magnitude of dealing with this particular piece of software OR whether the author wants you to appreciate the brevity/simplicity/approachability of the software in question. The first decision made in this decision tree is whether you just use wc , or whether you filter out empty lines. Next goes the comments. Next goes syntactically less significant lines (just a closing brace that could* go on the previous line). Wash rinse repeat.
It's a variant of the "I didn't have time to write you a short ____ so I wrote a long one instead" adage.
I would guess (but only guess) that this article erred on the side overstating size.
Why? If you have 100+ engineers at any given time, shipping features over a period of a few years, you'll hit 1M in no time.
It sounds like a lot, but it really isn't when you consider the amount of people working on it.
Now whether or not you can build the same thing with less LoC, probably. But it's not like it was built from the ground up with every piece of functionality planned out from day 1, so there will be inefficiencies.
Comparing it to Linux is pointless. Platforms should be relatively stable, products are ever changing and the shelf-life of the code is sometimes measured in weeks/months.
Cannot fathom why LOC is a metric? Me neither. Lots of stuff has millions of lines of code in various languages with wide ranging feature-sets and functionality. LOC has near zero meaning across the language/project boundary.
This article is bogus. They spend a whole bunch of time benchmarking build sizes, nothing about why they need so much code in an iOS app to begin with.
Apple has dropped limits for large app downloads on cellular. They now put up a dialog to tell the user how big the download is and if they wish to defer to a WiFi download.
I checked the size of the Uber app and it's about 300MB. Uber Driver is 232MB and Uber Eats is 228MB.
I would love to work on something like this. Optimizing assembler instructions for size, speed.. just writing some. How do you get a job in this? Is embedded land like this?
Why not do machine outlining in LTO/ThinLTO? `opt` doesn't really scale with huge module in terms of memory consumption and multi-threading, that's the reason ThinLTO was invented in the first place.
I think adding machine outlining into LLVM Pass pipeline is still doable with LLVM plugin (with new PassManager)...worst case just come up with a custom LLVM/Clang
So Uber has about 5000 engineers. If all of them write/click/draw 10,000 keystrokes per day, and it's all new code, and Uber has been around 10 years, thats 182 gigabytes of 'human input'.
There's apparently a bananas crunch/backstory to this, where they committed to Swift before realizing they would hit its limits, and had to come up with a bunch of this optimization madness on the fly. I guess this is the cleaned up version and the more final, stable optimizations for the company blog:
Is part of the strategy using private iOS APIs, tracking everyone outside of ToS, lying about it, getting caught, and then being too large for Apple to actually punish them?
[+] [-] bhupy|5 years ago|reply
[+] [-] jedberg|5 years ago|reply
This is why some global apps have different apps for different countries. It's a trade off. Would you rather have a single fat Uber app, or have to download Uber India when you arrive there?
[+] [-] jandrese|5 years ago|reply
Unfortunately the way Apple and Google set up their walled gardens makes this impossible. I guess Apple would prefer if the Uber app dropped all of that and just made everybody use ApplePay instead.
[+] [-] tobmlt|5 years ago|reply
https://mobile.twitter.com/stantwinb/status/1336890442768547...
I can’t seem to find the old link, but I’m pretty sure it was on hacker news and somebody posted a nice collation of the posts.
[+] [-] ljm|5 years ago|reply
Any discussion about software distribution will inevitably result in an argument about dynamic linking vs. static linking.
[+] [-] ChrisMarshallNY|5 years ago|reply
Hopefully, swift ABI stability will reduce that. The new bytecode stuff will help to reduce bloat, as well, but he notes that a lot of SDKs are used. In my experience, SDKs and dependencies often won’t work, compiled with bytecode. Hopefully, that’s changing.
That code repetition thing also happens when a lot of dependencies are used; which often reinvent the wheel. That just comes with the territory. It can be addressed by using highly granular dependencies, but that sort of flies in the face of why we use dependencies. One advantage that Uber has, is they are an 800-lb gorilla. They could contract for specific configurations of dependencies.
I’m not a big Uber user (but I’ve used it a few times). I think it’s a fairly well-done app, as a user.
[+] [-] nhoughto|5 years ago|reply
This post is next level though, deeep optimization. All of it is just increasing the ceiling though, there is some limit on number of features Uber can offer in one app.. and eventually that limit will be reached, doesn’t sound like they are willing to accept being over the limit either. Wonder what space saving techniques are left in the box?
[+] [-] sond813|5 years ago|reply
We parse Obj-C and Swift runtime metadata to determine size contributions of individual types and functions in your app. We use this analysis to post PR comments with granular size diffs to help devs write smaller, better code.
I tried it out on the Uber app and immediately noticed a disproportionate impact from their code-gen dependency injection framework, Needle. The codegen is responsible for over 30k classes in the app binary, and contributes over 10mb! In general codegen is a common problem with Swift binary sizes, and the fewer reference types generated the better, it even helps with startup time!
We’ve written a blog post with case studies about how 7 of the most popular iOS apps could reduce their size: https://medium.com/swlh/how-7-ios-apps-could-save-you-500mb-...
[+] [-] rememberlenny|5 years ago|reply
I read this as: Lyft installed Dark mode for 35mb. I can only imagine what my JavaScript modules are doing behind the scenes.
[+] [-] dang|5 years ago|reply
Launch HN: Emerge (YC W21) – Monitor and reduce iOS app size - https://news.ycombinator.com/item?id=26014180 - Feb 2021 (44 comments)
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] wodenokoto|5 years ago|reply
Glad to see it near the top - saved me a search.
[+] [-] rrdharan|5 years ago|reply
Just curious.
[+] [-] danpalmer|5 years ago|reply
Currently they only deliver the binary for the device's CPU, and only the assets for the device's asset class. There's then some tech targeted at game devs for on-demand assets for things like game levels that you don't need all of on device at one time.
I suspect the limitations of this are around the binary not being subject to this, but maybe it could be. I can see a couple of options, one is some way of extending the asset classes to code features, so that the App Store doesn't have to download iPad screens for iPhones, etc. Perhaps this could be extended with either App Store account region or locale so that, Uber in this example could not include the Venmo SDK outside of the US where no one has heard of Venmo.
Or perhaps Apple could extend the on-demand assets to allow for some sort of plugin system, perhaps backed by Swift Packages, such that apps can on-demand decide they need the Venmo SDK because they're in the US, and download just that. I don't think we want a generalised package manager here, I don't envision that SDK coming from Venmo directly, but allowing an app author to upload all their separate packages if they want to.
With feature heavy, international apps such as Uber I'd expect this to dramatically improve things. I'm not sure whether this benefit would translate to that much demand across the whole App Store though as I think this matters more to a very few big apps. Apple is at that optimisation point in the iOS lifecycle though so perhaps it's worth it to them.
[+] [-] novok|5 years ago|reply
[+] [-] ketralnis|5 years ago|reply
> The choice of Swift as our primary programming language, our fast-paced development environment and feature additions, layered software and its dependencies, and statically linked platform libraries result in large app binaries
but can somebody familiar with iOS development explain what makes app bundles so big? Actual CPU instructions or config can't contribute this significantly. The entire Bible is about 4.5mb. If you're writing an app by yourself you almost certainly didn't write that much text in the source code. A sibling comment links to https://news.ycombinator.com/item?id=25376346 which says that they have a lot of screens but even something like "PayTM (15+ screens)" is still just textual source code and config that I don't follow how it gets beyond kilobytes. The App Store places them at 309mb, so ~68 bibles.
I understand when games are large because they typically ship with images and videos included in the binary for game assets. But for a normal application where does the size come from?
Is it dependencies? (And how did _they_ get so big?) That weird intro video they have on the loading screen? Are they shipping bitmaps of the cities they have markets in?
[+] [-] sond813|5 years ago|reply
App size can be measured in many ways like download size, install size, binary size, thinned size. I wrote about the most important ones here: https://docs.emergetools.com/docs/what-is-app-size
[+] [-] bhupy|5 years ago|reply
[+] [-] novok|5 years ago|reply
[+] [-] mandelbrotwurst|5 years ago|reply
Also, the Uber app has a LOT more features than you would expect at a glance, due to extensive customization of the experience (i.e. feature flagging) along many vectors, and so it wouldn't surprise me at all if this ends up adding to a lot of code.
Edit: Linked post from sibling commenter bhupy outlines this in detail.
[+] [-] viktorcode|5 years ago|reply
I wonder if Uber is planning to do anything about that? The technique described in the article (whole program instructions outlining optimisation) is a band aid style solution, merely delaying the inevitable: the code produced by numerous teams independent of each other will inevitably cross first the download size limit threshold, and later maintainability threshold.
[+] [-] alien_|5 years ago|reply
The binary size is also from the same ballpark as the entire Windows 98 needed for installation.
I'm glad Uber is doing something about this, but in my opinion Apple should tackle this across their entire ecosystem at the toolchain level, devices with less than 64GB of storage can quickly run out of space with just a handful of applications installed.
Unfortunately it's in Apple's interest that people buy devices with more storage, so I don't expect them to invest much effort in this.
[+] [-] tomasreimers|5 years ago|reply
One worth calling out (and recently written about) is server-driven UI: https://artem-tyurin.medium.com/screenflow-an-unfinished-att...
The more the can make the app a "thin-client" (effectively just taking configuration from the server on how to display components w/in a Screen), the more product code they can pull from the app.
[+] [-] sjs7007|5 years ago|reply
I mean I imagine no one person or team completely understands the entire app. Different people/teams are responsible for different portions of the app. Each team only needs to understand their modules and the few other modules they interact with.
[+] [-] layoutIfNeeded|5 years ago|reply
[+] [-] draw_down|5 years ago|reply
[deleted]
[+] [-] dmitriid|5 years ago|reply
Includes, among other things: forcing Apple to increase cellular download limits, 45 seconds for letters to appear in XCode, 12 seconds to call main, rewriting the linker and so on.
[+] [-] ohiovr|5 years ago|reply
[+] [-] satya71|5 years ago|reply
[+] [-] saagarjha|5 years ago|reply
[+] [-] whoisburbansky|5 years ago|reply
Made me chuckle. Maybe the authors should look at getting an ACM subscription.
[https://dl.acm.org/doi/pdf/10.1145/1391984.1391986]
[+] [-] bangara|5 years ago|reply
[+] [-] etaioinshrdlu|5 years ago|reply
[+] [-] thegeomaster|5 years ago|reply
What?
Linux has 30 million of C!
I'm speechless. I cannot fathom how & why.
[+] [-] travisgriggs|5 years ago|reply
It's a variant of the "I didn't have time to write you a short ____ so I wrote a long one instead" adage.
I would guess (but only guess) that this article erred on the side overstating size.
[+] [-] dntrkv|5 years ago|reply
Why? If you have 100+ engineers at any given time, shipping features over a period of a few years, you'll hit 1M in no time.
It sounds like a lot, but it really isn't when you consider the amount of people working on it.
Now whether or not you can build the same thing with less LoC, probably. But it's not like it was built from the ground up with every piece of functionality planned out from day 1, so there will be inefficiencies.
Comparing it to Linux is pointless. Platforms should be relatively stable, products are ever changing and the shelf-life of the code is sometimes measured in weeks/months.
[+] [-] pid_0|5 years ago|reply
[deleted]
[+] [-] edoceo|5 years ago|reply
[+] [-] abductee_hg|5 years ago|reply
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] speedgoose|5 years ago|reply
[+] [-] donarb|5 years ago|reply
Apple has dropped limits for large app downloads on cellular. They now put up a dialog to tell the user how big the download is and if they wish to defer to a WiFi download.
I checked the size of the Uber app and it's about 300MB. Uber Driver is 232MB and Uber Eats is 228MB.
[+] [-] dj_mc_merlin|5 years ago|reply
[+] [-] mshockwave|5 years ago|reply
I think adding machine outlining into LLVM Pass pipeline is still doable with LLVM plugin (with new PassManager)...worst case just come up with a custom LLVM/Clang
[+] [-] londons_explore|5 years ago|reply
Compiling that down to 200 MB isnt too shabby!
[+] [-] lgats|5 years ago|reply
If your app has larger images, don't waste user bandwidth and optimize your assets!
[+] [-] elpakal|5 years ago|reply
> Overall, 5 rounds of outlining builds in 66 minutes — a 45-minutes addition to the baseline.
[+] [-] rockmeamedee|5 years ago|reply
https://twitter.com/StanTwinB/status/1336890442768547845
[+] [-] alien_|5 years ago|reply
[+] [-] pid_0|5 years ago|reply