Always a fascinating read when it pops up on HN :)
One thing to note - assumption that one might make is that "Rockstar shipped obviously ridiculously suboptimal code; how did this ever get past testing??"
However, based on the author's discussion of the code, and the issue being in inefficient parsing of an online store content, it is likely that this not only passed testing, but was sufficiently efficient in production too... upon go-live, when the store content database was small.
This likely grew slowly, and was especially bad for a specific subset of users (CPUs with poor single core performance). They may even have had trouble replicating it in non-production environments depending on their refresh policy and whatnot.
This is not to excuse it, of course; but I've seen a repeated pattern where a not-perfectly-optimal algorithm works well enough in development, testing, and initial production... then blows up in production with growth a couple of years later.
(In particular, I deal with a lot of complex SQL in my day job, and they have a nasty habit of going exponential with data growth. If you're implementing a greenfield application with no objective knowledge of its future size and growth patterns, it may be littered with trapmines despite reasonably good efforts to not make it so. Especially dangerous with "Big Bang Go-Lives" - though a game may have internal Agile sprints, I'd still consider it overall a bing-bang go-live in that majority of its content will go live at the same time, and then you get to see how your 100s/1000s/10,000s of SQL/code/libraries/etc behave:)
The loading times were atrocious even at launch though for Xbox 360. It wasn't a case of content piling up, they added a lot but probably a lot of their item catalogue was there from launch.
There was also server instability near launch which it may have been thought to be linked, as in we thought we were waiting 10 minutes for a server. But any actual server issue was probably more demand/load scaling based.
The devs were may have been too busy "sprinting" and getting their assigned tickets done to think about higher level concerns like "is this good? can we make things faster or better?".
That's insane to achieve that without access to source code.
It's also completely baffling that a company would invest such resources building such a complex game and be fine with 6 minute load times, caused by such ... dumbness. Like selling a sportscar with a 5 litre petrol tank.
My sneaking suspicion is that the real reason that everything is so slow is that noticeable loading or processing times give the software a feeling of heft. It strokes our ego as programmers because if something takes a while to complete then it must be very complicated and important.
I cant understand why there wasnt mass firings across managements and a post mortem, aswell as a hiring of the guy who fixed it, 6 figure reward and a new team setup to troubleshoot final build performance problems.
But I can understand thats its normal large companies are so incompetent, and at an older age I appreciate it gives a chance to the little guy or company to rise up, for now, until AI starts to catch these issues and large inefficient corporations can finally solidify their place in the hierarchy.
I remember reading this article when it first came out. It still gives me the same feeling of inadequacy. There are some people walking around us who just belong to a different league.
I can exactly imagine how it would happen in a large company:
1. It's a boiling frog problem which developed over several years. It started fast when the JSON file was small but then got a little bit slower each time new content is added.
2. It's also not exactly an "interesting" part of the code base. IIRC it's a JSON file which configures the ingame shop(?). This was probably delegated to a summer intern to implement in a couple of days, maybe in an entirely different department from the people who develop the actual core game.
3. New shop items are probably added by a content team without programmers in the loop who would know by instinct that something must be wrong. People on that content team also most likely don't stay on that team for very long, which together with the boiling frog problem above means that everybody thinks "it's always been like that".
4. Any specific feedback from the outside probably gets lost in the noise of other requests.
5. ...plus a general 'not my problem' attitude common in big organizations and an always-full kanban board ;)
I also cannot explain it. If I was a dev on a system that had a 6 minute loading time, I'd be stopping what I was supposed to be doing and debugging it after a couple of days of suffering through it.
I imagine the focus is on creating new things instead of fixing old things, since that will have more obvious and marketable impacts. The whales might tolerate longer load times over stale content.
If no one carries the responsibility, no one will care. Maybe the team who worked on that moved on and what’s done is done. Not sure if R* has a team that does optimization only?
Remember reading this way back, glad to see he/she got a 10k reward from R*, great piece of debug work and a bit of inspiration to improve my own skills.
10k reward is a joke. Reminds me of the guy who found you can submit a sitemap for a different domain in Google Webmaster Tools (if the site suffers from open redirect issues). You would then get huge search traffic for a brand new domain. He first got $1337 reward which was then "upgraded" to $5000 for a bug that's worth millions.
How do I become that good in debugging stuff like this? I'm guessing he works with gamedev and does this sort of debugging and reverse engineering regularly?
I learned some of this stuff being motivated by doing mods that interacts directly with game's memory.
I mostly build free-cameras, which allows you to detach the camera from characters, and for that I had to learn x86 assembly, some disassembler tools and how compilers generally work.
I think it's super rewarding because while you're seeing bits and bytes, and crashes, most of the day, at some point you can see you achieved something by being able to freely move in a restricted world.
I was surprised how far you can get with Cheat Engine. Its disassembler is really good, and you can do injections directly, if you then want to move to your own advanced stuff, you can build frameworks to do all of that using Windows APIs.
Learn to use a performance profiler and then a debugger/disassembler/whatever fit you needs. The profiler tells you where the slowdown happens, the rest then let you figure out what actually happens.
By learn to use I mean really do, it's like refactoring tools, it's sort of boring and tiring to learn those, you end skipping large parts because "yeah I already know / it's obvious", then you need one day for an emergency / something right now and feel like you're fighting against it.
For those three (and some others), the key to being very good at it is learning all the small tricks and details, when you don't need it, so the day you do it lets you skip 90% of the work.
As an overall point, I'm sure many people have various different experiences, but for myself while being in high school I needed to parse "replay" files for some games to extract and process information and allow players to post their replays to get stats on web forums, and it helped me to learn a great deal about how to properly reverse engineer and figure out stuff, things that I have no idea how I would have learned otherwise.
Is there a safe way to do this and not get your online accounts banned? I'm interested in tinkering in this way but I'm not a cheater and I don't want to be confused for a cheater and banned.
I mean GTA Online is a microtranscation/grindfest experience, mostly to milk kids of their money. It's not really worth the time due to how grindy it is.
Most PC gamers play multiplayer mods like RageMP that allow connecting to servers with well-made scripted gamemodes like RP, but also stuff like death match where hackers are actually banned I'd guess.
[+] [-] NikolaNovak|2 years ago|reply
One thing to note - assumption that one might make is that "Rockstar shipped obviously ridiculously suboptimal code; how did this ever get past testing??"
However, based on the author's discussion of the code, and the issue being in inefficient parsing of an online store content, it is likely that this not only passed testing, but was sufficiently efficient in production too... upon go-live, when the store content database was small.
This likely grew slowly, and was especially bad for a specific subset of users (CPUs with poor single core performance). They may even have had trouble replicating it in non-production environments depending on their refresh policy and whatnot.
This is not to excuse it, of course; but I've seen a repeated pattern where a not-perfectly-optimal algorithm works well enough in development, testing, and initial production... then blows up in production with growth a couple of years later.
(In particular, I deal with a lot of complex SQL in my day job, and they have a nasty habit of going exponential with data growth. If you're implementing a greenfield application with no objective knowledge of its future size and growth patterns, it may be littered with trapmines despite reasonably good efforts to not make it so. Especially dangerous with "Big Bang Go-Lives" - though a game may have internal Agile sprints, I'd still consider it overall a bing-bang go-live in that majority of its content will go live at the same time, and then you get to see how your 100s/1000s/10,000s of SQL/code/libraries/etc behave:)
[+] [-] whatyesaid|2 years ago|reply
There was also server instability near launch which it may have been thought to be linked, as in we thought we were waiting 10 minutes for a server. But any actual server issue was probably more demand/load scaling based.
[+] [-] atom_arranger|2 years ago|reply
"Scrum", "Agile", "Lean".
The devs were may have been too busy "sprinting" and getting their assigned tickets done to think about higher level concerns like "is this good? can we make things faster or better?".
I think your points also stand though.
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] stevage|2 years ago|reply
It's also completely baffling that a company would invest such resources building such a complex game and be fine with 6 minute load times, caused by such ... dumbness. Like selling a sportscar with a 5 litre petrol tank.
[+] [-] jvmboi|2 years ago|reply
[+] [-] DanielHB|2 years ago|reply
They are probably just adding more content, but the original dev's algorithm for loading that content didn't expect a bazillion new cosmetics
[+] [-] ionwake|2 years ago|reply
But I can understand thats its normal large companies are so incompetent, and at an older age I appreciate it gives a chance to the little guy or company to rise up, for now, until AI starts to catch these issues and large inefficient corporations can finally solidify their place in the hierarchy.
[+] [-] jcmontx|2 years ago|reply
[+] [-] el_duderino|2 years ago|reply
[+] [-] xxs|2 years ago|reply
[+] [-] kramerger|2 years ago|reply
I am baffled nobody at R* cared to investigate a 6min load time that was there for years. Doesn't sound very AAA-gamedev to me.
[+] [-] flohofwoe|2 years ago|reply
1. It's a boiling frog problem which developed over several years. It started fast when the JSON file was small but then got a little bit slower each time new content is added.
2. It's also not exactly an "interesting" part of the code base. IIRC it's a JSON file which configures the ingame shop(?). This was probably delegated to a summer intern to implement in a couple of days, maybe in an entirely different department from the people who develop the actual core game.
3. New shop items are probably added by a content team without programmers in the loop who would know by instinct that something must be wrong. People on that content team also most likely don't stay on that team for very long, which together with the boiling frog problem above means that everybody thinks "it's always been like that".
4. Any specific feedback from the outside probably gets lost in the noise of other requests.
5. ...plus a general 'not my problem' attitude common in big organizations and an always-full kanban board ;)
[+] [-] spuz|2 years ago|reply
[+] [-] Hamuko|2 years ago|reply
[+] [-] prox|2 years ago|reply
[+] [-] sonicanatidae|2 years ago|reply
[+] [-] aronhegedus|2 years ago|reply
[+] [-] mywacaday|2 years ago|reply
[+] [-] Phil_Latio|2 years ago|reply
[+] [-] bezier-curve|2 years ago|reply
[+] [-] thiago_fm|2 years ago|reply
[+] [-] etra0|2 years ago|reply
I mostly build free-cameras, which allows you to detach the camera from characters, and for that I had to learn x86 assembly, some disassembler tools and how compilers generally work.
I think it's super rewarding because while you're seeing bits and bytes, and crashes, most of the day, at some point you can see you achieved something by being able to freely move in a restricted world.
I was surprised how far you can get with Cheat Engine. Its disassembler is really good, and you can do injections directly, if you then want to move to your own advanced stuff, you can build frameworks to do all of that using Windows APIs.
[+] [-] nolok|2 years ago|reply
By learn to use I mean really do, it's like refactoring tools, it's sort of boring and tiring to learn those, you end skipping large parts because "yeah I already know / it's obvious", then you need one day for an emergency / something right now and feel like you're fighting against it.
For those three (and some others), the key to being very good at it is learning all the small tricks and details, when you don't need it, so the day you do it lets you skip 90% of the work.
As an overall point, I'm sure many people have various different experiences, but for myself while being in high school I needed to parse "replay" files for some games to extract and process information and allow players to post their replays to get stats on web forums, and it helped me to learn a great deal about how to properly reverse engineer and figure out stuff, things that I have no idea how I would have learned otherwise.
[+] [-] alecco|2 years ago|reply
Don't jump in with "hi please teach me" or asking questions in the FAQ/wiki or found easily on the web.
Try to give back to the community (like fixing/improving the FAQ/Wiki) as early as possible.
[+] [-] m-a-r-c-e-l|2 years ago|reply
[+] [-] ErneX|2 years ago|reply
[+] [-] rightbyte|2 years ago|reply
[+] [-] kowlo|2 years ago|reply
[+] [-] cedws|2 years ago|reply
[+] [-] skywal_l|2 years ago|reply
- Luke Stackwalker: For stack sampling (perf analysis)
- ???: industry-standard disassembler
- Process Dump: Dump process memory to file (for obfuscated code analysis)
- x64dbg: For debug stepping.
- MinHook: For patching/modding a binary executable.
[+] [-] whackx|2 years ago|reply
guess that's IDA Pro
[+] [-] mcint|2 years ago|reply
I cut GTA Online loading times (2021) (June 9, 2022 — 476 points, 213 comments) https://news.ycombinator.com/item?id=31681515
How I cut GTA Online loading times by 70% (February 28, 2021 — 3883 points, 699 comments) https://news.ycombinator.com/item?id=26296339
[+] [-] 2OEH8eoCRo0|2 years ago|reply
[+] [-] jpeter|2 years ago|reply
[+] [-] slifin|2 years ago|reply
Played recently and thought Rockstar must not have actioned the advice in the blog
But no it's just it is still slow but they did fix these issues kudos to them but performance still needs attention
People who play this regularly have their phones out/youtube videos on waiting for these load times
[+] [-] haezee|2 years ago|reply
[+] [-] oscarteg|2 years ago|reply
[+] [-] whatyesaid|2 years ago|reply
Most PC gamers play multiplayer mods like RageMP that allow connecting to servers with well-made scripted gamemodes like RP, but also stuff like death match where hackers are actually banned I'd guess.
[+] [-] asylteltine|2 years ago|reply
[+] [-] vicktorium|2 years ago|reply
this really makes me think:
if this obvious flaw exists ... imagine what other, more complicated and opaque mistakes exists and we're not aware of it ...
just food for thought
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] asadalt|2 years ago|reply
[+] [-] ghusbands|2 years ago|reply
[+] [-] two_handfuls|2 years ago|reply
[+] [-] robblbobbl|2 years ago|reply