I remember a similar bug in the original Asherons Call (amazingly, still running with regular content updates!). Three or four years in, they added a new, very expensive object to the game (a "Pyreal Scarab" for anyone who cares). It cost a lot, so the idea was that people wouldn't be (at that point in the economy) buying many.
Now, in Asheron's Call, the world is huge. There are hundreds, if not thousands of vendors. And three of these vendors were set to sell their goods in stacks of up to 1000. Unfortunately, Cost of Pyreal Scrab * 1000 > 2^31, which wrapped. I can't remember if you either just got the goods for free (which you then sold back for huge profit), or if you actually got paid to take these things. Either way, overnight, the economy was destroyed. The entire game state had to be reset from backups; a dreaded rollback. Worse, the developer took a few days to do this.
Trust me, out of all the customers whose data you don't want to muss with, it's hardcore MMORPG players. Even though I was just a player, I can still remember the outrage all these years later. It taught me to always use appropriate types for objects with "value", and I've never accidentally used signed or floating point storage for currency again.
Haha - I was the one who was responsible for adding the Pyreal Scarab. I remember freaking out that night when I was playing and went to Teth's vendor and saw what was happening. I called up the producer at the time at 2am and said there was a "problem".
I remember this scenario distinctly because it happened within a month of me joining the game, and not understanding what was going on :) The bug was discovered (or at least widely known) about a day after the patch went live, and rolled back two days later. Nearly every bug that users felt were somewhat punishing led to players hitting the VNBoards asking about possible rollbacks so they could stop playing until it was done. I'm actually not sure if they've ever performed one since.
Consider this completely anecdotal, but I think that around the time this bug actually occurred, Asheron's Call and Everquest were the only two 3D MMOs that were worth mentioning. I recall the delay in rollback having something to do with Microsoft bureaucracy at the time as well -- Turbine was plagued with MS as a publisher having some sort of veto power over their business and was frequently met with resistance.
(Full disclosure: I love Turbine unconditionally for creating such memorable adventuring experiences with Asheron's Call 1 and 2)
That was the first thing which came to my mind when I read this, too. Of course, Asheron's Call didn't have an (official) real-money trading system to deal with too.
A nightmare scenario for the developers in both cases.
>Trust me, out of all the customers whose data you don't want to muss with, it's hardcore MMORPG players.
Welp, that's what happens when you have only online play on only official servers. Single player offline wouldn't be affected - cheat all you want! Online play on unofficial servers means server admins can take whatever action they want - ban offenders, leave offenders alone, or rollback - depending on what the admin and the players want.
This is what happens when we let machine constraints trump semantics.
When you say int, you usually want an actual integer, not an integer with an arbitrary limit. In this day and age, having that limit there is simply premature optimization.
I think having a nice bignum type--one that looks and feels just like a normal numeric type--is very important. It should also probably be the default; you should only switch to a machine type if you have a good reason. With gmp, big integers perform well enough to be used widely.
I assume this was probably a server side bug, since all the accounting would never be trusted to the client side.
If you are writing highly-performant server code, the actually memory size is extremely important. You cannot (should not) abstract away the machine specifics of the datatype if you want to write optimized code.
In some cases where the underlying datatype isn't a concern (e.g. Javascript), I agree with you. But ultimately, this isn't a failure of technology, it is a failure of the software development process.
Completely agree about big integers. I expect that the next generation of programming languages will offer a "number" type that's big integer or decimal by default, and then allow engineers to refine the type as an optimization. One trouble is the data storage layer - typical database fields are fixed size and don't naturally accommodate big integers. Thus where MMOs are concerned, using big integers will still require some deliberate effort.
Another approach is to have compile time checking of overflow or bounded numbers. There's been discussion about the utility of this on the Rust list recently and I wrote up some of my experience with languages that let you do checking: http://bluishcoder.co.nz/2013/05/07/ranged-integer-types-and...
Just another reminder that it's never worth bypassing the normal deployment process. Every year or two I learn a similar lesson myself: it's so tempting, the fix is so small, it couldn't possibly break anything (heck, I once had a data-gathering script that made a bunch of read-only calls to our system's API cause a live issue). Just say no.
Reminds me of a time a coworker at a previous job almost brought down a production server by opening a file in vi.
That file was a customer log that had grown to 100+gb in size due to an error that she was debugging. She failed to check the log size, instead assuming that it was a small file left over after that night's log rotation. When vi tried to load the file to memory, it almost crashed the box before we could kill it (we still got calls about degraded performance though).
Why wouldn't currency by handled by the type system? You could still have an overrun. But it'd be handled more appropriately.
Long ago, I wrote a budgeting / estimating tool. Costs were represented with binary coded decimals (BCDs). Not floating point numbers. Just like an accounting system.
Competing products could have weird roundoff errors. Not mine.
To be fair, I am not sure if anyone would try posting 6 billion gold onto a PTR equivalent of the RMAH or whether the RMAH is even available on the PTR, which might explain why it wasn't tested.
Kingdom of Loathing had an integer overflow bug way back in 2004. It let players set their currency (in this case: meat) to the max value of a 64-bit integer. The game spent the next several months setting up meatsinks in an attempt to reduce the amount of meat in circulation.
Reminds be of a bug waaaay back. In the original Elite, you could obtained a missile lock on a space station, then dock and sell all your missiles, launch, and finally fire a missile (you still had the missile lock). Suddenly you have 255 missiles! 0-1 = 255 in unsigned 8 bit integer math!
You could then sell all your new shiny missiles for loads of money. Made a hard game a bit easier.
Those games (Frontier Elite and First Encounters included) had loaaads of bugs like that.
Another one in FE had you put in passenger holds, fill them with passengers, then sell the holds -- this would obviously not work as you had to evict the passengers first, however the game logic credited you with the cash anyway because the check came after the money had changed hands.
Hard to understand why people spend so much time on a short game like Diablo III. Even before I finished it on inferno the game didn't make much sense, I just finished for the sake of it, which I regret.
Most Super Mario games requires way more abilities than that and less time.
Don't waste your limited time on earth playing consumption-driven games. I've been trying Eve online for a few days, It does not looks promising, it seems that Eve also is also driven by item accumulation and not actual playing.
It's basically a very pretty slot machine. I played quite a bit of D2 and D3. The thrill comes from grinding for hours (and hours and hours) and finding that one item that earns you hundreds of dollars. (Yes real dollars) You can thank the the Real-Money Auction House for both destroying the game and being the one thing that keeps so many people coming back for more.
Eve was a black whole for me. Played casually for a week, then learned how to make money via trading, then I found out there is a Python API for grabbing data out of the game client, as well as some other JavaScript-APIs using the in-game Browser.
Next thing you know I was crawling all popular market hubs in Eve, storing price history of each item in mysql, and programmatic analyzing the data to find the best trade routes for profit.
Then I realized I need much more data, and prepared a small data-grabber client for other people to run, as well some cloud storage to upload it to.
I looked at the calendar and noticed 2 weeks have past and I didn't do much else, so I came to the conclusion this might not be the most productive thing to do and quit Eve :) Problem is, I can't play these games the "normal" way, when I see it got APIs etc. I just have to go all out on it - or just not play at all.
Incidentally, Eve is the source of my favorite overflow bug! By putting a ship into a region of space that reduces the range of its weapons and using various range-debuffs, players were able to decrease the range of certain weapons by enough to get the range variable to roll over. This gave them near-infinite range on weapons that were intended to be short range. Since the short range was a tradeoff for very high damage, suddenly having infinite range was game breaking. However, the people who discovered the bug were clever, and did not abuse the power enough to be detected for nearly a year.
You're very wrong in that assumption. Don't get me wrong, I wasted 9 months of my life becoming completely absorbed by the game but it's so immersive that no other game can compare.
It's about hoarding, alliances, corporations, mining ops, PVP ops, big alliance battles, 0-sec space mining/pvp ops (this is the best part of the game).
I've heard Eve online called an animated space themed spreadsheet before. I tried it myself a long time ago, and I felt it was too complicated and boring.
I played enough to get a lv. 60 Monk and lv. 60 Demon Hunter. The problem with Diablo III, compared to Diablo II from 10 years ago, is that nowadays there are so many options for online games without monthly fees that there's little incentive to stick to just one.
Then I started Guild Wars 2 a few months later, and played that to death. :)
Why? It's just a matter of scale. Until very recently, Romania's currency was such that most middle-class families had 8-10 digits in their bank account at any given moment, and the economy is relatively healthy.
Good items sell for hundreds of millions. The number of zeroes doesn't matter, as long as the balance between items and monetary value is stable.
I feel sorry for the players who were amused by this and will now likely get banned. I don't know when video games started to be like a bad elementary school where you get punished for experimenting or finding a loophole, but it seems like it's punishing one of the fundamental joys of games. Or at least one of the joys I remember being particularly rewarding as a child.
I agree to some degree but MMO games in general are serious business and a special breed, especially when real currency is involved. It was most definitely against the ToS and I am sure that most of the players who participated in the exploit knew that. Also there were people who were using this bug to make real life money through the real money auction house, those people in particular can't be surprised that they were banned.
Well, when you're messing with a real-money economy, I think that's a pretty big tip-off that you'll be punished for experimenting or finding a loophole. The same thing happens with other games that deal in real money, such as the stock market.
Also, AFAIK the only parties who exploited this bug were gold-farming bots. The computers won't mind, especially since their owners probably made bank off this.
It is odd that they're using 32-bit numbers when you'd be hard pressed to find a 32-bit only CPU and machines with over 4GB of memory are the standard.
I remember World of Warcraft had a similar issue. The total amount of copper and in turn gold a player could have was the positive half of a signed 32-bit integer.
Very different issues. The total amount of the transaction is preserved. For this to be analogous you'd have to retain the copper after converting to gold and I'm not an expert on WoW but I don't ever recall that happening.
The scary thing is that integer overflows are considered rare so unlike things like null-pointer dereference no one really checks for them (heck, it seems impractical checking for it).
In this case, how should they defend against an overflow? Impose an arbitrary limit on gold?
I told this in their forums many times: you cannot have both real world money and one digital currency at the same time. Farming gold is hillarious. You need to implement instead something like bitcoin and it's the only way to stop the inflation.
Something like this used to work in Sim Farm too: buy and sell a piece of land and watch taxes grow until they flow over and you get a bunch of money instead of paying. If only this worked in real life :-)
Is this sort of hyperinflation intentional? Or a sign of economic ignorance? I don't get it. Gold is as common as dirt, and players are pumping huge amounts of it into the economy on a constant basis.
[+] [-] archgrove|13 years ago|reply
Now, in Asheron's Call, the world is huge. There are hundreds, if not thousands of vendors. And three of these vendors were set to sell their goods in stacks of up to 1000. Unfortunately, Cost of Pyreal Scrab * 1000 > 2^31, which wrapped. I can't remember if you either just got the goods for free (which you then sold back for huge profit), or if you actually got paid to take these things. Either way, overnight, the economy was destroyed. The entire game state had to be reset from backups; a dreaded rollback. Worse, the developer took a few days to do this.
Trust me, out of all the customers whose data you don't want to muss with, it's hardcore MMORPG players. Even though I was just a player, I can still remember the outrage all these years later. It taught me to always use appropriate types for objects with "value", and I've never accidentally used signed or floating point storage for currency again.
[+] [-] jdk|13 years ago|reply
"Oops." --Devilmouse
[+] [-] lost_name|13 years ago|reply
Consider this completely anecdotal, but I think that around the time this bug actually occurred, Asheron's Call and Everquest were the only two 3D MMOs that were worth mentioning. I recall the delay in rollback having something to do with Microsoft bureaucracy at the time as well -- Turbine was plagued with MS as a publisher having some sort of veto power over their business and was frequently met with resistance.
(Full disclosure: I love Turbine unconditionally for creating such memorable adventuring experiences with Asheron's Call 1 and 2)
[+] [-] sveiss|13 years ago|reply
A nightmare scenario for the developers in both cases.
[+] [-] EvilLook|13 years ago|reply
Welp, that's what happens when you have only online play on only official servers. Single player offline wouldn't be affected - cheat all you want! Online play on unofficial servers means server admins can take whatever action they want - ban offenders, leave offenders alone, or rollback - depending on what the admin and the players want.
[+] [-] tikhonj|13 years ago|reply
When you say int, you usually want an actual integer, not an integer with an arbitrary limit. In this day and age, having that limit there is simply premature optimization.
I think having a nice bignum type--one that looks and feels just like a normal numeric type--is very important. It should also probably be the default; you should only switch to a machine type if you have a good reason. With gmp, big integers perform well enough to be used widely.
[+] [-] vinkelhake|13 years ago|reply
You say this with certainty. Do you know of studies of real-world programs where machine-sized integers were replaced whole-sale with bignums?
[+] [-] mckilljoy|13 years ago|reply
I assume this was probably a server side bug, since all the accounting would never be trusted to the client side.
If you are writing highly-performant server code, the actually memory size is extremely important. You cannot (should not) abstract away the machine specifics of the datatype if you want to write optimized code.
In some cases where the underlying datatype isn't a concern (e.g. Javascript), I agree with you. But ultimately, this isn't a failure of technology, it is a failure of the software development process.
[+] [-] jcrites|13 years ago|reply
[+] [-] doublec|13 years ago|reply
[+] [-] hebz0rl|13 years ago|reply
[+] [-] lmm|13 years ago|reply
[+] [-] mnarayan01|13 years ago|reply
I don't think you can just throw that out here with no story. Well...I guess you can, it just makes me sad.
[+] [-] lobotryas|13 years ago|reply
That file was a customer log that had grown to 100+gb in size due to an error that she was debugging. She failed to check the log size, instead assuming that it was a small file left over after that night's log rotation. When vi tried to load the file to memory, it almost crashed the box before we could kill it (we still got calls about degraded performance though).
[+] [-] specialist|13 years ago|reply
Why wouldn't currency by handled by the type system? You could still have an overrun. But it'd be handled more appropriately.
Long ago, I wrote a budgeting / estimating tool. Costs were represented with binary coded decimals (BCDs). Not floating point numbers. Just like an accounting system.
Competing products could have weird roundoff errors. Not mine.
[+] [-] Guvante|13 years ago|reply
[+] [-] bloaf|13 years ago|reply
http://kol.coldfront.net/thekolwiki/index.php/Black_Sunday
http://kol.coldfront.net/thekolwiki/index.php/Bugmeat
http://kol.coldfront.net/thekolwiki/index.php/Meatsink
[+] [-] russellsprouts|13 years ago|reply
[+] [-] tlarkworthy|13 years ago|reply
You could then sell all your new shiny missiles for loads of money. Made a hard game a bit easier.
[+] [-] mickeyp|13 years ago|reply
Another one in FE had you put in passenger holds, fill them with passengers, then sell the holds -- this would obviously not work as you had to evict the passengers first, however the game logic credited you with the cash anyway because the check came after the money had changed hands.
[+] [-] eduardordm|13 years ago|reply
Most Super Mario games requires way more abilities than that and less time.
Don't waste your limited time on earth playing consumption-driven games. I've been trying Eve online for a few days, It does not looks promising, it seems that Eve also is also driven by item accumulation and not actual playing.
[+] [-] vyrotek|13 years ago|reply
[+] [-] zyb09|13 years ago|reply
Next thing you know I was crawling all popular market hubs in Eve, storing price history of each item in mysql, and programmatic analyzing the data to find the best trade routes for profit.
Then I realized I need much more data, and prepared a small data-grabber client for other people to run, as well some cloud storage to upload it to.
I looked at the calendar and noticed 2 weeks have past and I didn't do much else, so I came to the conclusion this might not be the most productive thing to do and quit Eve :) Problem is, I can't play these games the "normal" way, when I see it got APIs etc. I just have to go all out on it - or just not play at all.
[+] [-] bloaf|13 years ago|reply
http://massively.joystiq.com/2010/09/17/new-eve-exploit-give...
[+] [-] Ixiaus|13 years ago|reply
It's about hoarding, alliances, corporations, mining ops, PVP ops, big alliance battles, 0-sec space mining/pvp ops (this is the best part of the game).
[+] [-] Hairy_Sandwich|13 years ago|reply
[+] [-] minimaxir|13 years ago|reply
Then I started Guild Wars 2 a few months later, and played that to death. :)
[+] [-] codeduck|13 years ago|reply
[+] [-] danceonfire|13 years ago|reply
Is this an error? :) Although I assume the players would very well enjoy game-breaking exploits, as long as they are to their advantage.
[+] [-] seanalltogether|13 years ago|reply
[+] [-] apetresc|13 years ago|reply
Good items sell for hundreds of millions. The number of zeroes doesn't matter, as long as the balance between items and monetary value is stable.
[+] [-] Glyptodon|13 years ago|reply
[+] [-] gebe|13 years ago|reply
[+] [-] chc|13 years ago|reply
Also, AFAIK the only parties who exploited this bug were gold-farming bots. The computers won't mind, especially since their owners probably made bank off this.
[+] [-] doublec|13 years ago|reply
[+] [-] astrodust|13 years ago|reply
Good luck overflowing a 64-bit unsigned.
[+] [-] sown|13 years ago|reply
[+] [-] minimaxir|13 years ago|reply
Although back in those days, if you had that much gold, you were controlling the economy by yourself anyways.
[+] [-] josh2600|13 years ago|reply
[+] [-] login1234|13 years ago|reply
[+] [-] mikevm|13 years ago|reply
In this case, how should they defend against an overflow? Impose an arbitrary limit on gold?
[+] [-] wtetzner|13 years ago|reply
[+] [-] maaku|13 years ago|reply
[+] [-] Jabbles|13 years ago|reply
[+] [-] meerita|13 years ago|reply
[+] [-] pilif|13 years ago|reply
[+] [-] nsxwolf|13 years ago|reply
[+] [-] ebbv|13 years ago|reply
[+] [-] yekko|13 years ago|reply
[+] [-] WhoIsSatoshi|13 years ago|reply