top | item 18845205

Game dev: Linux users were only 0.1% of sales but 20% of crashes and tickets

414 points| belltaco | 7 years ago |twitter.com | reply

400 comments

order
[+] rkido|7 years ago|reply
Linux is the perpetual scapegoat.

They selected a middleware, Coherent UI, that didn't work properly on anything but Windows. They also didn't make proper use of the Steam runtime, a mistake that continues to cause issues. Most games don't make these mistakes, so this isn't really representative of the larger state of Linux gaming.

It's worth pointing out that the devs did make a legitimately good attempt at making Linux/Mac support first class, by making a purely OpenGL engine for the game. But it seems they weren't aware of some other best practices.

[+] simias|7 years ago|reply
Technically speaking I'm sure you're right. In practice it doesn't really matter though, if they did it that way it's either because it was easier or because that's the way they were used to doing it. The fact that "technically" it's not Linux's fault doesn't really matter, unless you're more interested in the moral concept of guilt rather that the practicalities of making a Linux port of a game.

Why should they change the way they develop games for 0.1% of their sales? So that they'd get kudo "clean code" points from people on HN? Most for-profit software cares very little about that, and for good reasons.

Gaming on Linux sucks because Linux is not popular for gaming. Linux is not popular for gaming because gaming on Linux sucks. That's the root of the issue.

[+] stcredzero|7 years ago|reply
Most games don't make these mistakes...it seems they weren't aware of some other best practices.

I see this time and time again. Game developer runs into one of the difficult subjects in computer science/programming. Dismisses the difficulty. Gets themselves into trouble. Blames the library/product/platform.

The last time I was at a game jam, I found myself explaining generational garbage collection to a game dev. He then proceeded to "solve" all of his problems by writing a 5 minute hack to ensure all of his objects would be collected as quickly as possible. Which didn't work, because the lifespan of his objects is dependent on gameplay, so he can't ensure all of his objects will die in Eden. I kept trying to explain that he should actually hold onto his objects (especially bullets in his bullet hell game) and recycle them as much as possible, so that he would reduce memory pressure and have those objects promoted to old space, so they are looked at less often.

I see this sort of thing time and time again.

[+] politician|7 years ago|reply
Saw "Coherent UI" in your comment and thought, "I bet it's Planetary Annihilation" and was not disappointed.

The dev team struggled to get Coherent UI working on Windows, to say nothing of Linux. They switched to Coherent UI late in the development cycle and the beta/launch was constantly glitched out. The problem with trying to run an out-of-proc UI renderer within a game loop were so numerous -- the game would frequently lock up in full screen with multiple instances of the Coherent UI render process running if you managed to escape.

... in Windows, not in Linux.

For fans of TA and SupCom, early PA was terrible. Later, the team worked through a lot of the issues, but the damages to their finances (and reviews) were done and PA never got the e-sports league it deserved.

[+] smileybarry|7 years ago|reply
He points out that a very large percentage of the crashes were graphics driver related, not dependency issues that would be solved by the Steam runtime.
[+] sephoric|7 years ago|reply
From the article:

> "We eventually laid out a guide with known good versions of Linux and graphics drivers, but it didn't matter. Part of the allure of Linux is the customizability, so few actually stuck to it, and generally wanted to run the game on older hardware we didn't support."

It seems that it's because Linux users are unwilling to upgrade their hardware. From a philosophical standpoint, I agree with them, we shouldn't need to buy a new computer every 3-5 years, it's very wasteful and I am very against planned obsolescence and perceived obsolescence. That said, I can see the benefits of writing code to work on the machine you have in front of you, and not doubling your QA work by testing it on older machines that no longer even receive OS upgrades.

[+] fixermark|7 years ago|reply
But if the set of best practices that works for your target hardware platform is narrow-band, that set itself can severely constrain the set of game houses that can develop for that platform.
[+] zeroname|7 years ago|reply
I think you're in denial.

If developers need to get this, that and the other thing right specifically for Linux, but it's still only 0.1% of sales, then it's just not worth doing - even if there aren't any bugs.

[+] FigBug|7 years ago|reply
I work for a small company that produces a DAW and VST plugins. Supporting Linux is a huge amount of work compared to Windows and Mac.

The main issue is that 'Linux' is not a thing you can support. You have to pick the distros you want to support, and then once you've picked a distro, what versions you want to support.

And you need to use the C++ version that ships with that distro, so if you want to support old versions then the entire project is held back from using latest C++.

And distos aren't backwards compatible. ie when libcurl4 is released, they remove libcurl3. So you can't have one binary for Ubuntu 18.04 and 16.04.

So it means for every product, have a build for every distro / version combination you want to support.

So now the solution is AppImages where you bundle up your app with all dependencies like a container. Haven't investigated this yet, not sure it will work for plugins.

[+] plq|7 years ago|reply
> And distos aren't backwards compatible. ie when libcurl4 is released, they remove libcurl3. So you can't have one binary for Ubuntu 18.04 and 16.04.

If you don't want to depend on the libcurl provided by the OS, ship your own.

If you don't want to depend on the glibc provided by the OS, ship your own, with your own dynamic linker.

readelf -n <your binary> will tell you the oldest kernel that will run your stuff. Put that in the requirements document. Write a bash script that sets LD_LIBRARY_PATH correctly and make sure that's the main entry point to your application during deployment.

You're set.

[+] chrisseaton|7 years ago|reply
> So now the solution is AppImages where you bundle up your app with all dependencies like a container.

But isn't this what you're doing with Windows? No version of Windows comes with libcurl (as far as I know) so you put the DLL in your application directory. It's no different.

[+] vortico|7 years ago|reply
I work for VCV, which also ships for Linux. I don't find it an issue at all, since the build system is a Makefile supporting Mingw64/MSYS2 on Windows, Mac, and Linux.

We statically link everything except glibc, which we've decided to dynamically link to glibc 2.23 (meaning that Ubuntu 16.04 is the oldest we support). We've had no problems with this approach, although the disadvantage is that we have to use an old version of GCC to compile the software, since I haven't figured out a way to make a new GCC version produce binaries which link to old glibc.

No need for containers or anything, just make a mostly-static-except-for-glibc binary.

[+] zanny|7 years ago|reply
That sounds like more something to blame the Debian family over than the whole ecosystem. I might be running "bleeding edge" on Arch but I can still depend on and use libcurl3 with the libcurl-compat package.

I maintain about two dozen software packages in the AUR, including some really old stuff like the Heretic 2 Linux release from 1999 and RBDoom 3 BFG which has a boatload of dependencies. Breakages are extremely rare for the average package even with the rolling release and generally any breaking change in a common library will see the legacy version hang around since stuff will still depend on it.

[+] vetinari|7 years ago|reply
Why not pick a "distribution" like org.freedesktop.Platform/18.08 (i.e. pick a flatpak runtime) and support that? It runs on all desktop distributions. Updating supported flatpak runtimes is then on your leisure, not on their release cadence.
[+] chme|7 years ago|reply
Proprietary software on Linux is a bit of an alien there.

But there is always static linking, is it not? (And now flatpak+snap+...)

[+] jcelerier|7 years ago|reply
I am developing an open-source DAW with Qt (https://ossia.io), to solve the problem you mentioned I used AppImage with great success - of course you have to ship all your libs, but that's what you have to do on macOS and Windows too anyways.
[+] hkt|7 years ago|reply
I concluded ages ago that no substantial applications should be integrated into an operating system. Dynamic linking if you're building something portable is just an awful idea because of how much variety you'll inevitably have to support (or choose not to support).

Some people have written about AppImages, this also applies to FlatPak and similar techs too. Isolating anything more complicated than a command line tool is the way forward and the tech exists. Not using isolation this way is a recipe for pain, whether it be on the desktop or the server (or the phone).

It depends how the plugins are loaded - it'd be great if they could use sockets so that AppImages were viable. No idea myself, though.

[+] EamonnMR|7 years ago|reply
Music software's dependance on Windows is such a mess. Tons of cool plugins are stuck as windows-only VSTs! I was really hoping that Propellerheads would use the fact that it can ship on any platform to ship Reason and all associated REs on linux or in-browser, but instead they gave in and added VST support.
[+] mrspeaker|7 years ago|reply
The DAW isn't Bitwig is it? (just because your username has the same amount of syllables ;) If so, then PLEASE keep up the great work supporting Linux. I was only able to ditch my Mac a year ago because I switched from Ableton Live to Bitwig!
[+] ShakataGaNai|7 years ago|reply
While I love Linux, I think this is a totally fair "state of the union". At the end of the day, gaming or kickstarters or similar... are a business. You need to figure out your ROI (Return On Investment). It doesn't many any logical sense to spend a huge amount of time on a platform that nets you tiny amounts of sales (and gains you a huge amount of support costs).

This is why gaming on the Mac is only now starting to be a real thing. Not a decade or so ago, there were still relatively small numbers of people using a Mac. It cost a lot to develop for a new platform and didn't make much money. Now it makes more sense.

Of course Linux fragmentation will probably hold it back for significantly longer.

[+] zanny|7 years ago|reply
Is it really fragmented? You support Ubuntu (in either its flatpak runtime or the Steam runtime) and you let everyone else figure it out on their own. Distros like Arch have been very good about providing the Steam Native Runtime and Ubuntu flatpak runtime in its own package system for just that reason.

The support burden for developers is proportional to the operating systems you officially say you support. If you support "Linux" you are supporting thousands of completely dissimilar execution environments. If you support Ubuntu, or Steam, or a Flatpak target you are only supporting that one operating system.

And thats fine. Thats all Linux users really want anyway. If it breaks on your distro its your responsibility to fix it so long as it works on its officially supported target OS.

[+] rusk|7 years ago|reply
> This is why gaming on the Mac is only now starting to be a real thing.

Is it? I mean, there's a good amount of support for gaming on the Mac, but I think it's always been so ... I wouldn't call it "a thing" though ... not in comparison to PC or Console ...

[+] icebraining|7 years ago|reply
They said the problem wasn't fragmentation of distros, but the versions of graphic drivers, and it's not like Windows machines are all on the same version of the same driver.
[+] mrspeaker|7 years ago|reply
This is just an out-of-context tweet (well, the context is there - it's a reply to another tweet, which is out of context)... so perhaps it was just auto-crash reports, but I wonder if you get more tickets from Linux users, because Linux users are more likely to file tickets?

When something crashes on my partner's computer she yells at it and restarts it. When something crashes on my computer I spend an hour trying to figure out WHY. If I can't figure it out, I log a ticket thinking I'm being a good citizen!

[+] diffeomorphism|7 years ago|reply
https://twitter.com/bgolus/status/1080544133238800384

Follow-up tweet by the same author:

> As a follow up to this, I've been told by those actually involved with Linux stuff that this wasn't true. I probably just stopped paying attention to Linux issues at a time when everything was broken.

Just so the discussion does not overly focus on the (apparently wrong?) numbers.

[+] 60654|7 years ago|reply
This is a pretty well-known effect in gamedev. The specific numbers change, but Linux sales are only a small percentage of total, while generating much more development and support work.

At least dev work can be drastically simplified with middleware like Unity, which is how small studios can even consider Linux. But support difficulties are real. Players have so many possible distros, configurations, and drivers - actually supporting all of them would require a level of Linux expertise that gamedevs simply don't have (being typically Windows or OSX based themselves). Limiting yourself to something like Ubuntu LTS helps a bit, but there are still plenty of gotchas.

So it becomes a simple matter of numbers: given relatively small amount of extra sales, is it worth the extra work, and spending the time to learn Linux development and administration in sufficient depth? Sadly usually it is not.

[+] SteveNuts|7 years ago|reply
Gaming is the #1 reason I haven't fully switched to Linux on the desktop at home (at work I exclusively run Linux).

A lot of my Steam games work flawlessly with Linux but others will just silently fail to launch and it takes hours to properly diagnose and fix, which isn't what I'm after when I get some time to play games.

[+] city41|7 years ago|reply
I've switched to 100% Linux at home due to Steam Play. All of my Windows games (except 2) work perfectly on it. I really hope Valve keeps improving it, it has the potential to be a game changer for Linux gaming.
[+] jenscow|7 years ago|reply
Your comment is the perfect example of how there's an "invisible" demand demand for gaming on Linux.

Nobody can see it, because those who want it will mostly make do with Windows.

[+] Slaul|7 years ago|reply
I'm in the same boat. I'd love to be done with windows but since I spend a large amount of my free time playing games with friends I need to be on windows.
[+] tpush|7 years ago|reply
The situation seems to be similar for Mac: https://twitter.com/bgolus/status/1080327518983315457

> Yeah, I didn't want to touch on Mac support in the original post, but Mac support also cost way more than it made in sales.

[+] andai|7 years ago|reply
Innumerate person here, does that mean that supporting Mac is a net loss for a business?
[+] hawski|7 years ago|reply
Please mark it with (2014):

https://twitter.com/bgolus/status/1080380532108619776

> The world of 2014 Linux graphics drivers was not a friendly place. We absolutely encountered issues where some driver revisions only worked on certain distros, and AFAIK we did not use anything specific to a Linux distro.

[+] sz4kerto|7 years ago|reply
"I didn't want to touch on Mac support in the original post, but Mac support also cost way more than it made in sales."
[+] AnIdiotOnTheNet|7 years ago|reply
> Issues specific to Linux were almost entirely graphics driver related, and unique to the platform.

Just want to point this out before this discussion starts blaming the devs for 'not writing with portability in mind' or other such nonsense like the twitter thread did.

[+] bryanlarsen|7 years ago|reply
Part of the problem in 2019 is that the vendor with the best drivers on Windows is the vendor with the worst drivers on Linux. (aka Nvidia).
[+] Glyptodon|7 years ago|reply
I don't think the Linux situation is good per se, but I haven't heard anything this negative from the devs of other multi-platform Kickstarter games, like Obsidian and their Pillars of Eternity games.

(Not that they were terribly positive about it.)

To paraphrase the Obsidian take:

Supporting Linux for the first time was much much more work than they expected, but it was mostly coming to grips with everything, learning all the gotchas, and integrating that into their build system and software dev practices. Once they got over that hump it was much easier/not so bad and worthwhile enough to continue. But at the same time, if they'd known what they were getting into the first time they think they'd probably not have promised Linux support. So first game it had a huge marginal cost to support, but once the tooling and experience was there that marginal cost to support was much much lower.

Of course Pillars was a Unity game and probably not as demanding performance-wise as PA.

[+] bluescarni|7 years ago|reply
> We started working on the game in 2012

> Issues specific to Linux were almost entirely graphics driver related, and unique to the platform.

So this is essentially a rant about linux graphics drivers circa 2012?

[+] madez|7 years ago|reply
I'll take the step and say that most developers do the mistake to treat linux like any other proprietary system which they can target with stable interfaces and call it a day.

Free software changes more often, but when it does, it is not that big of a problem because you have both control over code and the code that your code interacts with, because all code is accessible.

If game developers opened up, and here I don't demand the optimum, which would be open sourcing their game[0], but just to get people involved that actually are active in both their code and the free software community, they could deal with the eco system much better and make their code actually work even on most of the many configurations that exist.

Of course, the usual argument is that game developers don't have to care about this on other systems, but in fact they did have to gather experience and training on the other platforms, and with free systems, once you are involved in it, you can deal with it much more efficiently than what it seems at the start. There is nothing magic about filing a bug against mesa.

I am one of the persons that contacted Uber about PA not working on my linux system. It was a recent Debian stable without modifications. They didn't offer meaningful support. I just asked them to install a debian stable with free mesa drivers on a computer with a recent AMD gpu, and try to build and run PA, and see why it doesn't work (there were obvious, easy to reproduce issues). That is not a lot of engineering time, and I bet solving those problem would have solved many problems in linux systems.

[0] This starts to become a cultural problem that seems to need laws to deal with. As a society, we shouldn't accept to never have our cultural heritage, which games are part of, in the public domain in a usable form (source code and documentation).

Edit 1: Writing closed programs for a free software community is difficult. I think that it should be difficult. That pain is a constant reminder that you're doing it wrong.

Edit 2: Steam counts unsuccessful launches of a game where the game is laughibly broken as playing time, so after some extenstive testing, you can't even refund a game on Steam because it says you've played more than 2 hours -.-

[+] schaefer|7 years ago|reply
The flip side of this is: I only game on Linux. If your game doesn't run well on Linux, chances are, I'll never play it.

There's already a back-log of games on my wish list that I'll get to as soon as i find the time.

[+] dejaime|7 years ago|reply
I was one of those who for every problem I had I would go out of my way to reliably reproduce the problem, then I would create a detailed description of it, with full system specs, terminal output (on both Windows and Linux) and all log files I could find.

Today, I still do that on Linux. But if it excludes Linux as a platform, it must be rock solid on windows, right? If it isn't, all it gets is a negative review going about the "crashes all the time", "glitchy" and "is unplayable". That saved me a lot of time.

Being an indie GameDev myself, I personally love well written tickets. But the passive aggressive responses I often receive from big devs makes me wonder if I'm actually the only one.

[+] stratosmacker|7 years ago|reply
Why don't we just support the Steam version of Debian and be done with it? Anyone running custom things should be ready to make their custom fixes...

Oh well, this guys sounds full of hubris. Just because something isn't immediately profitable doesn't mean the community as a whole doesn't benefit from you ironing out bugs.

[+] dcbadacd|7 years ago|reply
This sounds like complaining about users being able to report bugs.
[+] brianwawok|7 years ago|reply
Do you run a business, say software? Or even a non-technical business like window washing or pet walking?

If .1% of your pet walking clients made 20% of the complaints, what would you do?

[+] boomlinde|7 years ago|reply
I read it more as an admission that they underestimated how hard it is to a game to work consistently across Linux graphics drivers. The user reports are used as a metric, not to damn the users.
[+] Ajedi32|7 years ago|reply
In this case it sounds like the bugs were mostly Linux-specific ("gfx driver related").
[+] ergo14|7 years ago|reply
Well as he mentioned those were autogenerated reports. That being said... I tried Planetary Annihilation and it worked without issues for me on Linux.
[+] Tehchops|7 years ago|reply
Too bad Planetary Annihilation was a poorly optimized, buggy, under-delivered piece of shit on Windows too.