Sunday, November 9, 2014

State of Linux Gaming

I've got one more blog post before I depart for Dallas. Here's an interesting report showing framerates and loading times of various big titles on Linux vs. Windows:

Slashdot: PCGamingWiki Looks Into Linux Gaming With 'Port Reports'
PC Gaming Wiki: Linux port report

Sadly, it's pretty clear that if you run these games on Linux your experience isn't going to be as good, and you'll be getting less "gaming value" vs. Windows. We're not talking about a bunch of little indy titles, these are big releases: Borderlands: The Pre-Sequel, Borderlands 2, Tropico 5, XCOM: Enemy Unknown, Sid Meier's Civilization V. My take is the devs doing these ports just aren't doing their best to optimize these releases for Linux and/or OpenGL.

A nice little tidbit from this report: "Unfortunately, Aspyr are currently still unable to provide support for non-Nvidia graphics cards, as with Borderlands 2. This doesn't mean the game won't work if you have an AMD or Intel GPU, but just that you're not guaranteed to receive help from the developer - the current driver situation for non-Nvidia cards may lead to degraded performance." Huh? This is not a good situation.

I know it's possible for Linux ports to equal or outperform their Windows counterparts, but it's hard. At Valve we had all the driver devs at our beck and call and it was still very difficult to get the Source engine's perf. and stability to where it needed to be relative to Windows. (And this was with a ~8 year old engine - it must be even harder with more modern engines.) These devs are probably glad to just release anything at all given how alien it can be for Windows/Xbox devs to develop, debug, and ship stuff under Linux+OpenGL.

Hey, this is just a thought, but maybe Valve developers could stop locally optimizing for their bonuses by endlessly tweaking and debugging various half-broken dysfunctional codebases and instead do more to educate developers on how to do this sort of work correctly.

The entire Intel driver situation remains in a ridiculous state. I know Intel means well and all but really, they can do better. (Are they afraid of pissing off MS? Or is this just big corp dysfunctionalism?) Valve is still paying LunarG to find and fix silly perf. bugs in Intel's slow open source driver:

Major Performance Improvement Discovered For Intel's GPU Linux Driver

Surely this can't be a sustainable way of developing a working driver?

Anyhow, onto SteamOS/Steambox. Here's a surprisingly insightful comment I found on Slashdot. I don't agree that SteamOS is done just yet, but you've got to wonder what is really going on. (So where are all those shiny Steam machines they showed earlier this year anyway? Does all this just go into the Valve memory hole now?)

by Qzukk (229616) on Friday October 24, 2014 @11:56AM (#48222551Journal
Let's be honest, SteamOS is done. Steam got exactly what they wanted from Microsoft and dropped it like a hot potato (so sorry, you'll never get to use that cool controller).
Consider that for decades Microsoft has not allowed anyone, anyone to touch the user experience. Even after Netscape's antitrust lawsuit over active desktop, even after BeOS withered and died hoping someone would sell a windows computer with dualboot, or hell just a windows computer with a "Setup BeOS" icon on the desktop. Steam is facing the Microsoft Store and a real threat that the Microsoft Store will become the way to buy programs (see also: iOS). Steam trots out SteamOS, and Microsoft snickers. The hype train builds up, and Microsoft sweats. Games start to port and Microsoft snaps.
Alienware ships a Windows 8 PC that boots to Steam instead of Metro.
Now, let's step back a second and look at the big picture here. At the time, windows 8 adoption is absolute total shit, swirling the drain of a public restroom that hasn't been washed for years. The last windows evangelists are all hanging on imploring people to just try it out, just give it a chance, and oh by the way install Start8 to fix metro. Think about that. PC vendors are on the verge of revolt, their customers refuse to buy their goods, and all for the want of installing a $5 program to fix the metro experience. Best Buy is probably screaming at Microsoft, begging them to allow them to remove the metro experience so they can move their inventory. Hell, they're probably begging them to let them advertise their Geek Squad services to "optimize" the experience and install that $5 program for $100. But no, the Microsoft Experience is inviolate, the holiest of holies, eternally immutable. No matter how much hatred it gets, it Must. Not. Be. Changed .
And then Alienware ships a Windows 8 PC that boots to Steam instead of Metro.
SteamOS's job is done. When no-one was looking, Steam took Microsoft and snapped it like a twig. We'll never know exactly what dark magicks were invoked here, but in the blink of an eye, Valve routed Microsoft in a war that nobody even realized was being fought. When Japan makes an anime out of this event, GabeN will point at Steve Ballmer, say omae wo shindeiru and Ballmer's head will implode, without GabeN throwing a single visible punch.
Steam OS will probably putter along, we'll probably see a few things be trotted out to keep the dream alive, after all the hype train did build up a lot of steam (pun not intended). Eventually a few of these AAA developers will say "it's really just not ready for the prime time" and we'll go back to getting a few wine ports and indie games from hardcore dedicated guys who just really love Linux.
But the masses will probably never get to hold that controller.

27 comments:

  1. I'm not exactly sure that it "must be even harder with more modern engines". I'd rather say it "must be even harder with more modern PRE-EXISTING engines".

    Modern OpenGL is about explicit fences, texture arrays, unsynchronized mapping (GL3)/persistent mapping (GL4), and multidraw. There's literally very little where a driver could screw up or stall due to the GL calls getting scarce.
    Although the lot of historical legacy baggage (which force validation overhead or otherwise unnecessary stalls) and scarcity of examples showing the modern paths don't help.

    As I've been porting the Ogre engine to more modern GL, many times has crossed my mind the thought "oh man, I'd wish I were starting this whole thing from start" (and assuming GL4 as minimum requirement would be another awesome assumption).
    We ended up designing a 'new' system that lives side by side with the old one; split by RenderQueue settings (i.e. user can ask RenderQueue 0 => use new fast system, RQ 1 => Use older slower system).
    Granted, Ogre 1.x is not the best example of a modern engine, but the leap from old ways to AZDO GL is very similar to going from D3D11 to D3D12.

    Using the modern GL renderqueue, our performance gap between Windows and Linux minimized. So did the bugs and crashes. Although we're far from finished, thus conclusions shouldn't be made yet.

    What seems to be hurting Linux gaming badly are X11 and the DRM stack. X11 is a mess. The DRM isolates the user space program from the hardware too much.
    Because of these, most drivers are struggling to get something as basic as propper VSync for God's sake. In Ubuntu (Unity) it's really hard for a GL program to take over the screen (aka exclusive fullscreen), and when you do, a crash or hang is really hard to recover from (because the screen is basically unresponsive).
    While getting VSync, too often I've seen my applications or WebGL browsers locked up to 30 fps instead of 60, because of some really weird interaction with the compositor or the windowing system, or who knows why.
    Wayland is suposed to replace X11, but we're still waiting for it. It's not like adoption is going to happen in an afternoon.

    The trip data travels starting from user-space through the graphics driver until it reaches the GPU on Linux is so long... there are so many things that can go wrong in the middle. THAT is the problem, in my opinion.
    Windows had this very clear defined and had it redesigned multiple times. It wasn't a popular choice (DX10 Vista only, then WDDM 1.2 on Windows 8+), but it shows it's advantage over a windowing system that was designed in the 80's and last time rewritten from scratch was in the late 90's.

    ReplyDelete
  2. Just being practical here: I wonder how many devs will spend the time (like yourself) to properly create an AZDO backend for what will probably be a single digit percentage of the marketplace? Also, until AZDO is available and working everywhere that matters (OSX, and Linux with all relevant drivers) devs will have to create and maintain two different GL backends (one core, one with AZDO). Sounds like a ton of work to me.

    Maybe Valve will release AZDO enabled Source engine games, and a AZDO togl layer (but hey I'm not holding my breath).

    ReplyDelete
  3. The entry barrier to create an AZDO backend is huge. However it turns out most of the Azdo code that matters can be used in GL3 and DX11 as well (and DX12); because it's not really a single feature, but rather a collection of extension and techniques.

    Azdo is about merging all meshes into one buffer. Under DX11 buffers are separated by category (vertex, index, texture buffers, etc); however the merging approach can still be used.
    Explicit memory management makes easier to perform partial updates on those big buffers, which is something that fits naturally well for GL3, GL4 and DX12, but requires a little more effort on DX11 (on static buffers, when possible, use MAP_NO_OVERWRITE, otherwise resort to staging buffers; or delay initialization as long as possible so immutable buffers can be used); but it still benefits all APIs. Even GLES2 can get benefits with certain limitations though.

    Azdo is also about reducing overhead when setting textures. Bindless textures is only available on GL4/DX12, so we've decided for the next best thing that is texture arrays. This translates well to GL3 and DX11 as well.

    With some clever design, buffers can be mapped using persistent mapping and non-persistent methods automatically depending on hardware support (non-persistent mapping ends up mapping the buffers usually once per frame, like any non-Azdo application would do).

    For togl I noticed glMultiDraw* was not in the whitelist. But since we build a simple emulation layer for it (so that it can be used in GL3), tools like togl can be supported with help from our side.

    The benefits of an Azdo-enabled pipeline is that it works well to the other APIs for the most parts; and may improve peformance on them as well; whereas the opposite doesn't held true (i.e. a DX12 Azdo-hostile pipeline will likely not run efficiently on modern OpenGL)

    TBH I see Azdo GL and DX12 as converging. DX12 just enables preexisting code to run faster giving lower access and access to more indirections, but Azdo practices takes them one step further in optimization (takes away a couple of indirections). GL on the other hand *forces* programmers into using Azdo or die from API overhead.

    We're playing/betting the longer game and going Azdo for all the APIs. I expect future games to eventually start using practices now labelled as "AZDO", but not because of OpenGL, but rather as an optimization step.

    Well, I'm getting derailled here. The original post is about Linux Gaming. And whether it's Azdo or not, it has a lot of work to do. It doesn't matter having zero driver overhead when the OS/driver/x11 for any reason decides to lock your framerate to 30fps because of bad VSync (and tearing still visible, who knows why!)

    ReplyDelete
  4. Good comments Matias; it's obvious you know your stuff here.

    Yea, togl was just doing DX9 plus some GL mode specific backdoor API's for things like more efficient uniform updating, so even if we added glMultiDraw support the higher level engine wouldn't use it. Also, each title in the catalog typically used a different branch of the Source engine, so hacking the higher level code to use the MultiDraw path (and keeping it working - because there are no testers, or testing process or culture at Valve) would have been some unforgiving work.

    NVidia devs and I did toss around the idea of deferring up all the DX9-style API calls and trying to intelligently batch them to reduce driver calls. For example, we could have dynamically created GLSL uber shaders from multiple individual DX9-shaders, and the togl layer could have quickly switched between them by setting uniforms (instead of selecting entirely different programs). I even did a quick proof of concept profile pass to see just how well we could cut down the # of shader and other state changes, and the results looked promising on AMD and NVidia. I think it would have been doable but we had bigger fish to fry at the time (like shipping the thing at all).

    ReplyDelete
  5. > it's really hard for a GL program to take over the screen (aka exclusive fullscreen), and when you do, a crash or hang is really hard to recover from (because the screen is basically unresponsive).

    This used to be a problem for years, in SDL1 and OIS input systems - over-zealous input grabbing, i.e. volume keys and alt-tab being unavailable during fullscreen. The X11 api makes it really difficult to implement a proper fullscreen. However, it appears that the SDL2 guys have virtually solved this. I've never had a full-screen crash lockup when using SDL2, volume keys and alt-tab work even when the application locks up. I'm assuming you're using a different input system?

    ReplyDelete
  6. Might be this article have to be titled as "State of OpenGL Gaming"? There is OpenGL-only games (example) that have nearly same performance on all platforms. Of course initial investment into developing effective OpenGL renderer for Windows must be huge, but once it's here there is no need in tons of platform-specific optimizations.

    Obviously "big titles" that was never developed with GL in mind likely will be always behind just because resources spent on porting it's at best 10% of what D3D version get and this understandable because as you said who care about 1% market. Only solution I'm see it's somehow engage developers to use or at least keep GL in mind. With games that using engines like Unity / Unreal having multiple renders out of box this become more or less possible.

    Also this doesn't really change fact that drivers situation on Linux is still pretty bad, but let's say fair state of OpenGL on Windows isn't any better. Not that long ago I made extended post about GL problems on Windows for OpenMW developers and here is short list of main issues that affect OpenGL 3.1 game mainly on laptops:
    1. Many users on Steam don't update their drivers even if they can do that easily.
    2. AMD Legacy drivers GL is broken: bugged, crashes, lockups. They no longer updated since December of 2012.
    3. Windows 8+ distribute AMD Legacy drivers with OpenGL removed and installer can't install it.
    4. Intel don't have driver auto-updater on Windows and on laptops there almost always "not validated for this computer" so manual install required.
    5. AMD laptop driver installer it's a mess: never works, mostly always require non-trivial cleanup and tons of time to update.

    This all doesn't hurt D3D because 2-3 years old drivers works well enough for it usually, but for OpenGL-powered game it's guaranteed grash or serious graphical corruption. It's make think why Valve for example don't want to start with improving situation with OpenGL on Windows.

    From one point of view it's would only improve Microsoft platform and D3D, but will benefit GL greatly because games using GL will actually work. Obviously this won't make console-oriented developers to use OpenGL, but at least companies which only target PC market might switch to OpenGL and this will benefit it in long term.

    ReplyDelete
  7. Perhaps after bonus season is over at Valve (around Feb./March), and assuming the right devs are even still there (i.e. not fired/cleansed due to petty politics), something could be done to improve the crappy state of OpenGL on Windows which will indirectly help the Linux driver situation. Devs publically speaking their minds about the situation could make a difference.

    Valve has started the ball rolling and can't let up now without squandering considerable developer street cred.

    ReplyDelete
  8. > Perhaps after bonus season is over at Valve (around Feb./March), and assuming the right devs are even still there (i.e. not fired/cleansed due to petty politics), something could be done to improve the crappy state of OpenGL on Windows which will indirectly help the Linux driver situation.
    If situation will be improved it's will be good news because there tons of games that can run on 6-8 years old hardware just fine and for them driver performance doesn't matter. Though when game like Broken Age crash on startup because player don't have reliable driver it's hurt GL more than just 20% lower performance.

    > Devs publically speaking their minds about the situation could make a difference.
    This is why I think what's you're doing is great thing with these articles because development of open standard impossible without public discussion and it's also interesting to watch it for non-developer. Hopefully reaction of Linux crowd won't make you upset because those guys usually don't understand how game development works.

    Though I'm think you're as other developers underestimate importance of open source drivers for long-term future of Linux as gaming platform. From my point of view fact that Valve can improve Intel driver it's good and not bad thing because there is example of AMD proprietary driver that permanently broken for every 2nd non-Unity game released on Steam and even AMD can't fix it. Other good example it's abandoned Windows AMD legacy driver while open source one on Linux have a lot better stability.

    ReplyDelete
    Replies
    1. > Hopefully reaction of Linux crowd won't make you upset because those guys usually don't understand how game development works.

      I doubt that, we Linux guys know pretty well about the state of our current graphics stack including drm, x11 and drivers. You can't get much heat for stating the obvious ;-).

      Though, I do think that Aspyr/Feral are doing a pretty good job in porting. It may not be the same performance or experience you get in Windows, but it's "good enough" for a solid and fluid gameplay. They're not in charity business either, and doing it "right" would probably consume a lot more time, considering they only have a share of 1 % of the customers who may be on linux - if they didn't buy the games before in Windows.

      Delete
  9. FWIW - while at Valve I seriously supported open source Linux GL drivers and I got seriously burned by doing so. There are some powerful concerns out there that are not yet on board with the idea of open source GPU drivers. It cost us quite some time.

    ReplyDelete
    Replies
    1. > There are some powerful concerns out there that are not yet on board with the idea of open source GPU drivers. It cost us quite some time.
      Would be great if you share those concerns with community one day. Pretty sure drivers devs will comment on it too.

      Delete
    2. Pretty sure he meant concerns as in businesses.

      Delete
  10. my projections: STEAM OS will be kept in dev indefinetly for anyone or any relative small company who wants to build their own Steam box(es)


    However. SteamOS is still important.

    The Alienware Alpha is $550, meaning that up to $100 may be from including windows...

    With SteamOS you can slash that price down and compete with consoles. THIS IS THE WHOLE GOAL: TO KEEP *these pc component* CONSOLES FROM EATING THE WHOLE PIE and GETTING ALL THE EXCLUSIVES. The notion that PCs can also be used on the couch and get more gamepad oriented games.

    Microsoft doesnt want to compete with it's own console, so I don't think they will slash the price too much.

    SteamOS/linux-machines are not ready yet. So maybe in a couple of years when the "Steampad" is ready and prices has gone down and Linux has 1.5 to 2k games (has like $700 now).

    Also Valve will need to create some "beta or time exclusives" (3 to 6 months) for those that invested in their SteamOS-machines.

    Also STREAMING COULD BECOME BIG THING, if they make some type of "cheap Amazon fire-stick-tv" like Streamer. A sub $100 device that Streams the bigger 3d titles and plays natively the indie/older ones.

    So yeah that's just my projections, I think it could happen if they're creative, competitive and stay interested in it.

    ReplyDelete
  11. One thing is Valve paid LunarG to rewrite the compiler, only after they rewrote the compiler did they find that it wasn't as fast as Windows, i.e. rewriting the compiler didn't help that much (it does help but not as much as haswell chicken bits).

    Intel Linux suffer from not being allowed to access Windows team at all due to internal Intel policys (politics?), they may not even be able to run Windows drivers for fear of evil. Also LunarG reported the slowness not the fix to Intel, Intel engineers once they were notified had the answer a few hours later. Its bad they didn't come across this themselves, and I guess that is mostly due to Intel's internal policies on team interactions.

    ReplyDelete
  12. Hello,

    Thank you very much for your informative articles. Please continue addressing the problems effectively.

    As a linux gamer, I believe that the main loser in this arena is us, the linux gamers. I do believe that the gamers do not care about which API is chosen for their games. They just want their games to run perfectly under linux. I am pretty sure that if vendor X provided the gamers with better gaming support in linux, its products would be bought even if they implemented DX12 under linux and ereased opengl for ever.

    There are some prevailing thoughts in the linux ecosystem that has effected other areas like gaming. Why on earth does anybody want underperforming open source drivers for their high end graphics card? Do we buy a graphics card which can just run games well or do we want opengl and/or open source drivers? There is too much ego injected by some old, linux-is-only-for-servers and linux-is-only-opensource thoughts into every topic including gaming under linux. The gamers do not care about these things. All of us need to get these thoughts off gaming.

    Come on venders and server admin and developers! We are gamers! We want good game support, period. Implement whatever you want in linux as long as it plays well.

    While the users are the losers, in this arena, they are still doing the best thing: Support. We are realistic and supportive. We pay higher than what other OS users pay for whatever is available. I personally play AAA games but I always try to buy whatever Steam provides for linux -- just to support the trend. About the linux market share, it must have been much lower, but the users are keeping it up around 1.5%. We are doing everything here on our side.

    Finally, If you have any advice for the linux gamers, we would be so glad to listen to them.

    Thank you again.

    ReplyDelete
  13. Non-expert here, wondering how much / what parts of this situation improve if Wayland adoption proceeds as well as can reasonably be expected, eg several major distros are using it by the end of 2015?

    ReplyDelete
    Replies
    1. Not much will change because Valve/SteamOS are fully invested in X11 and no game dev would be insane enough to break off compatibility with a chunk of the already small 1% pie. Xlib will work on both X11 and Wayland.

      Delete
    2. It's basically a non-factor because the majority of Linux gaming's issues are unrelated to X11 and those issues it does cause are not fatal / can be worked around or made into somebody else's problem with things like SDL.


      Delete
  14. This comment has been removed by the author.

    ReplyDelete
  15. Valve is still paying LunarG to find and fix silly perf. bugs in Intel's slow open source driver [...] Surely this can't be a sustainable way of developing a working driver?

    It indeed can't. But don't call me Shirley.

    ReplyDelete
  16. Just ship cheap (Iand not cheap too) Steam Machine consoles with Nvidia Maxwell, as happened with Android or Chrome OS, the market like cheap and good, and some would notice the OS works far better than others. Glamour is working faster than directx in MS WOS, it can be the way for AMD and Intel GPUs SMs but they will late, and of course Nvidia would love to improve their good enough GNU/Linux drivers if consoles are shipping vs MS and Sony AMD consoles, perhaps they develop a even cheaper and better console to compete with AMD ones.

    ReplyDelete
  17. Selected tidbits:

    Unfortunately, Aspyr are currently still unable to provide support for non-Nvidia graphics cards, as with Borderlands 2

    So, Nvidia has the only working driver out there. But Nvidia is evil for keeping it closed source, right?

    I know Intel means well and all but really, they can do better. (Are they afraid of pissing off MS? Or is this just big corp dysfunctionalism?) Valve is still paying LunarG to find and fix silly perf. bugs in Intel's slow open source driver:

    So, Intel slapped-together a POS driver in their free time (understandable, with the Desktop Linux community being tiny and toxic - see above), and just because they unloaded that pile of bad code on the pavement for everyone to see, they "mean well"?

    I have trouble following that reasoning. Of course, other people may have trouble following my reasoning, because I am making remarks on the state of Desktop Linux in 2014.

    My take is the devs doing these ports just aren't doing their best to optimize these releases for Linux and/or OpenGL.

    Devs have no problem optimizing for Android's OpenGL (and OpenGL ES). Only the broken OpenGL implementations of Desktop Linux (minus the Nvidia one) are a challenge for devs.

    PS: Can you imagine if Google allows ChromeOS to run Android games and Android apps? It probably won't happen, save for a few "social" apps, but what if? For Desktop Linux, it will be a clear case of "last person to leave, please close the door".

    ReplyDelete
    Replies
    1. Uhm, are you a troll?

      > > Unfortunately, Aspyr are currently still unable to provide support for non-Nvidia graphics cards, as with Borderlands 2

      > So, Nvidia has the only working driver out there. But Nvidia is evil for keeping it closed source, right?

      No. Plenty of people have been playing it on radeonsi from pretty much day one with okay performance.
      If you google borderlands 2 radeonsi, you will probably find my video I made 1.5 months ago: https://www.youtube.com/watch?v=EKC0p7Zol7s
      The somewhat low performance seem to come from my hybrid gpu setup that has had plenty of performance problems in windows too at the beginning. The tearing will also only happen on PRIME + dri3 AND when the performance is bad.
      You can watch other peoples' videos to see much better performance on radeonsi.

      So no, nvidia does not have "the only working driver".

      > So, Intel slapped-together a POS driver in their free time (understandable, with the Desktop Linux community being tiny and toxic - see above), and just because they unloaded that pile of bad code on the pavement for everyone to see, they "mean well"?

      Uhm no, Intel has the most developers working on the open source driver stack and they indeed have developed and are still developing a lot of the important infrastructure that is used by other drivers.
      https://en.wikipedia.org/wiki/Free_and_open-source_graphics_device_driver#Intel
      "Intel has 20 to 30 full-time Linux graphics developers"
      They do not "unload a pile of bad code", they have been actively improving it for years. They are also supporting it: there are usually always intel people availabe to ask questions in #dri-devel or #intel-gfx and usually they are very helpful in the case of problems.

      > Devs have no problem optimizing for Android's OpenGL (and OpenGL ES). Only the broken OpenGL implementations of Desktop Linux (minus the Nvidia one) are a challenge for devs.

      It's funny because when you read what the actual developers say, then the closed source drivers for android are terrible and mesa is decent. For example the dolphin developers: https://de.dolphin-emu.org/blog/2013/09/26/dolphin-emulator-and-opengl-drivers-hall-fameshame/

      The question is: Are you intentionally getting everything wrong?

      Delete
  18. I'm not a dev guy just an avid Linux enthusiast. Thank you very much for this educational insightful article and engaging the dev community's discussion here. I honestly feel the wiser for stopping by and hope things improve.

    ReplyDelete
  19. Hey, Rich, what do you think about the steam controller? Is it true that most of the devs at Valve are happy with it?

    ReplyDelete
    Replies
    1. I played Portal2 using the steam controller a bunch - never really got used to the touchpad things myself. I'm old school - I like the Xbox360 controller.

      Delete
  20. The open source drivers are badly written, poorly functioning. They don't support Windows, they won't support Linux, they don't work with Open CL or even meet OpenGL 4.5. Pressure needs to be laid on the useless parasites at AMD, Intel and Nvidia.

    https://www.youtube.com/watch?v=IVpOyKCNZYw

    Linus speaks for us all. You guys suck!

    ReplyDelete