Pandemonium

January 22, 2009

It’s not the size that counts

Filed under: Games Development,Games Industry,Personal,XNA — bittermanandy @ 10:37 pm

Indulge me as I walk down memory lane…

It was summer 2002 and I’d just landed my first professional job. (I’m still not sure how. I wasn’t very good back then. I like to think they saw that I’d become good, and I like to think I am good now. It might just have been luck!). I’d spent some time in the RnD department, to learn the ropes, become familiar with devkits (console hardware with added stuff to enable game development), and understand a bit about how the shared game engine worked; and then the time came to move onto a game team, of which there were five or six to choose from. The choice wasn’t freely given. Each team had different needs, so the question was asked: what did I want to work on? What did I want to specialise in?

I didn’t really know how to answer that. I was vaguely aware of the division between systems programming and gameplay programming, and knew that I preferred the former (I actually think the latter is better solved by a good data-driven system and a good designer, though I have worked with gameplay programmers who produced excellent results); but which system? And how would my choice affect my working day?

Games teams on “AAA” titles nowadays can easily have several tens of programmers. Clearly, if they all worked on whatever they fancied from day to day it would be a disaster. Each programmer therefore gets designated an area of responsibility. Generally speaking, the programming lead and the most senior programmers determine the overall architecture of the game early in development (or it may be mandated by the engine, particularly if it is middleware), and the programming team splits into several sub-teams, each led by a senior programmer who reports to the lead programmer. Examples of roles in the team (in no particular order) include:

Graphics: the celebrities of programming because they get to write code that produces awesome looking screenshots (or at least… code that lets the artists do so). Every time the publishers come for a visit, they’ll get led into the graphics programmers’ office and shown all the latest particle-laden explosions on flashy HDTVs. Graphics programmers spend a lot of time writing shaders, optimising the renderer, and talking with artists.
Networking: modern games are immensely complex and with multiplayer online being practically compulsory nowadays, every game will have network specialists. They tend to spend all their time trying to teach other programmers how to write code that doesn’t break the online mode, for example by sending 600KB packets every frame or updating something on the local client but not the game server. They usually look a bit stressed.
Physics: even with middleware like Havok (or in XNA, JiglibX and the like) available, physics remains one of the most complicated things in a game because it affects just about everything else. One game I worked on had five physics programmers.
AI: most games have enemies or some kind of non-player entity. While scripting and other designer-facing tools mean AI is not as hard-coded as it once was, someone’s got to write the code that interprets the scripts – that’s the job of the AI coder.
Audio: given that the only two ways your game can influence the player is via the screen and the speakers, audio is half of every game. Unfortunately it’s the second half (because it doesn’t look good in screenshots) and all too often audio is neglected. Done well, it can turn a good game into a mind-shatteringly atmospheric epic. Audio coders spend a lot of time talking to the musicians and SFX engineers, and they’re usually slightly bitter that the graphics programmers get all the plaudits (and flashy HDTVs).
Tools: there might be twenty programmers on a team, but there might be ten designers, fifty artists, and five audio engineers (as well as testers, producers, marketing, translators…). You can’t just give them a copy of Photoshop and Maya and tell them to get on with it. Every game needs specific tools that enables these people to get their assets into the game and tweaked until fun and in my experience, the better the tools the better the finished game.  Historically tools were DOS-based and unreliable; increasingly, they’re now written in C#/.NET and actually work more often than not. Tools programmers are the unsung heroes of the programming team. Their work is almost never seen by the public, but without them, the game itself won’t get seen by the public either.
Systems: asset loading. The game camera. Multithreading. Save games. Text, and menus. Achievements. DVD file layout. TCR compliance (rules that the console makers require you to obey before your game can be released). Support for steering wheels, dance mats, webcams and chatpads. The build process (putting together versions of the game to give to artists, management and testing). A veritable pot pourri of tasks that no game can go without. Some of these tasks will be given to the most senior programmers because they’re critical to the game’s success. Others will be given to the most junior programmers because they’re relatively self-contained and can be developed in isolation. Most don’t get noticed, until you try to make a game without them!

There’s more, but that’s a good initial summary. (Just think – when you write an XNA game, you’re responsible for all of the above! Lucky XNA itself is brilliant at doing it loads of it for you). My first ever task on a game team was to write the game camera, and looking back I’m pretty proud of how it turned out. Very soon afterward I also took on the audio programming. Eventually, on that first game (I worked on it for three years – some were working on it for twice that), I was responsible for, or otherwise involved with, text and menus, localisation (making the game support other languages), asset loading, the build process, save games and TCR compliance. Later I’d work on asset optimisation and arranging files on the DVD, and probably some other stuff I’ve forgotten. So it became clear: my specialisation was that I was a Jack-of-all-trades. Hurrah!

With all these people working on different things, it’s critical that they don’t interfere with one another by writing over one another’s changes. This is achieved by use of a Source Control System. Basically, this keeps track of every code file and asset in the game, keeps records of how they change over time, and tries to ensure that if two people make changes at once, those changes are seamlessly merged together. Different teams use different products and approach this in different ways. Some examples using the codenames of games I worked on:

Game Two: used CVS for source control. This is an horrific abomination that should be scourged from the earth. A coder would make all the changes he wanted, send out an email to the team saying “please don’t commit”, commit all the files he’d changed, and send another email saying “OK to commit”. Inevitably he’d have missed something so the next coder to update would have to come and ask him to fix the problem before they could continue. Worst of all, the artists couldn’t get anything into the game without giving it to a programmer to commit it for them. This was frustrating for everyone involved and meant that changes to a level, for example, could take two or three days to get into the game. Putting together a build was an eight hour manual process; all too often it wouldn’t start until 5pm the night before a deadline. I’m not completely sure how we managed to get the game finished, and I’m stunned that it turned out as good as it did despite everything. This is How Not To Do It.

Pocket/Pikelet: an improvement in every area, these teams used Source Depot (basically the same as Perforce) for source control. No more emails to control who could commit – a tool that lived in every developer’s System Tray would lock out commits while someone was going through the process of updating, building the game, running the unit tests and committing. A separate build machine would then automatically update to that version, run the tests again and if they were passed make the build available to all the non-programmers – who, incidentally, had the tools available to do all their work without needing to go through a programmer. It was brilliant, broken builds and artist downtime were unheard of – the game was bulletproof throughout development. There was only one small problem. Updating, building, running unit tests and committing took half an hour or more. In a normal eight hour day, only sixteen programmers could do it – at most. With twenty programmers on the team, there’s an obvious problem and it got very frantic near deadlines – and leaving at least a day between committing meant that you’d commit too much at once, introducing bugs (which would fail the unit tests and delay your commit even longer). This was a problem, but in general this was the best system I’ve had the pleasure of working in.

Polished Turd (not the real codename, just what I call it): again this used Perforce for source control, except this time without a formal commit queue like Pocket/Pikelet. The commit queue had originally been introduced to stop CVS from breaking everything, but Perforce is so much better than CVS that actually things rarely break even if you’re free and easy with committing. It meant that you could fix a bug, commit your changes, fix another bug, commit your changes, and iterate very rapidly through your work. You’d update over lunch and overnight, or when you noticed someone had committed something you need. If we’d only had the same team build system, unit tests, build suite, and artists tools from Pocket/Pikelet this would have been the ideal way of working. (Unfortunately all those things didn’t exist so the game was unstable, the artists couldn’t do their job, and every deadline was a mad scramble to put together get something vaguely playable that lasted more than five minutes between crashes).

Now, most people reading this will be hobbyists working in XNA on their own. At the moment, that applies to me too. However, I’m uncomfortably aware that the big blue blobs and soulless grey polygons that currently represent the actors and props in my game won’t inspire other people to play my game – and I’m closer to being autistic than artistic. So sooner or later, I’m going to need other team members to make models, textures, animations and sound for me. (Any volunteers?) And, while I’m currently planning to save all the coding duties for myself (with the XNA Framework itself and the multitude of excellent community libraries out there, this is a realisitic proposition in a non-trivial game for really the first time this side of about 1998) many of you will no doubt be thinking of forming small coding teams, to split the work among you. (I don’t blame you. Just look at the list above. Even a simple game has a lot to do…) So – how well does XNA support medium-to-large game teams? And what would such a team require?

Source control is absolutely essential the moment your team grows larger than one. And, with a team of one or two, there’s only one choice – Perforce, which is free for up to two users, and so good I use it even though no-one’s forcing me to. Unfortunately, as soon as your team size hits three people, it’s something like $700 a license. Ouch. Worth absolutely every penny and more if you’re a pro developer with your outgoings covered by a publisher, but out of reach of everyone else. I am told that Subversion is good for a free product, but I look at the lack of atomic changelists and cringe. Probably the next best if you can’t afford Perforce though. There’s nothing here that’s different for XNA than if you were using C++ or any other language.

It will be essential to have a configuration of your game that allows your artists to put their assets into the game without waiting for you to do it for them. The XNA content pipeline is a wonderful, wonderful thing of great beauty, but by default it is tied into Visual Studio – the content pipeline build occurs just before you run your game, and that’s it. An artist doesn’t want Visual Studio, and he doesn’t want to restart the game every time he changes a texture – he should be able to save off the texture file and see it change in-game. So you need to provide them with a version of the game that runs on its own (perhaps in a WinForm) and a tool that hooks into the content pipeline and will let them build assets while the game is running, notify the game, and the game then reloads that asset. This is pretty easy for assets that already exist in a content project, but the tool will need to support adding to the content project (hidden from the artist) and even creating new ones. All this needs to hook into your source control system without your artist needing to learn to type “p4 -d -a -q -z” or whatever.

The content pipeline causes a few other problems too. (Though don’t think for a second that I’m knocking it. What it does is amazing). Big games tend to generate thousands, even hundreds of thousands of assets – and in the content pipeline, each of those is a different file. People often don’t realise that opening a file, even from a hard drive, can sometimes take as long as a quarter of a second. That doesn’t sound like much until you multiply it by a hundred thousand. (You can very easily see this for yourself by using WinZip to zip up one large file of say 10MB, then compare it to zipping up a thousand files of 10KB each – I guarantee the latter will be much slower). To give an XNA-related example: Kameo, Pocket and Pikelet all used XACT, the same audio tool as provided with XNA (and which I’ve used very extensively). Pocket, in particular, had well over 200 soundbanks which all needed loading at startup. Even though they were only between just 1 and 4KB each, this took a long time to load from the DVD - so long that the 20 second load time mandated by Xbox 360 TCRs seemed an impossibility. The fix was pretty simple: we altered the build process to package all those soundbanks into a single file, loaded the file into memory, and loaded the soundbanks from that memory. Instantly the problem was solved. The same solution won’t work in XNA: there’s no method to load soundbanks from memory. It expects each soundbank as a separate file, and that’s that. A big game in XNA would really struggle under such restrictions. (Loading on a background thread won’t help, 20 seconds of disk access is 20 seconds of disk access and that’s that, though admittedly an XNA game would be on a hard drive not a DVD which helps a lot). You’d have to plan ahead very carefully – in terms of the game design, not just code; which is hard, because designers don’t understand arbitrary restrictions from code, and nor should they have to - to ensure that there was never a need to load so many individual files all at once.

With these caveats in mind I’m convinced a medium-to-large game can realistically be made in XNA. The “poor performance” of C# compared to C++ is for the most part a myth, though to get the absolute 100% optimum out of the hardware might require C++ – but you could get at least 95% of the way there with C#, and 98% of all games don’t need 100% performance. It’s unfortunate that the biggest win for hobbyists and small-team endeavours – the brilliant content pipeline – appears to be the biggest limiting factor for large teams. The issue of when and how assets are built, and how they can be rapidly iterated without restarting the game, is definitely solvable, though with a fair bit of coding effort; the fact that so much of the pipeline relies on a one asset, one file relationship is trickier. A hundred-thousand-asset game would need a hundred thousand files, and loading times would be horrific.

So really large games might not be easy in XNA. Happily (?) few large games nowadays are single platform, and since XNA is not available on PS3 and Gamecube, I don’t think many large teams will be evaluating XNA anyway. Medium teams, independents, and small hobbyist teams can definitely have a field day with it (though profitability of XBLCG at least has yet to be proven). Just remember that the limiting factor for such games is content, and put a lot of effort into making your artists’ lives easy. If they can make a model, put it in your game, play with it, tweak it, play with it again, tweak it, play with it, and check it into source control, they’ll beg to be allowed to make more. If they have to send you the model and wait two days for you to send them a build with it in, they’ll give up within a week. Frankly – even if you’re working on your own, and doing all the code and art yourself, a little bit of work to make good tools will make your own life easier; and isn’t that what we all want?

January 21, 2009

Any requests?

Filed under: Games Development,Personal,XNA — bittermanandy @ 1:51 pm

Wow, it seems like an age since I posted. I guess that’s because it is! Fable 2 took up far more of my time than I expected, I’ve barely touched Nuts and Bolts, and I still have to complete Tomb Raider and Prince of Persia before March, when Empire: Total War is released. This doesn’t leave much time for any meaningful development on Pandemonium, though I have recently chucked together a few little prototypes of stuff I might one day make into fully-developed games.

Anyway I don’t know when it will be that I’ve done enough on Pandemonium to make an update worthwhile. But I don’t want to leave the blog stagnant, it’s still getting a fair number of hits every day and people must have expectations… so I’ll put the question out there: is there anything (preferably XNA-related) you want to read about? Check out the stuff I’ve already written to get an idea of the kinds of things I know. Clue: I’m not the man to ask about whizz-bang graphical effects, but if there’s anything else you’d like to see, just ask.

The Rubric Theme Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.