I’ve been with Google now for nearly seven months. It’s pretty great. Game development was fun, but I’m glad I’m done with it. As far as good development practices are concerned, the games industry is just plain immature. Not as in childish – I’m not entirely sure how to explain it to be honest. Too intense and political, I guess? The freedom to work on what I want is probably the best thing about my role. Right now I’m looking at more automation, to eliminate a tedious manual process and free up more of my seniors’ time.
My role on Space Marine was on the Tools team, administering the build farm. I really didn’t know much about pipelines going into it, though on my 2007 Co-op at EA, I did have some exposure to the central build group for sports games in Burnaby, called Orca (at the time, I know they’ve changed names a few times since then.) So I dove into being build guy at Relic, and learned quickly. Now that I’ve been through a full product cycle, I have a better understanding of how things work, and the role of a build system. For the same of this topic, I define:
A build system is an automaton existing to convert a set of source files of varying formats and produce a set of output files representing the product in development or to be released. These source files are usually the output of members of a project’s production staff (programmers, artists, animators, audio designers, level designers, etc.) but can also be previously generated output from the build system. The output files can consist of loose files, archives, install packages, disc images (anything which could run the product, conceivably), tools produced from the same set of input files (not to be shipped), and log files for the build operations.
That definition is all well and good as a starting point, but it’s obviously insufficient. How do you determine what needs to be built? How do you provide the input and retrieve the output? What if there are special cases (e.g. a tool has changed, requiring a rebuild of all animations) or dependencies between assets? Complicating all this is the wetware producing the input data – humans don’t always follow the pipeline properly, for a variety of reasons. Sometimes it’s a problem with the pipeline, they forget, there’s a new step they haven’t amalgamated yet, and so on. As well, third-party tools are not always reliable. They have bugs, and so will your build system. We also don’t want to release nonfunctioning output to the team. So, let’s come up with a more concrete set of requirements.
A build system should:
- Detect changes to the set of input files and automatically rebuild output data as necessary
- Test the output at the completion of an operation for some basic level of functionality
- Have the throughput to build each submitted change individually for finely-grained detection of failures (continuous integration)
- Have redundancy in place to tolerate bugs in the system which may erroneously report failures
- Have a facility in place to request rebuilds of certain data sets as required
- Distribute output data to users, at their convenience, promptly.
Each of these requirements looks simple at first, but expanding each requirement quickly expands the scope of the system. It is no longer just a process running on a server, but an integrated part of the pipeline and tool chain, with deeper integration for different parts of the system. To expand on the complexity of some requirements:
- Detecting changes to the input files requires some kind of notification that files have changed. A straightforward way of doing this would be to set up a trigger in your source control system to notify the build system that files a, b, and c have changed. It is naive to assume that just because a, b, and c have changed, you only need to rebuild three files. What if a is a texture shared between five other assets? Then you also need to rebuild d, e, f, g, and h. How do you know this programmatically? Some sort of asset tracking system is now needed.
- Automated testing of basic functionality can fall into two categories: smoke testing and unit testing, the former seen in use more commonly than the latter in my experience. A smoke test is a very basic test to determine if the software can be started and some basic operations performed without a crash. If that fails, there’s not really any point sending the build out to the team for consumption. On the other hand, a unit test is used to validate specific classes and functions, in much finer detail, and would be useful to find bugs before QA does. A failed unit test should throw warning flags, but not prevent the entire build from being released to the team.
- Output data from a build process is more than the program and its dependencies, it is also the logs and reports from each stage of building and auto-testing. Networks are fast today, but on a large team with a large build, the speed of the drives containing output data will become a bottleneck. Thus, an intelligent distribution system is needed to perform load-balancing and deliver files to users in a manner such that they can pull their data from multiple sources. The logging system should parse the return code, stdout, and stderr from each process ran, extract the relevant data, and present a human-readable log to the users for troubleshooting. The full logs should always be available in case the summary is insufficient.
This concludes a very high-level overview of what a build system can do to improve the productivity of the team using it. In future posts, I will discuss the varying subsystems in greater detail and propose algorithms to address their requirements.
My time on Space Marine has ended. My first game as a professional game developer launches tomorrow night in North America (though street dates seem to have been broken in Europe already), and I must admit I am glad to have a break from being build guy. The last few months have been very intense: running release candidates, preparing submissions to MS/Sony, dealing with several 11th-hour emergencies, and all the extra hours of finaling. I knew finaling would be tough, but I definitely underestimated just how tough it would be.
I hope Space Marine does well – general reaction to the demo from those who have played it, as far as I can see, is quite positive. The people claiming we’re a Gears clone are finally quieting down, and a following seems to be starting up. I hope it translates into more sales.
What did I learn? Well, I certainly put my Distributed Systems course to work, as I took the build farm from a fault-intolerant eight-node system to a significantly more tolerant 33-node system. The biggest nagging issue I was unable to solve was that for some asset types, on a blade server with no physical GPU, a tool would occasionally fail on a given asset (and then succeed in subsequent executions), and the logging I had in place was not sufficient to track down the issue. Then we went into pipeline lockdown and we let the issue lie.
Besides that, I learned a bit about the file systems on the Xbox 360 and Playstation 3, and the different methods of distributing content on those platforms. I learned how to publish on Steam, and while I think Microsoft has the best publishing tools of the three, I have to give Valve credit on how approachable they were for any publishing/distribution issues I had. The PS3 and X360 discs, as well as the Steam digital download, are my largest user-facing contributions.
I learned a bit about Flash too, as I fixed some UI audio issues in the endgame.
Overall, Space Marine was great to work on. I proved myself as a programmer, rose pretty quickly from junior to intermediate, and will be moving into gameplay on my next project, on what looks to be a pretty cool feature. I won’t say what, yet, until I’m sure it’s fine to talk about it. THQ has announced and shown it, so it’s probably fine, but I want to play it safe.
Annual reviews at Relic were last week, and, I am no longer an Associate Programmer III. I’ve been promoted to Programmer I, woo! This is awesome on two levels. First, it’s a silent I-told-you-so to the companies that wouldn’t even give me the time of day when I was looking for work in 2009. One company flat-out told me they don’t hire juniors, period. I think that’s a terrible HR decision, but that’s for another post, and I don’t want to sound too vain.
The second is for the major confidence booster. For the past 17 months, the vast majority of the code I’ve written is build server code in C#, then tools in an assortment of languages, and a very little bit of C++ game code. I’ve been worried that when I finish on Space Marine and move into a role more directly involved in game code, my C++ skills will be too rusty, and I’ll take too long to ramp up to a new role and a language I haven’t seriously used in a long time. On the other hand, I’d only touched C# in a single class in university before diving into our heavily distributed build farm, and I picked it up just fine. I’m surely making mountains out of molehills, as usual. The specific language is a minor part of being a good programmer, after all.
Anyway, promotions, woo! An additional woo is reserved for when I finally get over my self-doubt issues.
Random other stuff:
- The lack of posts on here has been due to the combined factors of not wanting to simply repost everything we officially post about Space Marine (and seeming like a drone in the process) and not having anything meaningful to post about.
- I am totally going to buy an Emissary later this year once Space Marine is done and we move to something bigger.
This came about as a result of Angelina commenting on Persona 3. I looked for it on the shelf, and didn’t see it. “Oh, it must be in the games to sell box,” I said. An amusing few minutes later, Angelina now has FF12 Collector’s Edition, Persona 3, and Scribblenauts. Apparently this is Christmas morning/shortly after moving in all over again. I have been encouraged to post the contents of the Games – Sell box, so, here it is.
Kirby: Canvas Curse
Metroid Prime Pinball (rumble pak not included)
ATF: Advanced Tactical Fighters
The Bard’s Tale (2005)
Battlefield 2: Special Forces
Command and Conquer 3
Command and Conquer: Generals
The Elder Scrolls III: Morrowind (two copies)
IL 2 Sturmovik
The Lord of the Rings: The Battle for Middle-Earth
Medal of Honour: Allied Assault
Need for Speed: Porsche Unleashed
Pool of Radiance: Ruins of Myth Drannor
The Sims Double Deluxe
The Sims Hot Date
Tom Clancy’s Rainbow Six: Rogue Spear: Black Thorn
Unreal Tournament 2004
Who Wants to be a Millionaire
Medal of Honour
Harvey Birdman: Attorney at Law
The Lord of the Rings: The Return of the King
The Lord of the Rings: The Two Towers
The Godfather: Blackhand Edition
Goldeneye: Rogue Agent
The Lord of the Rings: The Third Age
Medal of Honour: Heroes 2
Army of Two
Forza Motorsport 2
Kameo: Elements of Power
Tony Hawk Project 8
Some of these are pack-ins from motherboards and graphics cards, some of them are free games from undergrad co-op jobs, and some are just games I either just could not get into, or aren’t good enough to keep a spot on my shelf.
If you’re reading this and have an interest in anything on the list, drop me a note and we shall see what can be worked out. Due to the account registration policies of yesteryear, I make no assurances regarding the online playability of the PC games.
The announcement was made at the beginning of last week. The fans seem thrilled with the news!
Magazines are starting to appear featuring our game. So far, I’ve seen us on the cover of Gamers.at and consol.AT, Austrian gaming press. We’re also in their German counterparts, ConsolPLUS and GamersPlus.
I’m looking forward to seeing what English-speaking press have to say about the game. I’ll probably be snapping up physical copies of any magazines featuring Space Marine for memorabilia’s sake. It’s my first game as a professional, rather than a co-op, after all!
At E3 this year, we showed a new trailer for Space Marine.
We were nominated for awards just based on this trailer! It seems as though people really want our game, which is a really nice feeling. Working on FIFA and Euro, most of what I saw online was speculation of how we’d compare to Konami’s Pro Evolution Soccer (Winning Eleven depending what part of the world you live in). It’s a completely different vibe having people looking forward to your game like this.
In a bit of cross-promotion with Dark Millenium Online, we had some people in costume at the booth. They went over incredibly well. We had an Ultramarine, Chaos Marine, and some kind of Ork. Below is the Ultramarine, posing for pictures with fans.