Developer Interview - The Crater

In our fifth Developer Interview we're talking to the engineer TheCrater, who's mostly operating out of the shadows but still has a watchful eye over the forums.

The interview was done on Monday, April 6th, before the release of Alcatraz.

Could you introduce yourself?

My name is Martin, I go by TheCrater when I post infrequently on the forums, and I’m an engineer on Defiance.

How long have you been working for Trion/on Defiance?

I’ve been working at Trion for almost a year and a half, on Defiance the whole time. So I was around for Defiance’s 1^stbirthday, but not at the game’s launch.

What is your job and role on Defiance?

My title is “Software Engineer II” and I write code for Defiance’s servers, client, and internal developer tools. My job is to create new game features with the designers and to fix bugs. I generally see the role of a game programmer as creator of the pieces that designers and artists can use to make their vision into a real game; that often involves some minor design and art creativity beyond just writing the specific code functions that were requested.

So, you're an engineer. What do you work on exactly?

Each engineer on the project has some specialty knowledge, but we are all generalists. I’ve written code for everything from the Xbox patcher to pump-action shotgun animation to grouping for the DLC arenas. Some of the biggest features I’ve implemented include the scoring system for events, armor plates and the different damage types, and the upcoming Cyber Rigs.

When and how did you discover your passion for programming?

I find that my history is somewhat unusual among programmers, most of whom had some introduction at a young age. I started university without knowing what programming was, but on a professor’s recommendation I took an intro class and it really clicked with me.

I discovered that I loved the analytical process of breaking down a problem and building a solution with programming languages as my tools. Even the standard boring programming class work (a fake “account book” or Towers of Hanoi) was fun to think about, though I think everyone has to overcome early frustrations with debugging.

Of course I’ve also played a lot of games (pc, console, board games), and I was even more excited when I realized how well the two interests meshed. I started out in school writing bad code in Python, C++ and Java – C++ is primarily what I use these days since it’s a good tool for games like Defiance that need to focus on speed and memory use.

How long have you been in the games industry?

I’ve been a programmer in the games industry since 2009. I got my start as an intern on Bioshock 2, an amazing opportunity I’m still extremely grateful for. I then shipped The Bureau: XCOM Declassified after 3 years before moving to Trion and Defiance.

What are some of the difficulties you face in your work?

Lots of my work is spent fixing bugs. The first step in bug fixing is usually to see it in action and identify what related parts of the game could be causing problems. However lots of bugs are not easy to reproduce, with just a report that it was seen by one of our testers or by a player in PTS or the live game. In those cases – and sometimes even if I’m seeing the bug right here at my desk – tracking down the bug is often a case of educated-guesswork. A player got a “disconnected from server” message after completing a major arkfall? Maybe it’s related to the loot they just received. Maybe it’s related to the leaderboard menu popping up. Maybe it’s related to the enemies from the event despawning. For most bugs there are a hundred different paths of investigation and the engineer’s job is to follow the paths that seem most likely based on their personal experience and additional information gathered from server logs or crashdumps (a small, targeted snapshot of active program memory – surprised that we get access to that? Should’ve read that EULA! :P )

If you could introduce one feature to the game, what would it be?

Oh man it would probably be something totally stupid like fishing & cooking minigames or dance-off duels.

Seriously though, I think Defiance really shines when it comes to collecting a wide variety of weapons and I would love to add some features that help with that – inventory management improvements (which are already being discussed by the team), more things like equipment sets and collections, and more strange and unusual weapons like infectors and bmgs (I’m glad that some of the recent weapons like the new Defiant Few charge weapons are experimenting with the shooting mechanics in a weird and interesting way).

From your time working on Defiance, do have any interesting stories you can share?

A couple months after I started on Defiance (the game had been live for more than half a year at this point) our QA team discovered that some specific items were never being rolled from some loot drops. This was really surprising to everyone on the team, but a couple of us starting looking into the gritty details of the random number generation of item drops – and we found a problem. This is one of the scarier things that you can say to developers on a loot-driven game like Defiance. It turns out for some of our core randomization functions we were using the oldschool C-rand(), which can only return values up to RAND_MAX (32,767 on most compilers). This wouldn’t have been a problem on its own except that some designers had also used randomization weights that were much larger than 32k (in the millions in some cases) so the code was accidentally skipping the upper portion of the intended possibilities in a lot of cases. I can still remember clearly the moment when Trick slapped his forehead and said “shit, this explains why so many enemies seem to always dodge to the left!”

We fixed the bugs by switching over to random number generation that could handle a larger range of numbers, but I’ll probably remember this as one of the funnier development moments for my whole career.

What are some of the differences between working on Defiance and other games?

Defiance is the first online game I’ve worked on – it turns out this makes a huge difference. The previous games I’ve worked on were both shooters released on PC, Xbox360 and PS3 so Defiance is similar in a lot of ways, but those games are both single-player (Bioshock 2 has a deathmatch multiplayer mode but it was contracted out to a different studio and XCOM’s multiplayer mode was abandoned years before the game shipped).

The first big difference is the client-server game engine architecture which changes a programmer’s life in a lot of ways. Events needs to be communicated through network messaging, game state needs to be replicated between multiple servers and clients, and the simulation from the client needs to be verified for hacked clients or cheating programs. A lot of problems that are faced by all games are also solved much differently in online games – save/load, text localization, memory constraints. It’s a lot to keep track of and it means most of the gameplay code is structured differently.

The other big difference is that Defiance is a live game. With the previous projects there was no interaction between the developers and the players – once we shipped the game it was out of our hands, and playtesting during development was extremely limited. Afterwards we would read reviews and player’s comments online and wish we had made some changes, but beyond patching any critical crashes and similar game-breaking bugs it was on to the next project (or the next DLC which was effectively the same as a new project with the way patches were handled). With live games like Defiance there’s a lot more connection with the players (even if some of us don’t post on the forums as often or appear on livestreams, we are paying attention).

What's on your desk?

• PS3 & 360 devkits
• Guardian Force Odin figure (in box)
• Psyduck plush (my programming muse)
• A heaping pile of various teas
• A photo of some of us Defiance devs at the PAX Trion community party
• A poster for the band Catacombs
• And a puppy (well… not ON the desk)

Can you tell us about your typical workday?

The first thing I usually do when arriving at the office is checking whether the nightly automated build had any failures. Usually Phantasie is on top of those, but if there are any failures from code that didn’t compile or tools that had an error it might be up to one of us engineers to fix. We also run much more frequent automated code builds to catch compile errors, because it’s easier than you might think to submit a change that seems great on your local machine but fails one of the 12 or so executable configurations.

At Trion thankfully we don’t have a ton of scheduled meetings (at least for the engineers), so I can focus on programming most days. Most of the Defiance team has desks really close together though, so we are frequently having discussions about changes to the game and ideas about bug fixes and about what constitutes a “rogue-like”. I find it really valuable to have that close proximity, quick turnaround communication so I’m often asking questions of the other developers about parts of the game where their knowledge is stronger and I try to help out with answers when I can. Aside from those kinds of interruptions programming requires a bit of concentration and focus, and I usually wear headphones to listen to music and block out distractions for part of the day. Working on Defiance also requires some multitasking though, since I often have several active tasks that I switch between based on the priorities of the day or hour. If I’m working on a feature that will take most of a week to complete I will inevitably still need to jump around to at least one higher priority bug every couple days. That can be as simple as tracking down some specific logs for someone who needs more information for their own bug-fixing or as broad as “xbox EU has been experiencing disconnects”. Of course on any day when new servers or client-patches are deployed the engineers are especially watchful for problems and we know that special bug-fixing attention might be needed.

Some days one of the developers will send out a message asking for playtesting groups and we’ll run through some new piece of content together on the internal development servers. We’ve been especially doing this a lot for the holiday events and for expeditions.

And some days I bring my puppy to the office so I need to make sure she’s comfortable and feeling loved while I work by visiting the other dogs in the office or going for short walks.

Is your nickname a homage to "The Crater" in Defiance? If so, why did you choose it as your nickname?

It is! It’s one of the most defining Defiance locations for me. I love drinking beer and I brew my own at home so I guess it’s natural that I’d be drawn to the game’s bar. And don’t let Fuzzy tell you that German beer is better – American IPAs ftw!

Excluding Trion Games, which games do you like to play?

Most of my favorites are JRPGs but I obviously play a lot of shooters too. I also really love the indie games that are able to experiment and branch out into more uncharted territory with their mechanics and storytelling. The non-Trion games I can remember playing most recently have been:

• Borderlands 2 - I skipped some of the DLCs before
• Legend of Dragoon - oldschool!
• XCOM: Enemy Unknown – this time I named all of my soldiers after other Defiance developers. It was a bloodbath!
• Kentucky Route Zero
• Luftrausers

What do you do in your free time?

I already mentioned that I’m a homebrewer, which is a great hobby especially if you enjoy experimenting with recipes. I’ve made a variety of ales: some good, some bad, but all drinkable. I’m rather proud of the vanilla-nutmeg stout I made most recently. There are enough brewers in the office that we actually had a Trion homebrewing contest about a year ago! I didn’t win.

I’m also really into underground music and I go to quite a few concerts. Living in Oakland there’s a great metal and punk scene in the area, not to mention some of the best hip hop.

And of course I have a couple game projects that I tinker with off and on at home. Some of the other Defiance devs are even more active than me with this kind of thing, building games or even robots when they have the free time.

Out of your recent contributions to the game, what are you most proud of?

I’m really happy that I got to add Cyber Rigs to Defiance. A lot of the code for that was fun to write – it touched the player persistence database, server messaging, loot dropping, new menus, the stat bonus systems used by mods and synergies, the item-creation tools for designers, and the item-viewing tools for customer service. I love adding features that are broadly connected to so many parts of the game. But mostly I like how it’ll change the game for players. It adds a lot of character growth and collectability, which I think will renew the interest of players who feel like they’ve already maxed out a lot of their equipment. But it also ends up allowing for a lot more play-style customization which is something I hope to add to every game I work on.

My buddy Rarnok (you might have seen him on last week’s Armistice livestream) was experimenting with Chip combinations that focused on boosting the power of his Wolfman SMG loadout and it was really cool to see the difference compared to a Rig that’s customized for something else. Also I love that the Cyber Chips that increase infector bug damage are called “Bite Intensifier”. You can thank Rashere for that one.

What are you looking forward to the most coming with Alcatraz?

I loved that as soon as Cyber Rigs hit the PTS servers someone made a forum thread for posting which Rigs and Chips people had received. And now when Alcatraz hasn’t even gone live yet, DefianceData already has a thorough guide. I’m really excited to see the builds people come up with for their existing loadouts, and hopefully people also try out different weapon/synergy/perk combinations based on the coolest chips they find!

What's your favourite outfit or weapon?

VBI Infiltrator