Wednesday 16 March 2011

playtest system

I've been using the following system for playtesting Vertex Dispenser. Recently I was a tester for Jason Rohrer's new game INSIDE A STAR FILLED SKY, and he was using a similar system to great effect. I thought I'd write it up because it's extremely useful; I would have benefited from implementing it sooner than I did (and I should probably use it for more of my games), so maybe this will help someone.

Basically, when someone plays my game all their input is recorded and logged to a file. Then they send me the logs, which I can play back and watch what they did in the game. I see (essentially) exactly what they saw, as though I was watching behind them as they played.

This is so ridiculously useful. Actually watching someone play is a million times better than sending them a game and getting written feedback. With written feedback there's an additional overhead from them having to explain where they had a problem, and if they've missed or misunderstood something important then they might not even know they have a problem, or might associate the problem with something completely different. If you can see them you know exactly what went wrong. Watching someone play in person is good, but it's easy to mess it up by telling them something to help them out. When someone downloads your game and plays it, they won't have you there to give hints, so recording remote playtests gives a much more accurate (and scary) depiction of what will happen when your game is released. It also enlarges your pool of possible testers, because they don't have to be geographically proximate and can do it in their own time.

(Written feedback is still useful, because you might determine from it ways that someone's mental model of the game is wrong that weren't clear from watching them play, or you might ask them why they were doing something that you didn't understand in the game. Also, suggestions from testers can be really useful; often you have to filter out stuff that conflicts with your own vision of the game, but sometimes another person will be able to come up with really excellent ideas that you couldn't think of because of your preconceptions.)

For this system to work, you need to make sure the game is completely deterministic from input. This means that updates to the game state shouldn't depend on the frame rate, and you have to be really careful with floating point numbers - they can give different results across different platforms, so it's best to stick with fixed point arithmetic if you can.. this is the hard bit. (Pseudo)random numbers are okay as long as you store the seed used with the log file. The input might be at the level of keystrokes and mouse movements, or at some higher level of game actions (i.e. "order unit X to move to position Y" for an RTS).

What else is great about this method is that once you've implemented it, several other things become trivial:
- It's useful in your own testing. If you think you glimpsed something out of place in one frame, you can just replay the log file frame by frame to check for sure.
- Replays are cool to have anyway. In an action or puzzle game, some players might like to share "look what I did", and in a strategy game some might like to look back over their games to see how they lost and can improve.
- Replays can be handy for tutorials. In Vertex Dispenser I have a couple of levels where I play a recording of myself solving a puzzle or using an ability, because playtests indicated that just using words wasn't explaining them adequately. These only took a few minutes to hack in because I already had this replay functionality to use.
- Networked multiplayer can be easier to implement (though still far from trivial). All you need to do is pass player input around the network, and advance the game simultaneously with the same input on each copy, and everyone will stay in sync (although you might want to check hashes of the game state regularly just to be sure). For some types of game it's more important to have a low response time locally than to have other players exactly in sync, so you want to do some kind of prediction instead of waiting for remote input, but a modification of this system can still work - I rewind when remote actions are received and then recompute up to the present time taking them into account.

Something Jason did but I didn't was to add to the file as they play, before input is acted upon, so that if there's a crash then the log will crash the game at the same point. I made the mistake of just saving the file at the end of each level, which means if there's a crash then a few minutes of recording are lost. Next time I'll know better.

Saturday 12 March 2011


GDC was awesome.

It's ridiculously huge. 19000 attendees, what? If there are that many people making games, how are there not a hell of a lot more games?
(Answer: most of these people aren't making their own games, they're working on other people's games, or hiring people to work on other people's games, or funding other people to make games, or distributing other people's games.. the mainstream industry is an alien place.)

So I felt a bit lost at first, until I met up with some folk I know from the internet and UK indie meetups. It helped to be involved with something as well, I think if I hadn't been speaking at EGW I'd have felt more lost and overwhelmed. (Speaking was surprisingly not overwhelming, despite being to an order of magnitude more people than I'd ever spoken in front of before. It helped that I wasn't on my own, but maybe it's easier to talk about games than mathematics too.)

I happened to have gotten a full pass for free, so I went to talks and stuff, but it would have been well worth going just to meet up with people, even to just stay at the hostel and meet people there outside of GDC! I'm used to gamemaking just being my own weird hobby that nobody else does, so being surrounded by others who do the same was an amazing experience. Some of the talks were really good, but it wouldn't have been worth me actually paying full price for them.

Michael Todd's talk about depression really spoke to where I am right now with my mathematics. I've been in a downward spiral where lack of progress leads to lack of motivation and then lack of motivation leads to lack of progress. He suggested some strategies for dealing with this (in the context of game development, but applicable outside of it): focus on highly rewarding projects, stop being a perfectionist, work on shorter projects, measure work hours, design to suit your abilities.

Brian Moriarty gave an entertaining, well-researched, thought-provoking, but mostly wrong talk defending Roger Ebert's (dull, poorly-researched, completely wrong) claim that games cannot be art.
He pointed out that to determine whether games are art we need to agree on a definition of "art". Clearly games are are art under some definitions. But after this good start he failed to define his term "sublime art"! One of the common features of his examples of "sublime art" seemed to be historic significance, but this was presumably not the intention.
He made the point that videogames are not exceptional among games; they should only be considered to be art if board games are also. This seems obvious to me, but I've come across those who think otherwise so it's worth saying. He used this to support his argument that games aren't art because some philosopher didn't include them in his enumeration of art forms, which seemed a bit weak to me.
He also claimed that nobody considered mathematics to be an art form; I do, and I know many others who do also (mostly mathematicians).
He said we can't expect big studios to produce great art, because they need to make a profit, but then made the unusual claim that the same applies to independent game developers; he is perhaps unaware of how many people are making games while financially supporting themselves (or not) by other means (or with other games). But this is irrelevant to Ebert's claim, which was that games cannot ever be art, not that nobody is motivated to make artistic games.

Other talks I enjoyed were Steph Thirion's on designing by accident and experiment, and Brenda Braithwaite's on designing games about tragedy.

Overall, an incredible week. Hopefully I'll be able to go again - this happened to be on my way back from a friend's wedding in NZ (after not having been home for two years), so there was essentially no travel cost, and then I had a speaker's pass as well. Might not work out so perfectly next time.