playtest system

I've been using the following system for playtesting Vertex Dispenser. Recently I was a tester for Jason Rohrer's new game INSIDE A STAR FILLED SKY, and he was using a similar system to great effect. I thought I'd write it up because it's extremely useful; I would have benefited from implementing it sooner than I did (and I should probably use it for more of my games), so maybe this will help someone.

Basically, when someone plays my game all their input is recorded and logged to a file. Then they send me the logs, which I can play back and watch what they did in the game. I see (essentially) exactly what they saw, as though I was watching behind them as they played.

This is so ridiculously useful. Actually watching someone play is a million times better than sending them a game and getting written feedback. With written feedback there's an additional overhead from them having to explain where they had a problem, and if they've missed or misunderstood something important then they might not even know they have a problem, or might associate the problem with something completely different. If you can see them you know exactly what went wrong. Watching someone play in person is good, but it's easy to mess it up by telling them something to help them out. When someone downloads your game and plays it, they won't have you there to give hints, so recording remote playtests gives a much more accurate (and scary) depiction of what will happen when your game is released. It also enlarges your pool of possible testers, because they don't have to be geographically proximate and can do it in their own time.

(Written feedback is still useful, because you might determine from it ways that someone's mental model of the game is wrong that weren't clear from watching them play, or you might ask them why they were doing something that you didn't understand in the game. Also, suggestions from testers can be really useful; often you have to filter out stuff that conflicts with your own vision of the game, but sometimes another person will be able to come up with really excellent ideas that you couldn't think of because of your preconceptions.)

For this system to work, you need to make sure the game is completely deterministic from input. This means that updates to the game state shouldn't depend on the frame rate, and you have to be really careful with floating point numbers - they can give different results across different platforms, so it's best to stick with fixed point arithmetic if you can.. this is the hard bit. (Pseudo)random numbers are okay as long as you store the seed used with the log file. The input might be at the level of keystrokes and mouse movements, or at some higher level of game actions (i.e. "order unit X to move to position Y" for an RTS).

What else is great about this method is that once you've implemented it, several other things become trivial:
- It's useful in your own testing. If you think you glimpsed something out of place in one frame, you can just replay the log file frame by frame to check for sure.
- Replays are cool to have anyway. In an action or puzzle game, some players might like to share "look what I did", and in a strategy game some might like to look back over their games to see how they lost and can improve.
- Replays can be handy for tutorials. In Vertex Dispenser I have a couple of levels where I play a recording of myself solving a puzzle or using an ability, because playtests indicated that just using words wasn't explaining them adequately. These only took a few minutes to hack in because I already had this replay functionality to use.
- Networked multiplayer can be easier to implement (though still far from trivial). All you need to do is pass player input around the network, and advance the game simultaneously with the same input on each copy, and everyone will stay in sync (although you might want to check hashes of the game state regularly just to be sure). For some types of game it's more important to have a low response time locally than to have other players exactly in sync, so you want to do some kind of prediction instead of waiting for remote input, but a modification of this system can still work - I rewind when remote actions are received and then recompute up to the present time taking them into account.

Something Jason did but I didn't was to add to the file as they play, before input is acted upon, so that if there's a crash then the log will crash the game at the same point. I made the mistake of just saving the file at the end of each level, which means if there's a crash then a few minutes of recording are lost. Next time I'll know better.