I'm about ready to send the Phoenix RPG book to the playtesters, but before doing that my editor BabyBat and I have done a complete pass through the book. The book right now is a little over 100 pages and in full print layout, so a complete pass takes a bit of time. The book represents about a year of effort so far and clearly there is much left to do. The more I do this, the more I appreciate the hard work that goes into game books before they are released.
Being from the software industry, we have treated this project similarly and created a defects log. Each defect has a status - New, Open, Fixed, Futured, or Closed. As I make changes I update the status and she confirms whether the defect is really fixed or not.
Here's a sample of the defects log spreadsheet:
|Holy defects, Batman!|
As you can see, my editor isn't being kind! But that's perfect, because the quality of any product is directly correlated to the quality of the testing that goes into it.
The editor of an RPG product isn't responsible for whether the mechanics are sound, or even if the game is fun, but rather for making sure it can be understood clearly and concisely. Peter D wrote a good post recently about how game writing is like technical writing, which is a great way of thinking about tasks like this. Noone wants a book that is indecipherable or full of contradictions, although I'm sure there are widely varying tolerance levels.
My upcoming playtest campaigns - both OTB and Virtual (via MapTool) - are essentially Beta tests. They're also where I will test prototypes of things like monsters and magic items, as I need to get a good feel for how a few of those work in real play before delving into larger content design on them. We're reimagining most monsters from their original mythological sources (where applicable) and coming up with some great ideas so far. I can't wait to share some of the results.
During playtesting, BabyBat & I will be recording new issues in a similar log, as they are discovered in play or commented on by the players. This will form the basis of an Agile process where I can drop new releases periodically with the latest fixes and updates, and I'll be able to have release notes from the log that can help players know what the latest changes are.
As for me, I'm looking forward to running some games!