M13 Playtesting Has Limits?

I've learned more than a few things during this project, and I hope those following along have gleaned a nugget here and there as well, but I've come to accept an unusually large lesson very recently: There are some design decisions that you can't rely on playtesting to help you make. I'm not sure I would have ever stated the counterpoint (that every design decision can be based upon playtesting) but solidifying the concept of what can and can't be handled this way took me the better part of two months.

Specifically, I'm talking about our experience with lair in Magic 2013. We did a lot of thinking to pare down the choices of mechanics for our set in the beginning, and we did playtest to check the validity of our choice. When it came to the details of how to execute lair in the set, we discussed the possibilities and determined some potential paths as well as eliminating a few that wouldn't even be worth exploring. We laid some preliminary ground rules for the mechanic and that's where things started to get tricky.

Rather than just assume those rules were golden, we decided to analyze and test a few of them to see if there were better options. I explored the possibility of enters-the-battlefield lair effects by designing some reasonable possibilities and playtesting them. While the results were helpful, they weren't concrete. The impressions my testers and I took away were enough to support my concerns with that alternate path and so we've continued with the static-abilities-only model. It's worth noting however than I'm not 100% convinced this is the right way. If I was 65% sure before the testing, I'm maybe 85% sure now. But we can't go part way on a binary decision, so we obviously take 85% over 15%.

Playtesting helped there, even if it didn't answer the question fully. Playtesting is good at answering tangible questions like "is this fun?" "is this too complex?" and "is this too weird?" but it's not so good at answering philosophical questions like "is this a good structure?" "does this fit the color pie?" or "is this the best use of our resources?" Want an example? As if you could stop me.

While I was testing that stuff, Nich Grayson was exploring the possibility of asymmetrical use of lair in M13. If you look at keywords and ability words historically, very few are used in equal numbers across all colors so it was important to ask the question whether that's right for lair. Nich removed lair from green and blue and added Village Skycaptain to even things out for various color combinations. Check out the wiki page for full details.

I can't say the way he executed asymmetrical lair is exactly how I would have done it, but it's more than respectable and absolutely worth consideration. Ideas don't start finished and if we went this path it would be refined (maybe a little, maybe a lot—who knows) but there's a lot of solid work here.

Here's the problem: I was thinking, "Oh, we'll just playtest it to see if it's better than what we've got now." ENHHH! Wrong. While statistical data on the level of global sales that Wizards sees every quarter will be affected by a decision like this, a few hours of playtesting (or even a few hundred) aren't going to reveal the better choice. Players will be aware of things like how much fun they had playing, what cards felt weak, what cards excited them, whether games went long, etc. They're unlikely to take much notice whether green did or didn't have lair cards and even less likely to have an opinion on the matter.

You and I are on the whole—by nature of the blog we met on—the players most acutely tuned to Design. When you poured over the Innistrad file, you saw that green had a lot fewer flashback cards than red, but did you consciously notice it? Did you wonder why? The opposite is true of morbid. Did you ever question that allocation? I didn't. In fairness, part of the reason is that Wizards' impeccable craftsmanship makes such decisions seemless: We probably would've noticed something clumsy, like if blue had morbid but not flashback.

You know who wouldn't have? Every other Magic player. Okay, not every other player, but the vast majority. Why? Because normal players don't analyze the game. Those that do are analyzing draft picks, memorizing pack collation orders, sniffing out new combos, retrofitting decks or speculating on card values. The folks who care about the color pie, card templating, design space and other subtle details that don't help them win games or puzzle together storylines are we few Rogers. And even we take a lot of things for granted.

So we can test the set with lair in all colors or in just a few, but as long as the cards are simple, fun and interesting, it's not going to tell us which way to go. We have to be designers and design the answer. And we have to do it together. If I'm right about design 33% of the time, and you're right 40% of the time, and he's right 25% of the time and we all work together, we'll be right 70% of the time as a group. That's about double the quality with three people and we've got a lot more smart people sharing their insights on this blog than that.

I think I've run out of room here, but my next post(s) will catch you up on what we know and think we know about lair in 2013 and hopefully we can nail down some decisions that the team can build around going forward.