Sunday, November 27, 2011

M13 Playtesting Has Limits?

I've learned more than a few things during this project, and I hope those following along have gleaned a nugget here and there as well, but I've come to accept an unusually large lesson very recently: There are some design decisions that you can't rely on playtesting to help you make. I'm not sure I would have ever stated the counterpoint (that every design decision can be based upon playtesting) but solidifying the concept of what can and can't be handled this way took me the better part of two months.

Specifically, I'm talking about our experience with lair in Magic 2013. We did a lot of thinking to pare down the choices of mechanics for our set in the beginning, and we did playtest to check the validity of our choice. When it came to the details of how to execute lair in the set, we discussed the possibilities and determined some potential paths as well as eliminating a few that wouldn't even be worth exploring. We laid some preliminary ground rules for the mechanic and that's where things started to get tricky.

Rather than just assume those rules were golden, we decided to analyze and test a few of them to see if there were better options. I explored the possibility of enters-the-battlefield lair effects by designing some reasonable possibilities and playtesting them. While the results were helpful, they weren't concrete. The impressions my testers and I took away were enough to support my concerns with that alternate path and so we've continued with the static-abilities-only model. It's worth noting however than I'm not 100% convinced this is the right way. If I was 65% sure before the testing, I'm maybe 85% sure now. But we can't go part way on a binary decision, so we obviously take 85% over 15%.

Playtesting helped there, even if it didn't answer the question fully. Playtesting is good at answering tangible questions like "is this fun?" "is this too complex?" and "is this too weird?" but it's not so good at answering philosophical questions like "is this a good structure?" "does this fit the color pie?" or "is this the best use of our resources?" Want an example? As if you could stop me.

While I was testing that stuff, Nich Grayson was exploring the possibility of asymmetrical use of lair in M13. If you look at keywords and ability words historically, very few are used in equal numbers across all colors so it was important to ask the question whether that's right for lair. Nich removed lair from green and blue and added Village Skycaptain to even things out for various color combinations. Check out the wiki page for full details.

I can't say the way he executed asymmetrical lair is exactly how I would have done it, but it's more than respectable and absolutely worth consideration. Ideas don't start finished and if we went this path it would be refined (maybe a little, maybe a lot—who knows) but there's a lot of solid work here.

Here's the problem: I was thinking, "Oh, we'll just playtest it to see if it's better than what we've got now." ENHHH! Wrong. While statistical data on the level of global sales that Wizards sees every quarter will be affected by a decision like this, a few hours of playtesting (or even a few hundred) aren't going to reveal the better choice. Players will be aware of things like how much fun they had playing, what cards felt weak, what cards excited them, whether games went long, etc. They're unlikely to take much notice whether green did or didn't have lair cards and even less likely to have an opinion on the matter.

You and I are on the whole—by nature of the blog we met on—the players most acutely tuned to Design. When you poured over the Innistrad file, you saw that green had a lot fewer flashback cards than red, but did you consciously notice it? Did you wonder why? The opposite is true of morbid. Did you ever question that allocation? I didn't. In fairness, part of the reason is that Wizards' impeccable craftsmanship makes such decisions seemless: We probably would've noticed something clumsy, like if blue had morbid but not flashback.

You know who wouldn't have? Every other Magic player. Okay, not every other player, but the vast majority. Why? Because normal players don't analyze the game. Those that do are analyzing draft picks, memorizing pack collation orders, sniffing out new combos, retrofitting decks or speculating on card values. The folks who care about the color pie, card templating, design space and other subtle details that don't help them win games or puzzle together storylines are we few Rogers. And even we take a lot of things for granted.

So we can test the set with lair in all colors or in just a few, but as long as the cards are simple, fun and interesting, it's not going to tell us which way to go. We have to be designers and design the answer. And we have to do it together. If I'm right about design 33% of the time, and you're right 40% of the time, and he's right 25% of the time and we all work together, we'll be right 70% of the time as a group. That's about double the quality with three people and we've got a lot more smart people sharing their insights on this blog than that.

I think I've run out of room here, but my next post(s) will catch you up on what we know and think we know about lair in 2013 and hopefully we can nail down some decisions that the team can build around going forward.


  1. Good article. I also feel like I’ve learned a lot. This is one of the first times I’ve designed cards with a larger group. I don’t want to let Jay down, so I’ve put more effort into M13 then I would have for a personal set design. I’ve made entire sets where I never actually playtested the cards. (I famously made a homebrew years ago where I printed out hundreds of color copy cards for a sealed prerelease with my friends and learned AT THE EVENT that many of the cards were just horribly undeveloped.) It’s easy to look at those sets and feel like they are finalized. But without playtesting, it’s just not the same thing. From my perspective, here are some of the pros and cons of playtesting.

    Pro: You can identify broken cards quickly.
    Con: You won’t always see a card gets its proper due.
    Pro: You can identify limited warping cards quickly.
    Con: Playtesting doesn’t address all formats.
    Pro: You can get a feel for how a card plays versus how it looks like it would play.
    Con: You get a lot of data from playtesting and not all of it is captured.
    Pro: You see a pet card, warts and all, for a more objective analysis.
    Con: Not every cards gets equal playtest time.
    Pro: You get to see people with no connection to the card designs react to them.
    Con: It’s time consuming to make cards, design playtest decks and play sample games.
    Pro: You get a better sense of what your set is missing and what it needs.
    Con: Assumptions and previous experience can let cards slip in when they shouldn’t.

    Jay, can you say more about what you mean when you say we have to be designers and design the answer. Are you saying an executive decision on the direction we take must be made and then it’s our job to make cards that work for that direction? The reason I ask is that you were discussing the idea that Lair effects would be the same not just at common, but across other rarities (IE Forest lair always gives +1/+2.) And I’m afraid that if that direction is decided upon, the cards we design will not be very deep or engaging. So if it’s something I can try to talk you out of, I will. But if it’s a decision I need to just design around I need to know so I can stop worrying about the basis of my argument. (Similar to common Lair auras.)

  2. Thanks, Nich. I appreciate your effort.

    While I /am/ saying an executive decision needs to be made, I'm very much /not/ saying it needs to be made by me alone. I'm saying we need to talk it out as a group and make the decision based on everyone's feedback.

    I'm not married to the idea that Forest lair will always be +1/+2 and nothing else. I'm curious about it, because there's a chance it's right but I'm well aware there's a somewhat larger it's not. I just want to make sure we think it through and make an informed decision rather than take a gut decision for granted.

  3. Hey, look over there - a whole new article was posted. I did not see it when I replied to this. Looks like it addresses most of my concerns so thanks!