Archive for the 'xbox' Category

21
Feb
08

The Microsoft Play Test Experience

What’s a play test?

It’s not unusual for developers to allow their games to be played by the public before it’s full release – most commonly in demo or Beta form. This allows them to gain feedback about a game before its full release. However, there are disadvantages to them. They’re expensive because they require the game to be in some resemblance of a Final build. A demo build by itself has to go through its own quality assurance process which further adds to the cost. They’re also risky because if a demo is received poorly, it can hurt actual retail sales.

Lots of people have downloaded a demo or participated in a Beta. For me, I’ve also had the unique opportunity to participate in play test sessions for Microsoft. Play testing is different from the Demo or Beta and, from a developer’s standpoint, has some important advantages. A play test is invite only. This allows the developer to screen for a certain demographic that the game is targeted for, therefore, giving them relevant opinions. It also lessens the risk of bad word of mouth because the relatively few amount of players involved. On top of that, chosen play testers are silenced by signing a Non Disclosure Agreement (NDA). Promise of silence gives the developer more confidence in showing a product that’s unpolished and/or very early in the development process.

So how do I get in?

Well, it all starts with first signing up for the program. I found out about the program in a local paper which directed me to a URL to sign up. The current URL is http://www.microsoft.com/playtest/default.mspx. That was a few years ago and I’ve been an active participant in the program ever since. When signing up, you give information about yourself to Microsoft. Probably the most important piece of information to give them in your location because play testing is done at a physical location on the Microsoft campus (read: Seattle applicants only!). In addition to the basic information, you tell them your gaming experience and habits.

So what’s it like?

Disclaimer: No particular games will be discussed as the NDA extends even to ones that have been released. Only the process will be discussed – their product testing methodology is fairly standard. Pictures of the actual test lab where not taken by me – they are found under the following link at xbox.com (thanks google!)

During the development cycle, Microsoft may deem it necessary to take some time to allow someone outside the development process to try the game. They’ll look at their pool of applicants and pick a certain demographic based on the game. For example, on my survey, I’ve identified myself as a male, 20s, and like shooter games. As such, I commonly get the “hard core” shooting games.

Once they pick their initial pool, they’ll call people on an individual basis. The purpose of this call is to further screen their applicants with more detailed questions. The bulk of the questions is how much time the individual has played certain games – the screener will say a game and the applicant will respond with the amount of time he/she has played it. They won’t say what game is to be play tested, however, all the games on their list will be similar to what the game that is to be tested. For example, if they where to ask me how much time I’ve played Halo 1, 2, and 3, chances are I’m going to be playing a shooter game. They’ll also ask some basic game behavior questions such as playing games on a PC, playing multiplayer, etc.

If the applicant’s answers lines up properly with what the screener needs to hear, they’ll give an invite to a play session. These play sessions (or at least the ones I’ve been part of) are done in Redmond, Washington, USA which is the home of Microsoft. The particular building is a Microsoft building but is not part of the main Microsoft Campus. Applicants are to provide their own transportation and not allowed to bring friends or family (sorry kids, daddy waits outside for you). Also, as I mentioned earlier, the length of the play test can vary. I’ve had single sessions that where less than 45 minutes. The longest session I signed up for was two consecutive days with individual sessions of 8 hours each.

Upon entering the building, the play tester fills out and signs a few documents. One of which is to register your vehicle so it doesn’t get towed. Also is another survey describing your gaming habits – essentially a paper copy of the questions asked during the phone interview. There’s also an NDA that says you can’t talk about the game. The last piece of paper is a list of gratuity items to pick from. The play tester can pick any software from the list regardless of its individual value. If the play session is one of the long ones, you get to pick more than one – the most for me was four.

If you’re thinking “Hey, I’ll just get Windows Vista and ‘return’ it to my local software retailer”, don’t bother. The UPC is different from retail SKU’s. I think part of the reason is because Microsoft sells their own products directly. They sell them even to their own employees, but at significantly discounted prices (in many cases over 80% off MSRP).

Playing the Games

So far, I’ve done two very different types of play sessions. The most common one I’ve done is very straight forward. They invite a large group of people at once to play their game. Once everyone gets their paper work done, they bring everyone into a room with 20 or so game stations (think cubicles). At each station are headphones, an Xbox controller, an Xbox 360. The 360 is sometimes a development unit instead of a retail unit, depending on how early the game is. There’s also a PC, keyboard, and mouse at each station as well as two monitors (one for the PC, and one for the 360). The PC is there for taking surveys about the game to be played either during or after the play session. Also, the PC is used for games when (obviously) the game to be tested is a PC game.

There is a study coordinator that lays down all the ground rules. The rules are generally the same for single player games. He/she tells us that there are instructions in front of us that will tell us how to play the game. The study coordinator will not tell us how to get through a game – this is probably to allow them to figure out how intuitive everything is and if the instruction booklet is missing anything. The study coordinator also tells us not to worry if we suck by saying the game is being tested, not the individuals. For multiplayer games, the study coordinator is more involved. He/she tends to tell us when a round needs to end, what maps to play, and even balancing teams if they seem unfair. It was really fun (at least for me) when I was team switched multiple times because I was dominating at Game X.

After all the ground rules are set, we play. Depending how long the session is, at set intervals, we’re told to stop playing. We then immediately fill out a survey regarding what we just played. The questions include “What’s fun?” “How much fun are you having?” “What isn’t fun” “Did you understand X part of the HUD?” and so on – some questions generic, some very specific to the game. Most of the questions are on some kind of number scale such as 1 for not fun to 5 for very fun – very quantifiable information. The same questions are asked at each stopping interval to see how our opinions change as we progress which I thought was interesting.

When playing, one thing I do when playing 360 games is click on the guide button. Sometimes there’s been a few games stuck in the game history and I get to see what else they’ve been working on. I haven’t seen any surprises but it was exciting – no secret game reveals or anything like that – but just to see that certain games where being worked is strangely exciting, especially hotly anticipated ones

For long sessions, every two hours we go into a room a separate room to break. The room is essentially a conference room with one of those long rectangular tables in the middle with chairs around it. This gives time for people to use the bathroom and stretch their eyes. There is a projector TV in the room. We’re allowed to talk but we can’t talk about the game as to not influence each other’s opinions about the game. For long sessions (8 hours), we get fed at the half way point. It has always been pizza. Back in the day when Microsoft was the Google of its time, one of its famous perks is unlimited drinks – soda, juice, water, etc. The study coordinators take us to a break room to raid their fridge for drinks.

The process of breaks, play, and survey periods repeats until the end of the session at which point we fill out a final evaluation survey. Upon completion we get our gratuity and are escorted out. The process is fairly straight forward from start to finish. It seems like their goal is to gather quantitative data from the survey. Figure out how many people of X liked Y. Straight forward, easy to measure, data.

The other kind of session I’ve done gathers qualitative data and it’s very different. First of all, instead of gathering a group of testers, this test is one on one between the play tester and the coordinator – kind of unnerving at first. Also, instead of going into a large room with multiple stations, it’s in a single small room which itself is interesting. It’s a room that measures approximately six by six feet – point being, it’s small like a closet. The room has a single, extra cushy arm chair with a TV in front with a 360 or PC unit. If it’s really early in the development cycle, it will be a PC with an Xbox 360 controller connected to it. Pointing at the arm chair are 3 – 4 cameras and a couple of microphones designed to record the testers reaction to whatever it is that will be played. There could be even more but it’s impossible to tell because the test coordinator, along with anything else (and anyone else) is behind a two way mirror.

Before this type of play session starts, I’m told very specific instructions and to follow any specific instructions during game play that will be given to me over a speaker in the room. The biggest (and longest to get used to) instruction to follow is to vocalize all of my thoughts as I play a game. For example, if I’m driving a car in a game, I can’t just zone out and drive. I have to be aware of what I’m actually thinking and put it into words. So instead of just driving, I might say “I enjoy the scenery” or “I don’t know what I’m doing so I’m just going to drive until something interesting happens.” I’ve forgotten to voice my thoughts a few times – the guy can see my expressions change from one of the many camera screens pointed at me – and had to have the instructions repeated to me.

Because games are typically in a very early state, I may have to go through debug menus in addition to just playing games. Also, the questions are different from the other play sessions – they also reflect how early the game is. Instead of a questions like “Are you having fun” with a scale, the questions are more like “Why are you having fun” which means you have to think of reasoned response. It’s really weird because most of the time when playing games, the last thing you do is think critically while you’re playing. So, it’s startling to be asked “Why” a game is fun when most of the time, one only thinks of whether or not a game is fun.

Because of the open nature, the session can vary greatly depending on how fast the instructions are gone through. Although typically, they’re on the short side.

So Why Do It?

Free shwag is good. As mentioned earlier, testers get free software. Over a year in the program (you can be called multiple times) and someone could conceivable rack up a thousand dollars worth of software. The down side to this is you would have to be required to fill out an IRS tax form when you reach a certain limit.

As someone who wants into the industry, the whole process is an interesting first look. Having some education in marketing, it’s neat seeing how market data is gathered. What I found most interesting is how text book the methodologies are. I have a marketing book I kept from college and it goes through the ideas of using surveying and direct observation to gather information.

Academic interests aside, the chance to play games before they’re released is, of course, awesome. At least, when the game is fun to play anyway. In fact, more times than not, the game isn’t fun to play. Not necessarily because it’s a bad game, but the bugs can be very frustrating. It’s a pain in the butt to get half way through the game and have to reboot the machine because of some weird error. If there’s one thing this whole experience has taught me, it’s that being a game tester would really suck. In fact, it’s spoiled some of the games I’ve played when they hit retail. For example, I might experience a lot of cool things in a game during the play test, but it will be ruined from all the bugs I encounters. When I play the retail game with all the kinks ironed out, I feel bored because I already played the game.

For me, I like the idea that I put something useful into a game. As I alluded to, I buy the games I test. Not really for fun, but because I want to see I had any impact in the game. Most of the time my complaints are ignored – the gunshots still suck, a certain enemy is too hard, this part of the design is boring, etc. But a few bits make it through and a weapon is balanced or an enemy is made easier. Even little things like that make me feel like I had some part in the whole process. It’s satisfying.




About

Hi, I'm a nobody. But I want to be Somebody - specifically when it comes to games. I've always thought I'd grow out of them, but ever since I was a little kid, I've been playing them. On my quest of becomming Somebody, I've been looking at The Biz and had some crazy ideas of my own. Instead of abandoning them to memory, I've decided to keep a log of them here. It just might come in handy some day.
August 2017
M T W T F S S
« Feb    
 123456
78910111213
14151617181920
21222324252627
28293031