A Brief Primer on Methods and Critiques
Over the years since I first put out the Norrathian Scrolls, I've read many reactions from the player base to the findings presented. On forums and message boards, some players draw out critiques that they argue invalidates the entire study. While some of these critiques are valid, their severity is often overestimated. More importantly, it may be hard for non-statisticians to understand why a valid critique may in fact not matter for many of the findings presented here at The Daedalus Project. In retrospect, I should have provided a primer to these potential weaknesses long ago, but it's better late than never. I'm going to start with the more simple critiques and move onto critiques that appear harder to resolve.
Small sample size. Some players argue that a survey of 2000 gamers from a player base that numbers in the millions won't show anything, but sampling is inherent to almost all surveys. For example, the Gallup Poll survey less than 1% of the total US population. A small sample size in and of itself doesn't say anything about the representativeness of the findings. Oftentimes, people on forums will say that a sample of 2k is unreliable and then they will go on to expound their own opinion. Please call them out on their sample size of 1.
Players can't possibly be that old. Some balk at the finding that the average age of MMORPG players could possibly be as high as 26 and thus all other results must be wrong. Oftentimes, they claim that the average age must be closer to 18-22 because they always feel like they're interacting with adolescents in the game. But industry reports have shown that the average age of video game players in general is 30, so the MMO average from the surveys is actually lower than the overall average.
====
Biased sampling. But the underlying argument of the "age mismatch" critique is a valid one. For example, some argue that only hard-core players would care to fill out a survey. Or that younger players are too lazy to fill out surveys. Or that the informed consent for minors discourages participation. These critics then conclude that all findings derived from this biased sampling must therefore be flawed.
This is actually not the case. Let me show this with an example. We know that the real-life gender ratio is close to 50-50. On the other hand, we know that the gender ratio in MMOs is much closer to 85-15. Imagine that I ask MMO players to tell me their height in RL and I find that women are on average shorter than men. Is it the case that I cannot extend my finding to the real world because my gender ratio is significantly deviant from the actual ratio? Or is it the case that the validity of this particular finding isn't tied to overall sampling representativeness?
We need to make something clear. While it is true that my MMO dataset would show that the gender ratio is 85-15 (incorrect, since the RL ratio is 50-50), what I am trying to show is that this is independent from the secondary finding (that there is a height difference between the two genders). In other words, it is very possible to find gender differences, age differences, differences between Alliance vs. Horde, and so on even though the overall sampling may be a bit skewed.
And these secondary differences are what The Daedalus Project is more focused on. To me, knowing the precise gender ratio in MMOs is not very interesting. What is more interesting are gender differences (or age, or personality differences) in game play behavior. Thus, the critique of an overall biased sampling, even if it did exist, may have a negligible effect on the bulk of findings at The Daedalus Project. In other words, even if the real gender ratio was 75-25, it would probably still be the case that male players gender-bend more often than female players.
====
But here in this short article, I have been on the defensive and I would like to take some time to emphasize how well the survey data here has actually matched up with other data points. After all, empirical research is not about instant gratification, but instead it is often about slow accumulation of data and sifting of incongruous findings over time.
Match with data from other academic studies. Most of the key demographics variables match up well with other online survey studies of MMO players.
Griffiths, M.D., Davies, M.N.O. and Chappell, D. (2004). Online computer gaming : A comparison of adolescent and adult gamers, Journal of Adolescence, 27: 87-96.Griffiths, M.D., Davies, M.N.O. and Chappell, D. (2003). Breaking the stereotype: The case of online gaming, CyberPsychology and Behavior, 6: 81-91.
Match with company marketing data. Most of the key demographic variables match up with data provided in a marketing report on MMOs. A specific section on Sony's EverQuest draws on data provided by SoE.
Source: DFC Report
====
Match with aggregate WoW census data. Perhaps the most compelling support comes from a data source that isn't based on the survey methodology. The PARC PlayOn data is based on census snapshots from within World of Warcraft. Every 10-15 minutes, an automated script collects information from every single character online. In other words, this data source is almost as perfect as we can get in terms of actual behavioral data.
For example, from the survey data we were able to get the average achievement motivation of players who prefer different classes in WoW. From the PlayOn data, we were able to get the average time it take different character classes to make each level. The classes who score low on the achievement motivation (survey data) are also the ones who level the slowest (census data).
Of course, not every data point from the survey can be supported by the census data. After all, the census data can't track RL age or gender information. Nonetheless, this congruence goes a long ways to showing that the survey data is able to reflect actual behavioral data in a real game.
On the other hand, there are no perfect methodologies. Every methodology comes with its strengths and weaknesses. But the weaknesses of a methodology must be understood in their proper context and scope. Thus, while some argue that the sampling bias invalidates the study, it in fact has very little to do with the bulk of the findings presented at The Daedalus Project. Furthermore, their critiques often miss the important point that there are seldom optimal solutions to many problems. It would be nice if we could survey all MMO players. It would be nice if everyone were willing to fill out surveys. But social science researchers are not omnipotent and the world is not perfect. Thus, sometimes the most practical and sensible way to answer a question isn't perfect. But it would be unfair to judge The Daedalus Project against a perfect world of omnipotent researchers. Finally, there really isn't that much empirical information about MMO players – who they are, their preferences, and what they do in the game. And here's what I've always believed and what led to the genesis of the Daedalus Project. Knowing something is better than not knowing anything at all.