How should companies judge a game’s success?

The interview held with Peter Moore has sparked some good discussion and the reaction has been interesting to follow. One of the more frequently commented upon topics is Moore’s use of Metacritic as a major basis for considering Madden 08 to be an excellent game. While in some ways it may have been, everyone would agree that there is a lot of room for improvement in the series.

That got me to thinking. I’m of the belief that Metacritic is not the best method for determining how good a game was, how well it has been received, or what would need to be improved on going forward for reasons I’ll go into shortly. The Wall Street Journal touched on this topic with a piece about how important the scores are to companies (a great read that can be found here). What other ways could the overall success and reaction to a title be determined if not by reviews?

• Metacritic and Reviews

Metacritic grabs review scores from a pool of established sources to present a final average. In theory this should be a perfect way to compare it to other games and give a relative picture of how good it is. Unfortunately it depends solely on reviewers, most of which have shown to be unreliable in the area of sports games.

When it comes to preparing reviews simply enough time is not put into the games to complete a fair analysis. Anyone with experience with sports games knows that in many cases for the first few hours, days, or sometimes even weeks that new game can feel fresh and exciting. However then the problems start to creep in that can ruin the experience. MLB 2K7 and Madden 08 would be two good examples of that (although I was even somewhat down on Madden right away at Community Day). So most reviewers sit down with a game and play it for a few hours before writing up their thoughts.

With Madden it just simply isn’t plausible to expect reviewers to spend hour upon hour playing through one of the deepest sports games ever made while getting the review prepared and out prior to the release date. You can’t expect them to get through a ton of play now to be able to discuss the gameplay, play through years of franchise mode, play through a bunch in Superstar mode, and find other reviewers to play against online. So obviously they generally don’t reach the point where the negatives start to overshadow the positives or have an opportunity to include a detailed overview on everything included in the game.

In many cases reviews end up reading like previews with scores attached. They’ll go down the line of features but there is no way they played through Madden’s franchise mode long enough to realize it was broken and required a patch. We know they didn’t play MLB 2K7’s online leagues to find out they didn’t work. But of course both are praised within reviews for those games and factor into the scores. You can even look at MLB 2K6 on the 360, a game that froze before completing three innings rendering it unplayable and went patch-less for weeks, yet that game somehow remarkably has an average score of 64%.

That brings me to the area of online play, which as I’ve written about before is completely ignored in these games (read through that link for some good examples). The most you’ll get is a recap of the fact sheet for online and some generic comments. It’s obvious they don’t even play online given they’ve bypassed discussing broken 2K leagues or NCAA Football and Madden’s horrendous online experiences. How a full mode could be ignored in reviews and that be acceptable is beyond me.

One thing I’ve noticed with reviews in general is that the fun factor has almost completely been ignored. To me that is the most important thing in gaming yet it gets the least representation in reviews. In the end fun factor is going to be subjective yet that shouldn’t mean it isn’t considered. Was NFL Tour a deep enriching experience? No. Did it have a good single player mode? No. Was it fun as a multiplayer game? Absolutely. But if you read the reviews you’d think no one out there enjoyed the game which simply isn’t the case. When I get more play time in with NFL Tour than many of the sim games of the past year I couldn’t care less whether someone thinks the game deserves a three or a nine and I know I wouldn’t have had any idea how to label that with a score of my own.

While there are a few reviewers out there I would take under consideration, the vast majority don’t give the proper care and respect to sports games. So as a whole an average score isn’t the be-all-end-all on how good a game really is.

• Sales

Companies tend to look at sales as another factor in how well a game was received. However in many cases this may in part be attributed to the previous year’s title and the marketing push. Although still very strong Madden’s sales have dropped over the past few years. I feel that is due in large part to the shift in console generation, but it could also have to do with disappointment in the efforts of the past few years. And because the marketing varies and has such a huge effect that makes it difficult to evaluate the current year. Word of mouth takes a little while to spread. So here we have sales and Metacritic contradicting each other in the case of Madden.

• Community Reaction

Ultimately the online community represents only a small percentage of those who purchase the games but it is an important group. Some great feedback and suggestions can come from here and the more hardcore fans of the games have some insight that should be considered.

Again as time is spent with the game a lot of the time reaction starts out positive and turns negative. If all you were to read was the online message boards you’d think Madden 08 was the worst game ever made. Of course it wasn’t nearly that bad but the negativity spreads and takes over. There definitely is the sense that the hardcore group as a whole are just nitpicking whiners but given the right outlet there is the potential to turn the community into a positive. It’s also important to remember that the majority of people who are enjoying the game don’t go out of their way to post about it.

What can really be taken from the community is feedback through improved channels of communication and that is something EA deserves credit for. Whether they act on it is another story, but at least they’re listening.

• Miscellaneous

In the case of Madden you also have external events that are looked upon as representing the game. Those include the Madden Challenge and Madden Nation TV show. I’ve always felt that these represent EA, the NFL, and the majority of gamers poorly however they continue to be a success in terms of marketing. Still Madden Challenge numbers have dropped over the past few years. That isn’t likely due to the quality of the game so much as it is the environment and attitudes presented.

The companies conduct polling to get a general idea of thoughts from a wide range of people. This is one of the better ways to gauge reaction and what people want out of the game however a lot depends on how the polls are conducted. The type of people taking it will alter the results. Do it online and you’ll get vastly different responses than doing it at a Madden Challenge for example.

• Final thoughts

In the end there is never going to be a single way to determine the final worth of a game or how well it turned out. Opinions will inevitably vary because of its subjective nature. All of the factors I laid out are going to be considered in some shape or form and taking them together should give at least a general picture. But the reliance on reviewers is certainly troubling from the perspective of a fan of sports gaming that just wishes for them to improve in areas such as fun and authenticity.

So what do you think these companies should be doing to get a true feel for the games and the direction they need to go? Leave your thoughts in the comments.