Metacritic Scores vs. Sales

[mindcycle]

I found this article pretty interesting given all of the scandals we heard about last year with publishers trying to influence game reviewers so their games would score decently on metacritic.

Metacritic Scores vs. Sales: 2009 Holiday Season Analysis

Recently in GamePro, Julian Murdoch examined the reviews aggregator site Metacritic. His story opens with a rather startling observation from Activision Vice President of Marketing Robin Kaminsky at the 2008 Design, Innovate, Communicate, and Entertain (DICE) Summit. The opening slide to her presentation read: "For every additional five points over an 80 percent average review score, sales may as much double."

Incredible! If true, game sales must increase exponentially as they score above an aggregate of 80 on the website. Surely, an employee of Activision -- let alone a VP of marketing -- must have solid data to back up such a bold declaration.

But I'm the skeptical type, so I did a little of my own research. I’m here to tell you that Kaminsky's statement is a load of horse shit, and that publishers' insistence on a correlation between scores and sales is unfounded.

My analysis demonstrates that no link between Metacritic scores and sales exists. So why do publishers insist on such a connection? I can’t really say. That reviews scores would influence sales does feel intuitive. Perhaps publishers also feel reassured that they've based their decisions on a quantifiable number.

But I won't pretend that I have all the answers. I don't really know why or why not a particular game sells well or poorly. My data isn't equipped to answer that question, either. What I do know is that no correlation between Metacritic scores and unit sales exists, and I think it's foolish for publishers to tie monetary compensation to such scores.

If this topic interests you I highly suggest you read the full analysis here: http://www.bitmob.com/articles/metacritic-scores-vs-sales-2009-holiday-season-analysis

 

[Terzo]

Interesting but I'm not really sure what conclusions (besides the stated) to draw from it. My guess is that many poorer quality or budget games are bought by people who don't read reviews. They probably market themselves to that segment as cheap, kid-friendly, or something similar.
Making a game targeted towards "hardcore gamers" is probably more difficult, since you actually have to make a good game. And even then you need marketing or your game will fall under the radar.
Now that I think of it, pricing probably plays a large role as well. I often see the sentiment (especially with steam sales) that at price X you can't really go wrong. I see many people post in Steam threads about their large backlog of games, probably because most of them were bought at impulse prices.
Like I said it is an interesting article, but I wonder, what does it mean for gamers in general?

 

[BenSkywalker]

Unfortunatley, while he does a great job of presenting his case, his numbers fail.

Two titles jumped out at me, Unchartered 2 and Forza 3. He quotes the sales of those games as 1.1Million and 963K respectively. Actual numbers, using the *same source* he did is 3.18Million and 2.53Million respectively.

http://www.vgchartz.com/games/game.php?id=28733
http://www.vgchartz.com/games/game.php?id=30434

I understand he tracked titles for a certain amount of time and reported the numbers as such, but the real tie between Metacritic and overall sales is going to show in terms of sales based on 'legs', not initial sales impact. The higher quality a game is, the more likely it is to retain its sales for a longer period of time. I brought those two titles up just to demonstrate the point, higher review scores do in fact have a very strong relationship with Metacritic scores, they just don't necessarily happen within a 10 week period.

 

[mindcycle]

Unfortunatley, while he does a great job of presenting his case, his numbers fail.

Two titles jumped out at me, Unchartered 2 and Forza 3. He quotes the sales of those games as 1.1Million and 963K respectively. Actual numbers, using the *same source* he did is 3.18Million and 2.53Million respectively.

http://www.vgchartz.com/games/game.php?id=28733
http://www.vgchartz.com/games/game.php?id=30434

I understand he tracked titles for a certain amount of time and reported the numbers as such, but the real tie between Metacritic and overall sales is going to show in terms of sales based on 'legs', not initial sales impact. The higher quality a game is, the more likely it is to retain its sales for a longer period of time. I brought those two titles up just to demonstrate the point, higher review scores do in fact have a very strong relationship with Metacritic scores, they just don't necessarily happen within a 10 week period.

I'm assuming you meant to write..

higher "sales" do in fact have a very strong relationship with Metacritic scores, they just don't necessarily happen within a 10 week period.

It's a really good point you bring up though. Without further analysis it would be hard to to prove, but it may be closer tied than he's pointing out. I still think publishers determining game advances and bonuses off of metacritic scores is pretty ridiculous. But more-so them trying to influence review scores to bump up the average.. that's even worse.

 

[AstroManLuca]

I think you could actually turn things around, e.g., bigger sales = higher scores. Well, most reviews come out before any sales numbers are in, so more like bigger expected sales = higher scores.

I often get the impression that the more anticipated and hyped a game is, the higher the review scores. GTA IV, for instance. IGN gave it a 10.0. Is it because it is a perfect game? No, it's because GTA IV was hyped and marketed like crazy. I wouldn't be surprised if IGN was paid for their score. But even if there were no under-the-table deals going on, I think reviewers are pressured to give high scores to certain games. If they give it a lower-than-expected score, they are accused of being biased against some aspect of the game and may face the loss of their job.

What's interesting is you can read a ton of reviews and get a good idea of what a game is like, but the scores may or may not have anything to do with what is actually said in the reviews. Not to mention Metacritic always converts review scores to a 100-point scale, even if the original reviewer uses a 5- or 10-point scale. So a 4/5 score may be considered really good, but when it's translated to 80/100 it doesn't look as good (even though 80% should be considered good, it seems these days a game isn't "good" unless it gets a 92+).

 

[exar333]

Remember this if you don't remember anything else at all.

Correlation <> Causation!

🙂

 

[smackababy]

If I remember correctly, aren't a lot of "reviewers" flown into areas and given crazy perks to review upcoming blockbuster titles?

Also, the number of buyers who actually read reviews VS the number of buyers who see the cool trailer and buy based off that would be a far more interesting number. I suspect, marketing has a lot more to do than a metacritic score. I've never looked something up on metacritic myself, and only know of a few games scores due to it being integrated into Steam.

I think the greater you can hype your game, the better your sales. Even if reviews give it bad to mediocre reviews, if it is hyped well enough, people are going to buy it. That, and a cool trailer, are probably the biggest sellers.