Great article about Metacritic and its affects on the game industry

[Queasy]

Link

It?s fair to say that the founders of Metacritic never foresaw it generating the attention it has attracted.? ?Intended as a way of seeing at a glance whether a game was worth buying,? ?it?s now used as a measure of game quality by the largest publishers,? ?developers and retailers.?

John Riccitiello has used its scores to define EA?s business strategy to analysts?; ?Steam prominently displays them on its product pages?; ?developer Frontier uses them for sales forecasting.?

And this simple set of numbers is deemed responsible for many industry ills,? ?from over-examination of review scores to influencing developer royalties.? ??I?ve heard that publishers will try to put a step in royalty levels depending on Metacritic scores,? ?or some sort of Metacritic-related compensation structure to a deal,?? ?says Andy Eades,? ?development director at Relentless.?

Metacritic is still edited by just one man,? ?Marc Doyle.? ?But his focus remains very much on the reason why it was established in the first place.? ??I really see myself as a kind of gatekeeper to tell people that these are the games you should be paying attention to,?? ?he declares.? ?His role is to gather scores and comments for every game released in the? ?US,? ?choosing which publications are included and concocting the formula that? ?combines them into a single number.?

click through for the rest. 4 pages long.

 

[purbeast0]

while it's a pretty cool site in it's concept of having a huge "average" for every game, i can honestly say i've never once used the site to see reviews of a game to make a decision on it.

 

[R Nilla]

I use metacritic all the time, although I still usually only pay attention to the snippets from sites I visit anyway. I don't really care what sites like Edge, Wired, Variety or lesser known places have to say.

 

[erwos]

I, too, use Metacritic and Gamerankings (I prefer GR, in truth) to do research on potential purchases. The concept is excellent, and works as it should.

 

[oznerol]

I don't really use any of the reviews - except in the rare case of when there are 2 very similar games and I want to decide between the two (for example, NHL 09 vs NHL 2k9, Guitar hero vs. Rock Band, etc). Outside of that, I know what kind of games I like, I know what my friends are playing, I follow these boards to get suggestions I might not have paid attention to - and so I'll base what I buy on those things.

I'm pretty sure all of us are "in the know" about video games. Whatever arbitrary number a game gets should be irrelevant. At most, it should only give you some kind of guidance as to whether or not it's a buy vs. rent, or buy for $60 vs. buy when it gets cheap.

As far as companies using these numbers to give out bonuses and have some kind of metric to rate developers - I'm all for it - so long as both the publisher and reviewer remain independent for the most part. It's just like any performance review for any job.

And generally speaking - review scores are said to have little impact on a game's overall sales.

 

[erwos]

Originally posted by: ducci
I'm pretty sure all of us are "in the know" about video games. Whatever arbitrary number a game gets should be irrelevant.

You are both right and wrong. A single review is arbitrary. A bunch of reviews are not, when taken in the aggregate.

A bunch of arbitrary numbers are statistically significant because they begin to accurately reflect the views of the reviewer population at that point. Now, whether game reviewers are significantly different than you is a different story. I tend to think they are not that much different than most gamers, and this tends to be borne out on sites that allow viewers to rank games (in my experience).

Just writing the reviews off as "a bunch of arbitrary numbers" is like writing off opinion polls as "a bunch of arbitrary numbers". In the aggregate, they mean something, and that's why Metacritic and GR are helpful.

 

[purbeast0]

Originally posted by: erwos

Originally posted by: ducci
I'm pretty sure all of us are "in the know" about video games. Whatever arbitrary number a game gets should be irrelevant.

You are both right and wrong. A single review is arbitrary. A bunch of reviews are not, when taken in the aggregate.

A bunch of arbitrary numbers are statistically significant because they begin to accurately reflect the views of the reviewer population at that point. Now, whether game reviewers are significantly different than you is a different story. I tend to think they are not that much different than most gamers, and this tends to be borne out on sites that allow viewers to rank games (in my experience).

Just writing the reviews off as "a bunch of arbitrary numbers" is like writing off opinion polls as "a bunch of arbitrary numbers". In the aggregate, they mean something, and that's why Metacritic and GR are helpful.

i think what ducci means is that a lot of people here on this board know what games they are going to buy and skip months before they are released, and that we are "gamer concsious" enough to make our own decisions on our purchases w/out having to read reviews of them.

 

[AstroManLuca]

I think it's funny that GameRankings has been around much longer, but MetaCritic got all its attention thanks to its far superior user interface.

Honestly, who designed GameRankings? It's got a terrible layout. Lots of search options but man.

 

[oznerol]

Originally posted by: purbeast0

Originally posted by: erwos

Originally posted by: ducci
I'm pretty sure all of us are "in the know" about video games. Whatever arbitrary number a game gets should be irrelevant.

You are both right and wrong. A single review is arbitrary. A bunch of reviews are not, when taken in the aggregate.

A bunch of arbitrary numbers are statistically significant because they begin to accurately reflect the views of the reviewer population at that point. Now, whether game reviewers are significantly different than you is a different story. I tend to think they are not that much different than most gamers, and this tends to be borne out on sites that allow viewers to rank games (in my experience).

Just writing the reviews off as "a bunch of arbitrary numbers" is like writing off opinion polls as "a bunch of arbitrary numbers". In the aggregate, they mean something, and that's why Metacritic and GR are helpful.

i think what ducci means is that a lot of people here on this board know what games they are going to buy and skip months before they are released, and that we are "gamer concsious" enough to make our own decisions on our purchases w/out having to read reviews of them.

That's right.

I am sure Street Fighter 4 will get high scores across the board. But for me, they are all irrelevant because to me, fighting games are irrelevant. On the flip side, Madden 2010 can get killed by reviewers and I'll still buy it, and I'll enjoy it all the same.

Assassin's Creed is another interesting game to look at. It's ~80 on metacritic. What does that mean? Is it a slightly above average game? I don't really think so - the curve for that game's scoring looks like 2 humps - one around the 65-75 mark and one around the 85-95 mark. It's either mediocre or excellent. So yea, I think the actual number placed on the game is arbitrary, even if it is a statistical average.

More to the point of being arbitrary - if you don't think reviews are arbitrary, I'd like to know the difference between a 95.5 game and a 95.3 game (aside from 0.2 points). Or more so, how about the difference between a 4-star game and an 88.2 game? How about a C+ game a 3-star game?

Would you not buy a game you have been looking forward to because it only got a 70 on metacritic?

 

[R Nilla]

Originally posted by: ducci
Would you not buy a game you have been looking forward to because it only got a 70 on metacritic?

It depends on the reasoning for the score, but yes. Assassin's Creed is the perfect example. It looked great, but then I found out that it is extremely repetitive and has a lame story wrapped around it. I did still buy it, but I waited until I could get it for cheap and even then only played it for a few hours.

Reviews help to validate the opinion I have already established as to whether or not I'll be buying the game. There are plenty of games I will follow well before they are released, but I still read reviews to find out if they're any good.

Obviously, the words matter more than the score, but the score is generally reflective of the writer's opinion. I think anything beyond Ars Technica's Buy / Rent / Pass scale is unnecessary. There's no difference to me between an 8 or 10, or likewise an 8.4 or an 8.3 or a 9.1. If I'm interested in that game I likely will be buying it because that score essentially falls into the Buy category. If it's a 6 or 7 I'll have to look at the reasoning behind why it was rated lower, and decide if that will affect my enjoyment.

For established franchises it's easy to decide. Gears 2 was a no-brainer. There are also plenty of games that I won't follow or maybe won't even know about that I will later check out because of rave reviews.

 

[oznerol]

Originally posted by: R Nilla

Originally posted by: ducci
Would you not buy a game you have been looking forward to because it only got a 70 on metacritic?

It depends on the reasoning for the score, but yes. Assassin's Creed is the perfect example. It looked great, but then I found out that it is extremely repetitive and has a lame story wrapped around it. I did still buy it, but I waited until I could get it for cheap and even then only played it for a few hours.

Reviews help to validate the opinion I have already established as to whether or not I'll be buying the game. There are plenty of games I will follow well before they are released, but I still read reviews to find out if they're any good.

Obviously, the words matter more than the score, but the score is generally reflective of the writer's opinion. I think anything beyond Ars Technica's Buy / Rent / Pass scale is unnecessary. There's no difference to me between an 8 or 10, or likewise an 8.4 or an 8.3 or a 9.1. If I'm interested in that game I likely will be buying it because that score essentially falls into the Buy category. If it's a 6 or 7 I'll have to look at the reasoning behind why it was rated lower, and decide if that will affect my enjoyment.

For established franchises it's easy to decide. Gears 2 was a no-brainer. There are also plenty of games that I won't follow or maybe won't even know about that I will later check out because of rave reviews.

What I bolded is exactly right. And it's the leading cause of people getting incredibly pissed when a game they're anticipating gets a low score (or even a slightly lower score than a game they did not anticipate).

Edit: But this is a tangent from the point of the article - and that's how metacritic is apparently killing the gaming industry (it's not).

 

[Oyeve]

Never a fan. They had given Fable an 85 and then a year later they gave Fable:The lost chapters a lower score. WTF? It was the same game with MORE content! How can that be a bad thing?

 

[BoberFett]

Originally posted by: AstroManLuca
I think it's funny that GameRankings has been around much longer, but MetaCritic got all its attention thanks to its far superior user interface.

Honestly, who designed GameRankings? It's got a terrible layout. Lots of search options but man.

I prefer GR myself.

 

[gorcorps]

Originally posted by: Oyeve
Never a fan. They had given Fable an 85 and then a year later they gave Fable:The lost chapters a lower score. WTF? It was the same game with MORE content! How can that be a bad thing?

What do you mean 'they'? If you're talking about metacritic, then it's not thier fault as they just compile and average all the reviews from other sites.

 

[Inspector Jihad]

Originally posted by: erwos

Originally posted by: ducci
I'm pretty sure all of us are "in the know" about video games. Whatever arbitrary number a game gets should be irrelevant.

You are both right and wrong. A single review is arbitrary. A bunch of reviews are not, when taken in the aggregate.

A bunch of arbitrary numbers are statistically significant because they begin to accurately reflect the views of the reviewer population at that point. Now, whether game reviewers are significantly different than you is a different story. I tend to think they are not that much different than most gamers, and this tends to be borne out on sites that allow viewers to rank games (in my experience).

Just writing the reviews off as "a bunch of arbitrary numbers" is like writing off opinion polls as "a bunch of arbitrary numbers". In the aggregate, they mean something, and that's why Metacritic and GR are helpful.

but taken is rated so low on rotten tomatoes

 

[Inspector Jihad]

Originally posted by: Oyeve
Never a fan. They had given Fable an 85 and then a year later they gave Fable:The lost chapters a lower score. WTF? It was the same game with MORE content! How can that be a bad thing?

its cause of the douche reviewers...just like how HL2 is a 96, as is the orange box...which contains HL2 + so much other awesome shit.

 

[mugs]

Originally posted by: Inspector Jihad

Originally posted by: erwos

Originally posted by: ducci
I'm pretty sure all of us are "in the know" about video games. Whatever arbitrary number a game gets should be irrelevant.

You are both right and wrong. A single review is arbitrary. A bunch of reviews are not, when taken in the aggregate.

A bunch of arbitrary numbers are statistically significant because they begin to accurately reflect the views of the reviewer population at that point. Now, whether game reviewers are significantly different than you is a different story. I tend to think they are not that much different than most gamers, and this tends to be borne out on sites that allow viewers to rank games (in my experience).

Just writing the reviews off as "a bunch of arbitrary numbers" is like writing off opinion polls as "a bunch of arbitrary numbers". In the aggregate, they mean something, and that's why Metacritic and GR are helpful.

but taken is rated so low on rotten tomatoes

The thing about movie reviews is that movie critics don't really look for the same thing the general public looks for in a movie. I trust IMDB ratings more than I trust Rotten Tomatoes ratings, though with some movies I wonder how many people rate based on how much they thought they were supposed to like it (i.e. people are embarrassed to admit they didn't like a Best Picture contender).

 

[BD2003]

I personally think its a very good thing that theyre working so hard to get high metacritic scores.

It seems that unlike music or movie reviews, game reviews tend to be a much more relevant measure of overall quality, especially when taken in aggregate. They can try and pay off a site to get a single good review, but to make those metacritic scores move, they simply need to make a good game, and theres no way around it. I certainly dont agree with every review, and theres been plenty of 9's that just didnt do anything for me, but by and large, game reviews are a pretty good guide, and metacritic even better.

 

[Queasy]

Originally posted by: mugs

Originally posted by: Inspector Jihad

Originally posted by: erwos

Originally posted by: ducci
I'm pretty sure all of us are "in the know" about video games. Whatever arbitrary number a game gets should be irrelevant.

You are both right and wrong. A single review is arbitrary. A bunch of reviews are not, when taken in the aggregate.

A bunch of arbitrary numbers are statistically significant because they begin to accurately reflect the views of the reviewer population at that point. Now, whether game reviewers are significantly different than you is a different story. I tend to think they are not that much different than most gamers, and this tends to be borne out on sites that allow viewers to rank games (in my experience).

Just writing the reviews off as "a bunch of arbitrary numbers" is like writing off opinion polls as "a bunch of arbitrary numbers". In the aggregate, they mean something, and that's why Metacritic and GR are helpful.

but taken is rated so low on rotten tomatoes

The thing about movie reviews is that movie critics don't really look for the same thing the general public looks for in a movie. I trust IMDB ratings more than I trust Rotten Tomatoes ratings, though with some movies I wonder how many people rate based on how much they thought they were supposed to like it (i.e. people are embarrassed to admit they didn't like a Best Picture contender).

Not to mention that snobbery runs rampant with movie reviews with certain genres. Your over-serious Oscar worthy dramas or artsy independent films are going to get a lot more attention than your entertaining comedy or action film.

Game reviewers aren't so much genre snobs as they may be platform snobs but that isn't generally as much of an issue with the reviews as it is with the fanboys who read the reviews.

 

[erwos]

Originally posted by: ducci
More to the point of being arbitrary - if you don't think reviews are arbitrary, I'd like to know the difference between a 95.5 game and a 95.3 game (aside from 0.2 points). Or more so, how about the difference between a 4-star game and an 88.2 game? How about a C+ game a 3-star game?

I never said individual reviews aren't arbitrary. I did say that, in the aggregate, lots of arbitrary opinions are statistically significant and mean something. Indeed, take a list of the top ten games people on this board loved this year, aggregate it, and I will guarantee you that there is a high correlation to high Metacritic scores. _If MC scores were totally divorced from reality, that wouldn't happen._

Again, you need to make the distinction between an individual score from a reviewer and the composite score Metacritic derives.

Would you not buy a game you have been looking forward to because it only got a 70 on metacritic?

I would be far less likely to buy that game than one that scored a 95, yes, holding everything else the same. But I'd also read the reviews to figure out why it got a 70, too. I buy stuff that got 60-70s on MC, but I damn well research it first. Fundamentally, that's what GR and MC are - research tools. I don't think anyone's seriously advocating that you should never buy a game that ranked less than 80 on GR or MC, but denying that high GR and MC scores are correlated to overall game quality is a real stretch.

When will gamers figure out that previews are universally positive because previewers don't generally mention all the stuff wrong with the game? If you're basing your buying decisions on previews and ignoring reviews, you're doing it absolutely the wrong way around.

 

[BD2003]

There's a place where there's a huge disconnect between the scores, reviewers and reality. For most people, they only have enough money to buy a few games a year. There are more 85-90+ games each year than people can afford. So a reviewer might say a 70 or 75 is a good game, but it's not going to get me to open my wallet unless its for some sort of niche reason. Thats why metacritic is so important. If you rely on one site, you wont get that larger picture.