Are review aggregates a necessary evil?

[hanselthecaretaker]

So I read the entire article in this thread [http://www.escapistmagazine.com/forums/read/9.959817-Andromedas-Troubled-Development] and it made a clear case for how much importance is unfortunately placed on a review score in the final paragraphs-

Not long before Mass Effect: Andromeda launched, BioWare sent early builds of the game to mock reviewers, as nearly every AAA game developer does in the months before their game comes out. A mock reviewer will typically offer a private, early assessment of a game, a report on its strengths and weaknesses, and a predicted Metacritic range. Companies frequently make major strategic decisions based on Metacritic scores, so it's that number that gets the most attention.

When the mock reviews came in for Mass Effect: Andromeda, BioWare's leads were relieved, the Metacritic was expected to be in the low-to-mid-80s, according to two sources. Although Andromeda's developers knew the game wasn't perfect, they were fine with a score like that. If they hit somewhere between 80 and 85, they could use what they'd built for Andromeda to make the sequel way better, much like Casey Hudson and his team had done from Mass Effect 1 to Mass Effect 2.

Then the GIFs started. EA put Mass Effect: Andromeda out early for EA Access users on March 16, five days before the game came out, which led to a weekend full of memes, anger, and nasty harassment as players shared images and gifs of the game's many glitches. Combine that with the fact that three of the other games released in March 2017 turned out to be all-time classics (Zelda: Breath of the Wild, Horizon: Zero Dawn, and Nier: Automata) and you've got a recipe for low review scores. When the Metacritic score finally settled, Mass Effect: Andromeda wound up with a 70 (on PS4, where it has the most reviews), far lower than those who had seen the mock reviews expected.

The results were catastrophic for BioWare Montreal...(yada yada)

Which brings me to the question of how important should scores really be? I've heard many people in the past say "it's just a number" or "it doesn't mean anything", etc. but the opposite quite clearly seems to be the case.

 

[BloatedGuppy]

Review aggregates are a data point. Data is neutral. It cannot be good or evil, it's just data. You can argue for "good data" vs "bad data" in terms of how it's gathered and presented, but that's a whole other kettle of fish. It's not a charge that can be laid at "review aggregates" and rather at specific sites and how they carry out their business.

How companies choose to apply that data is another question entirely.

 

[CaitSeith]

I have several problems with the practice of making major strategic decisions based on Metacritic scores.

First: the effects of low scores on future decisions isn't predictable, and the general public can just speculate on what exact criteria the companies use and which actions they will take.

Second: I seriously question that strategy. Reviews are supposedly to be aimed at the consumers, not the companies. These are two different groups with different priorities. Reviewers aren't a regulatory entity that evaluate products based on perfectly defined set of rules and criteria. It's absurd to make major strategic decisions based on metacritc scores, because the reasons for each score can be too different from one to another.

Third: Fanboys see going after reviewers that give low scores to their favorite games as justifiable. Every 0.1 points below 10 is an increase on the risk of getting their favorite IP shelved forever.

Maybe it made sense back when companies had enough control on the reviews that if reviewers got out of line, they could get them fired.

 

[Kerg3927]

I'm sure they take it into consideration, but it's most likely far secondary to sales. They'd much rather have a game score 70 and sell well than one score 80 and not sell as well. Of course, scores can influence sales, but I would bet that the vast majority of gamers have no idea what scores the games they play are getting. Hell, I think half of gamers are near functionally illiterate. They don't read reviews or heavily research games before they buy or hang out on forums like this. I think the market is still mostly driven by word of mouth. If people like a game, they talk about it with their gamer buddies and it drives sales up. If they ***** about it, the opposite happens.

 

[SweetShark]

I am the kind of guy who want to go to a game I think I like completely blind.
For example if you want to enjoy games like OneShot or Undertale, you MUST do to yourself a favor and don't read a review.
However if I am not sure for a game I don't like, then I search reviews and I see the score.

 

[Pyrian]

Of course, scores can influence sales, but I would bet that the vast majority of gamers have no idea what scores the games they play are getting.

I'm sure sales are indeed weighted more heavily than reviews, but... Sales and review scores are relatively - relatively - well correlated in games, believe it or not. It's quite a contrast from movies, where review scores don't seem to drive ticket sales much at all, especially for heavily marketed films.

 

[Lufia Erim]

The majority of people what buy games don't do extensive research beforehand. The people who frequent forums are a minority. Even fewer are the people who read multiple reviews and watch let's plays to judge if they want to putchase a game.

Back before i started lurking on the escapist and staying informed, the only source of information i used was the description behind the game box in stores. And what do you see when you buy games now? 8/10 9/10 and 10/10 plastered everywhere on the box.

I know now that those are fake scores, but I'm willing to bet the vast majority of people who buy games, base their decisions on that.

 

[mrdude2010]

Review aggregates are a data point. Data is neutral. It cannot be good or evil, it's just data. You can argue for "good data" vs "bad data" in terms of how it's gathered and presented, but that's a whole other kettle of fish. It's not a charge that can be laid at "review aggregates" and rather at specific sites and how they carry out their business.

How companies choose to apply that data is another question entirely.

The important question then becomes: are review aggregators collecting good or bad data? It would be pretty easy to argue that they represent bad data collection and presentation. Also, the collection process is an inherent part of the data itself. You can't claim the data are neutral if the collection of it is not. A set of numbers can be neutral, but data sets are not.

 

[DrownedAmmet]

Review aggregates are a data point. Data is neutral. It cannot be good or evil, it's just data. You can argue for "good data" vs "bad data" in terms of how it's gathered and presented, but that's a whole other kettle of fish. It's not a charge that can be laid at "review aggregates" and rather at specific sites and how they carry out their business.

How companies choose to apply that data is another question entirely.

The important question then becomes: are review aggregators collecting good or bad data? It would be pretty easy to argue that they represent bad data collection and presentation. Also, the collection process is an inherent part of the data itself. You can't claim the data are neutral if the collection of it is not. A set of numbers can be neutral, but data sets are not.

I fail to see how the collection of opinions is a bad thing. User reviews are a little less reliable because people are more likely to review something if they feel strongly about it, so you will see more 10/10 and 0/10 reviews than you would with critic reviews.

But with critic reviews? I will never see the collection of the reviews done by people who make their living on reviewing games as a bad thing, it's still a great way to get a good idea of how people feel about a game.

Anyone who thinks they are more than that is a stupid jerk

 

[Phoenixmgs_v1legacy]

It definitely seems like the entirety of gaming is far too obsessed with scores from gamers to reviewers to publishers. Why would a publisher tie developer bonuses to a Metacritic score unless they just wanted an excuse to not pay? Or did Bethesda seriously care so much about scores when they tied Obsidian's bonus to Fallout New Vegas' score? Would say Paramount Pictures dare tie Michael Bay's bonus to the RottenTomatoes of a Transformers movie? I'm pretty sure Bay would laugh and walk out if that were part of the negotiations. So why would Obsidian even consider a deal like that let alone agree to it? The gaming industry just seems so immature compared all other mediums from the way publishers treat devs to how reviewers review games as if functionality is all that matters in what makes a game good or bad. Where's the bad reviews for FFXIII (or MGS4 or a GTA), it's OK to not like a movie, why not a game? Then, the gamers themselves (even the far less ridiculous GFaqs gamers) who say the 2.5 Ghost Recon Wildlands review is too low because the game is objectively better than a 2.5 somehow.

Sales and review scores are relatively - relatively - well correlated in games, believe it or not. It's quite a contrast from movies, where review scores don't seem to drive ticket sales much at all, especially for heavily marketed films.

That's because game reviewers don't really critique games, they just score a game on how functional it is because it was decreed games much be rated "objectively". It's usually pretty hard for most AAA games to score less than an 80 because they work for the most part. There isn't actual criticism in video game reviewers like other mediums. Look all the hate Greg Tito got for his GTAV (7/10) review because he didn't like the writing ("GTA isn't about writing!!!") or his Dragon Age 2 (10/10) review because he genuinely loved a game most gamers didn't. Apparently the gamer community proclaimed his opinions to be wrong.

 

[Imperioratorex Caprae]

I swear publishers act like a good MC score is like winning a fucking Oscar. And video game awards are a joke as it is... I'm not saying critics are useless, but rather people will buy what they buy regardless of score because most folks don't give a shit to read reviews. Sad part is that those numbers push publishers to pressure developers in the wrong way, be it adding features that aren't needed or copying other games aesthetics because "we gotta be like them" attitudes. Its not the numbers, its the fucking MBA that runs the show, folks who have no business being in games because they have no love for them.

 

[DoPo]

Review aggregates are a data point. Data is neutral. It cannot be good or evil, it's just data. You can argue for "good data" vs "bad data" in terms of how it's gathered and presented, but that's a whole other kettle of fish. It's not a charge that can be laid at "review aggregates" and rather at specific sites and how they carry out their business.

How companies choose to apply that data is another question entirely. It would be pretty easy to argue that they represent bad data collection and presentation. Also, the collection process is an inherent part of the data itself. You can't claim the data are neutral if the collection of it is not. A set of numbers can be neutral, but data sets are not.

The important question then becomes: are review aggregators collecting good or bad data?

Neutral. It can only be "bad data" if it's misused, but if you misuse it, by definition you're not using it correctly. Let's say you have a family of four walk into a bar - the parents are 32 and 34 years old, the children are 8 and 10 years old. This is is a neutral set of data. Their average age is 21.25. This is an aggregated data. It is not good nor bad by itself. Let's say the bar then decides to just serve the children hard liquor based on the average age of the family.

Similarly review aggregators also present neutral data. It's the average score for a game. If you want to implicitly trust that to be "the true" score for said game, that would be misusing the data. An average score of 6 based on 10 and 2 is different to an average of 6 based on a 5 and 7. The score on aggregate sites can just give you an idea but if you really want to know why that score is chosen, you'd go and do further research, in order to not misuse the data.

 

[TheMysteriousGX]

I fail to see how the collection of opinions is a bad thing. User reviews are a little less reliable because people are more likely to review something if they feel strongly about it, so you will see more 10/10 and 0/10 reviews than you would with critic reviews.

I swear to god, the number of 10/10 reviews I see that boil down to, or explicitly say, "eh, it's alright" is staggering. And that's discounting the ones for a contentious game that swing 10/10 or 0/10 because "f**k those guys".

 

[kitsunefather]

The main thing in the game industry, as far as I remember, is that the developers get bonuses if a game scores 80+ on Metacritic. This was another source of contention a few years ago with reviewers, as it came to light that some were bumping up scores (if it's a 70 or 75) to an 80 to make sure their friends who are devs get their bonuses.

Because of the nature of what the "Review" has become, anything under 80 is considered a failure by a lot of the general public and game companies.

For the record, I think this is stupid.

 

[Pyrian]

Or did Bethesda seriously care so much about scores when they tied Obsidian's bonus to Fallout New Vegas' score? ... So why would Obsidian even consider a deal like that let alone agree to it?

As I understand it, these sorts of arrangements are indeed driven by the developer. The publisher would of course rather tie it to sales, but sales depend not just on quality but also on marketing - which is done by the publisher. So developers wanted a metric they felt they had more power over. IIRC, there were a handful of highly-scored games with very little marketing that drove this arrangement.

 

[Phoenixmgs_v1legacy]

As I understand it, these sorts of arrangements are indeed driven by the developer. The publisher would of course rather tie it to sales, but sales depend not just on quality but also on marketing - which is done by the publisher. So developers wanted a metric they felt they had more power over. IIRC, there were a handful of highly-scored games with very little marketing that drove this arrangement.

Ah, I can see that. It's just weird that you can predict review scores better than sales and says something about the game industry vs other mediums.

 

[Major_Tom]

By "it's just a number" and "it doesn't mean anything" we mean that the number was chosen arbitrarily, not that the number has no impact. Review aggregates are even worse since they take a bunch of already arbitrary numbers and mash them up into an even.. umm.. arbitrarier number. The arbitrariest number.

 

[Wrex Brogan]

It is 'just a number', it's just the industry treats it as something far larger than it should be in terms of marketing, especially with the loathing that is given whenever something doesn't score high enough. a 6/10 or 7/10 is treated like a death sentence for a game, even though those are objectively 'good' scores in terms of a neutral data system.

It'd be nice it if didn't have so much impact - especially given Metacritic just gathers everyone elses shit into one big shit-pile - but given this is also an industry that will consider a game selling 5 million copies a failure... yeah.

 

[Vanilla ISIS]

I don't like how these aggregates work.
For example, a movie on RottenTomatoes needs to get a 60% or over to be considered "fresh".
For the aggregate, the score isn't counted, just the "fresh" or "rotten" label and the percentage of those are used to create the final score.
That means that a movie which got all the reviews at 60% can get a 100% fresh rating while a movie which got a ton of 100% but several below 60% scores will get a lower rating.
What's worse is that all it would take to change that is just replacing a few algorithms, meaning 1 person can do it in less than a day.

 

[Pyrian]

RottenTomatoes displays the "Average Rating" for each movie, as well, if you want to see it. RT's primary algorithm rates broad appeal over any other metric, and is intended to do so; you may not like it, but it's not a "mistake" per se. There's only such much information you can pack into a single number (nevermind a single binary). I wouldn't mind seeing a "mean+standard deviation" rating, but I have to admit, I think it would confuse more people than it would inform.