Is game reviewing 'broken' as a system?

[Amnestic]

In yesterday's Mailbox video [http://www.youtube.com/watch?v=o46KJeeVU7Q] TotalBiscuit raised the point that reviewing as a number based system had changed since he was a wee laddy. He stated that back in his day, 5/10 was average, 1/10 was bad and 10/10 was almost unreachable. Contrasted, he said that in today's world "anything below 8 is terrible, 8 is average, 9 is good, 9.5 is really good and anything between 9.5 and 10 is excellent, and 11/10 is amazing, and there is no score for perfect because that game doesn't exist and they've already used one more number than they should."

Direct link to the part where he starts reading and then proceeds to answer the e-mail [http://www.youtube.com/watch?feature=player_detailpage&v=o46KJeeVU7Q#t=632s].

My questions to the Escapist are thus:
Do you think he's right? - Both on the subject of 'how things were before and how things are now

Who do you think is to blame for the score 'inflation' that we see? Gamers, desperately crying out for validation that their chosen game is AMAZING? Reviewers, hoping to drive up traffic either via nerd-baiting or simply by giving a positive review? Developers/Publishers in an attempt to get their game more positive press in the hopes that they sell more copies? All of the above?

Do you think this should change? Why? And if you do think it would change, is there any way that we as gamers can make that happen?

To answer my own questions: I think he may have a slightly romanticised version of the olden days. At 27, he's not that much older than I am and I highly doubt that gaming journalism changed so much in the intervening time between his consumption of it and mine. I will grant that perhaps my consumption was in the middle of when things started to change though. I'd be interested for input from older Escapists about this.

As for how things are now, I'm inclined to agree with him. One need only look at the proverbial shit storms that erupted following recent reviews of blockbuster titles - BF3, Uncharted 3, MW3 are of particular note. I daresay that if a reviewer gives Skyrim a low score, we may just have a riot on our hands.

Who's to blame? Well, I'd put most of the blame on the general gaming public, spurred on by the reviewers in question. We get what we ask for, and people ask for these high review scores.

I think it should change, I'm tired of seeing 7-8 be an 'average' score in a ten point scale. It's just weird, and spreading scores out a bit may help to kill some of the screams of fanboys/girls. How can it change? Honestly I don't know. Besides a revolution throughout the gaming public (which considering it numbers in the double-if-not-triple digit millions, may be quite hard) I don't really expect things to change. Maybe small segments of the internet can change things; put less emphasis on scores, more on content, cut out scores altogether...but for all the hate that gamers give the score-system, they still flock to them. They still hit up metacritic. They still glance at review scores at the top/bottom of a review to colour their reading of it.


Honestly, I'm not sure it'll ever change. We may just have gone past the point of no return. I dunno.

Thoughts?

 

[Laser Priest]

The system's not broken. It's just shit to begin with.

Personally, I don't think you can measure quality and put a definitive number on it and it also seems to discourage people from actually reading the review to see what is good and bad about a game.

The number rating system is simply accommodating laziness.

 

[])rStrangelove]

He's partly right. Another problem is that a reviewers 8 points are not my 8 points.

I would stay away from points or scores altogether and just list all PROs and CONs and then everybody can decide for himself to get it or not.

 

[LordRoyal]

Game Reviewing's always going to be around, just the way people are doing it on something like say a number rating system is just pandering to people that are skimming your review

 

[2xDouble]

Of course the system is broken. That's because there is no system. You can't assign uniform numerical value to personal impressions. The numbers aren't on the same scale, therefore they cannot be compared directly, therefore there is no "system".

 

[Chezza]

There is no universal score system that considers the same factors and opinions so its difficult to get a consistent and reliable system. But I have noticed getting a 6/10 for a game is rare and if it is, the game is considered extremely crap which deserves a 3/10.

My bigger concern is that many reviewers could be sell outs. For example I cannot honestly accept so many CoD releases receiving so much praise and high scores with their typical recycle behavior meanwhile very similar titles but under a different company gets criticized for being "unoriginal". Total Hypocrisy.

 

[Woodsey]

Games are reviewed with words, not numbers. If people didn't put so much emphasis on the score then there wouldn't be an issue.

As for scoring being broken, it varies from publication to publication.

 

[Clive Howlitzer]

I'd say it is about as broken as news reporting is. You can't rely on a single source for your news and you can't rely on a single source for your gaming reviews. You need to view many, as well as player reactions and hopefully, reports from friends who own it.
You mix those all together and then you get an idea of how the game is. I think the closest you can say the system is broken is game developers making decisions to try and get a high metacritic score. That I consider to be retarded.

 

[Still Life]

No.

I think it's a hijacked system with misconstrued averages, fans who don't have a fucking life and marketing taking quotes and awards out of context.

 

[Bvenged]

I don't think reviewing works any more - It's evident where as a reviewer, when you register to review a new game - if a publisher knows you're not a fan or knows you're a harsh scorer, they wont give you a copy to review. That right there is where it is broken. If you ask a fan to review something, of course they're going to score it big.

The only place I know who still does fair reviews is PCZone. They even have a "dump" award they give to games that score less than 15/100. They even give a short Adv/Disadv. summary on the first page of a review, stating things like "Disadvantages: Short story, same old multiplayer, aged engine. Advantages: Action packed over-scripting, If you liked others in this series, you'll probably like this too, dated robust engine." Then score a game like that 60-85/100, depending on its entertainment.
(Not really a) Problem is they only do PC...

Aside from that, yeah - reviews are awfully generous and biased and if you bend the definition of fixing you'll find they fall right into it.

 

[Jack Rascal]

I read reviews but I don't care about the number. I consider the score to be the reviewers perception and opinion. It does not affect whether I'll be buying the game or not, that's the reviews job. I want to see pros and cons, even though it's the reviewers pros and cons. I can still get an idea of the game.

If you find yourself a reviewer who likes the same types of games as you, and you agree with him on almost every game, then the number can mean something. I have yet to find that reviewer... But it would probably be better if the score system was not in use at all.

I do find it funny how passionate gamers get if their game is given a low rating. I remember the comments on Edge after they gave 6/10 for FF13. I agreed with the reviewer, though I would have given the game much lower score. And apparently by older standards the game was above average, not ruthlessly low.

 

[Rawne1980]

Scoring is all down to opinion.

A reviewers opinion does not always match our own.

For example, MW3 has a 9.0 review score some places however a friend of mine played it and has taken it back to be exchanged he said it's utter shit.

Another friend played it and loved it (still can't get him to answer his phone so either he really does like the game or he's dead)

Point being, one persons 9.0 is another persons 0.

The point scoring system was always going to be shit, from when I picked up my first game in 1985 to now I rarely ever agree with the score given to a game.

As such I stopped paying attention to gaming scores some time in the mid 90's.

 

[StorytellingIsAMust]

The problem is that numbers have absolute values whereas a game's value is relative based on the person playing. There is no possible correlation that is accurate.

 

[Baresark]

Actually, systems such as reviews on scale of 100 (they are all scales of 100 basically, even on a 1-10 scale because they use to the tenths decimal place) tend to be totally subjective. The metric is, in and of itself, flawed. Leonard Mlodinow explains it beautifully in the book, The Drunkard's Walk. Essentially, people have tried to create a system such as this for grading papers for school. This one fella' took months drilling strict criteria for his out of 100 scale paper grading system. He trained 25 people. After the training, he gave them all the same stack of 25 papers to grade. When he looked at the results of his work, peoples grades varied by as much as 12 points on the same paper. He also tells the story of how two of his son's friends turned in identical papers, accidentally. Instead of getting caught, one received a 90 while the other received a 79. For the number system to mean anything, there would have to be a palpable difference from one point to the next, but there isn't.

The margin for error is huge. I would invite you to check out metacritic (I know, I hate it too, but it shows my point beautifully). In the official reviews column on a game that has seen international release, there tends to be a huge difference in scores out of 100. One game (it's been a while so I don't remember the game) had a top score of 71, while the lowest score on the list for the game was a 10.

Also, I know no one is a fan of a by the numbers system. But that exists (usually as a supplement) to the worded review. It's meant to exist as quick glance. Only, I find that lots of times the number never matches up with what is written. For instance, I watched a MW3 review on IGN. From the beginning to the end the results were vastly different. In the final notes the reviewer mentioned "a bunch of new MP modes" and gave it a great review based on that. But in the review, he only mentioned a single new MP mode, and even went as far as saying it was the only new mode as compared to previous iterations of the game. I don't care one way or the other about MW3, but it was interesting to see that. He also goes on to tell about the lack of emotional impact the big scenes have, even though they clearly should have more. He mentions how the graphics are good for a dated game engine as well, though they are still dated. Me mentions how annoying the SP is with the sections with endless amounts of AI combatants to kill. All of this, and the game somehow still warranted a 9.5.

It's fascinating to watch such a flawed system in play. Also, I wasn't picking on MW3. It was simply the last review I read.

 

[Bostur]

I only have to look at my newspaper to notice that reviewing in general is very much alive and working as it should. And they even review games sometimes. I don't always agree with the reviewers, but since they argue why they like or dislike something I have chance to make up my own opinion.
Scores are not an accurate or fair way to assess a medium, but they act as a quick guidance in combination with a written review.

So theres nothing wrong with the concept of game reviews in general, however some gaming magazines are simply not very good at reviewing. Sometimes this may be caused by poor journalism, and sometimes this may be caused by reviewers lacking integrity. The cause is not that reviews as a concept is broken.

In yesterday's Mailbox video TotalBiscuit raised the point that reviewing as a number based system had changed since he was a wee laddy. He stated that back in his day, 5/10 was average, 1/10 was bad and 10/10 was almost unreachable. Contrasted, he said that in today's world "anything below 8 is terrible, 8 is average, 9 is good, 9.5 is really good and anything between 9.5 and 10 is excellent, and 11/10 is amazing, and there is no score for perfect because that game doesn't exist and they've already used one more number than they should.

I guess TB read different reviews than me when he was a wee laddie. In 1990 I remember that a score below 7/10 was unplayable, 7/10 was bad, 8/10 average, 9/10 good and 10/10 was excellent or the reviewer was paid off. I find that scores nowadays actually use a bit more of the scale and have some extra granularity. I think metacritic actually helps on this keeping professional reviewers in check. Yes metacritic is a great thing when used right.

 

[austin9993]

The thing I think about all of this is: Games are vastly different. The game Modern Warfare 3, for example, may get an 8/10, same as the game Dragon Age, but those two games are extremely different. Someone may not like Dragon Age as much as MW3.

I would prefer a system that instead of uniformly groups games together for in reviewer's personal worth, but a system that recommended a game for gamers that liked more similar games.

Also, I don't really listen to reviewers anyway besides just out of curiosity of what they have to say. I'm a very accepting gamer and can put up with most games and their flaws that get crucified by reviewers.

 

[Vivi22]

Who do you think is to blame for the score 'inflation' that we see? Gamers, desperately crying out for validation that their chosen game is AMAZING? Reviewers, hoping to drive up traffic either via nerd-baiting or simply by giving a positive review? Developers/Publishers in an attempt to get their game more positive press in the hopes that they sell more copies? All of the above?

I think most of this shift to higher review scores boils down to publishers gaming reviews, and/or outright buying review scores with ad revenue to be honest. It's happened before, we know it goes on. At the very least there are companies who will give the game to reviewers that they think will treat it favourably before anyone else gets their hands on it, and when you start tying how much ad revenue you give a site, magazine, or show with scores, or you decide whether or not you're going to give them an early review copy (and therefore let them get more traffic by putting a review up before/at launch), you remove at least some of the ability most of these people have, whether it's the reviewers themselves or their editors, to actually remain objective and keep a critical eye.

So what have we had happen? There's been an increasing creep upwards in review scores, to the point that these days most gamers accept that the score is meaningless and you have to read the whole review to actually get a sense of whether the game is worth your time.

I can't really say that the actual gamers are to blame, because if the goal was to increase traffic, then surely ripping apart popular titles would do a better job than giving them undeservedly high scores. I'd say Yahtzee kind of proves this point as well, especially when he admits it's not that funny to hear him praising a game.

Do you think this should change? Why? And if you do think it would change, is there any way that we as gamers can make that happen?

Yes, I do think it should change because as it is now I think a lot of people simply see the current state of game journalism with regards to reviews as little more than a complete joke. The fact that most people are aware that anything below a 7 is considered terrible is proof of this. But I'm not sure that there's any way that gamers can make it happen beyond simply providing traffic for those who abandon this practice, and not bothering with the sites that don't. But since gamers are notoriously bad about actually voting with their wallet (or in this case their mouse clicks), I'm not sure anything will happen until scores eventually get so inflated that nobody can mention the phrase game review while keeping a straight face, and the journalists themselves finally decide that their loss of integrity isn't worth it. It'd be nice to see publishers taking some responsibility for bad reviews as well instead of punishing the reviewers, but I'm really not holding my breath on that one.

To answer my own questions: I think he may have a slightly romanticised version of the olden days. At 27, he's not that much older than I am and I highly doubt that gaming journalism changed so much in the intervening time between his consumption of it and mine. I will grant that perhaps my consumption was in the middle of when things started to change though. I'd be interested for input from older Escapists about this.

I wasn't aware he was 27, but as a 26 year old I can at least say that yes, his experience matches mine. I started reading various game magazines such as GamePro and EGM when I was maybe 8 or 9 years old more or less. And I clearly remember a lot of instances where games got scores that you will almost never see these days. 4's and 5's were fairly common. I've seen more than one game score 1's, not to even mention 2's and 3's.

And this wasn't just limited to the worst of the worst. Even games that are well loved in their genre from companies that were hugely popular at the time scored quite low sometimes. I clearly remember Xenogears for example getting what Metacritic is confirming for me was a 6/10 from GamePro.

I'll grant this is entirely anecdotal. It'd be interesting to know if anyone ever did an analysis of game scores by year to prove whether or not their was some inflation happening once and for all.

Hell, I'd probably be willing to do it myself if I had astronomical amounts of spare time.

But frankly, I'd rather get away from scores altogether. They hold little meaning frankly. For those interested in a quick overview of whether to buy a game I think a short, point form listing of the pros and cons the reviewer had is far more informative and relevant. But that doesn't exactly make for a great quote to plaster on the front of the box.