Are major game reviewers morally bankrupt at this point? (Wall of Text Warning) (Updated)

[Nemusus]

I tend to stay away from review scores and make my game purchasing decisions based on a couple of factors: the actual meat of a review (they tend to give me a general picture of the game, its strengths and weaknesses (somewhat)), Zero Punctuation (if he puts a vid out) since Yahtzee, although with a slight tendency for exaggeration, generally gives a fair assessment of the games he discusses. Finally, I follow the user feedback, although I give them a bit less weight, what with the community's tendency for bombing games they don't like. For me to seriously consider lending weight to a community opinion, it needs to be repeated several times. Don't like Total War: Rome II? Think it's too buggy? If I see the same bug crop up more than five or six times, then I'll take it as an issue. If it's something vague (e.g. "The game bugged out on me, I can't play"), then I tend to disregard the comment. Yeah, the reviewers do tend to be wrong sometimes, but then, they are human. If you stay away from overtly dodgy ones (IGN, Metacritic(simply because they tend to be skewed by user rage)) and stick to more reliable ones (hi there, Escapist!) you should be fine. But yeah, I wouldn't call game reviewers morally bankrupt, more...unreliable.

 

[MysticSlayer]

My overall experience with game reviewers is that they tend to be aware of the problems in a game, but they also tend to have a different idea of what those problems fully mean when held up to the game as a whole, which leads to a sometimes massive disparity between a professional review and a user, or even two professionals. In general, if I'm going to be watching/reading reviews, I'll get a few people's opinion and also maybe check out a few of the users on Metacritic. In some cases, I'll just wait for opportunities to try out the game in question, or talk to someone I know who has similar tastes as me, as the game is never going to get a fair reviewing from anyone (this is the case with Call of Duty).

Also, keep in mind, professional reviewers tend to be a little more level-headed than a lot of gamers. Granted, this level-headedness sometimes leads to them defending something that shouldn't be defended (ex. IGN defending the lack of diversity in ME3 endings), but it also prevents them from excessively overblowing every issue they have with the game, which is a problem that plagues user reviewers. They also don't get involved with the "RAWR! My opinion is right! Ignore anyone who disagrees with me!" ranting that many users get involved in.

Overall, I respect professional reviewers, but I don't take everything they say as gospel. I normally follow reviews from a few critics, some "less professional" ones (ex. AngryJoe, ZeitGeist Reviews, etc.), and I might check out a few user reviews. Normally, though, simply playing a demo, watching a gameplay video, and/or reading up on a few features is enough for me to gauge whether or not I'll enjoy the game, and the reviews are just there to catch any extra strengths or weaknesses.

It's probably this. I don't really follow most "professional" game reviews, so my opinion probably isn't valid, but I do know that users have a tendency to overreact to the silliest bullshit. Remember that whole controversy about the new Dante design for DmC: Devil May Cry? The average Metacritic user score for it is 4.7 (looking at the PS3 version) even though the game is quite playable and basically alright.

A 4.7 is actually an appropriate score for a game thats "playable and pretty much alright." A game that is just passable and mediocre should be about a five. Thats the problem with pretty much every video game review, they dont use the lower three-quarters of the scale so all of their reviews become meaningless. When every single game falls between 7 - 10 out of 10 you know something has gone wrong.

The problem with this is that it requires a fundamental shift in the way we view grades. When someone receives a 50% on a test in school, they aren't thinking "Well, I did average." No, they are thinking "I am so screwed right now!" Pretty much everything below 70% is a sign that you need serious improvement, with the exact score only indicating how much improvement there needs to be. If you want people to accept 50% as average, then you need to change their thinking on what those grades mean, but that would be very hard considering most people grow up viewing a grading scale as only being acceptable if their grade falls within the 70-100% range. Even in college you won't be able to move on to the next class if your grade is below 70% at the end.

Personally, I think the 1-5 star review score is the most effective. It's easier to use the full spectrum without getting redundant (seriously, there's very little difference between a 1/10 and a 5/10), and the middle grade (3/5) doesn't look as horrible, even if, taken as a percentage, it comes out to 60%. Not to mention, it distances us from viewing video game grades as the same as school grades. Of course, this runs into issues with Metacritic standardization and its way of tracking scores (doing so on a 0-100 scale), but even trying to use 5/10 as the standard of an average game would go against Metacritic's standards.

 

[bug_of_war]

I just watched Angry Joe

Joe is a good example of where the community of gamers are at. Very emotional, and prone to unwarranted outbursts of praise/hate for a game. His review is therefor heavily based on personal enjoyment rather than "Does the game play well", and that's fine, but you need to take his review (and well any review) with a grain of salt.

For the most part you get more out of what someone has written or talked about a game rather than just a number, if you have a bunch of different reviews and take away the score, you'll be able to form a consensus on what the games highs and lows are.

So yeah, I don't think big reviewers are immoral, I just think people have their own opinions and you've gotta figure out whom's opinion you agree with more.

 

[In Search of Username]

I think it's a mixture of both reviewers not wanting to give any major title a negative review, and consumers completely overreacting. Like in this case I'd probably give the game like 5.5. It's pretty much a mess, but there are some improvements from the last one and the problems aren't nearly as widespread as some would have you believe and most of the actual bugs will probably be fixed soon enough knowing CA.

The same thing happened with Mass Effect 3: people hated the ending; reviewers ignored that factor when giving their review and thus ended up scoring it highly; fans took absolutely nothing else into account when giving their review and thus ended up scoring it ludicrously lowly. In reality it was a pretty good game with a pretty disappointing ending.

Generally you'll want to take the average of the critic reviews and the user reviews to get something closer to a reasonable viewpoint.

 

[jab136]

yep [http://www.metacritic.com/game/xbox-360/mass-effect-3] remember all those perfect scores EA kept touting during the shitstorm surrounding ME3's ending. I honestly can not believe that money didn't change hands for that game to get all those perfect scores.

 

[In Search of Username]

The problem with this is that it requires a fundamental shift in the way we view grades. When someone receives a 50% on a test in school, they aren't thinking "Well, I did average." No, they are thinking "I am so screwed right now!" Pretty much everything below 70% is a sign that you need serious improvement, with the exact score only indicating how much improvement there needs to be. If you want people to accept 50% as average, then you need to change their thinking on what those grades mean, but that would be very hard considering most people grow up viewing a grading scale as only being acceptable if their grade falls within the 70-100% range. Even in college you won't be able to move on to the next class if your grade is below 70% at the end.

That's not a universal thing man. At my university you have to get below 40% to fail, and anything above 70 is the highest possible grade. I got 85% once and was fucking ecstatic. So it might be like that in most cases but don't assume it's universal.

 

[Not Lord Atkin]

Still, I doubt this has anything to do with people being paid off or being morally bankrupt or anything like that.

Understand that by morally bankrupt, I do not mean bought and paid for by a developer company, as I would lack any evidence to prove that.

However, it is a well known practice [http://arstechnica.com/gaming/2011/06/duke-nukems-pr-threatens-to-punish-sites-that-run-negative-reviews/] that a reviewer can be blacklisted by production companies for negative reviews. I doubt IGN has to deal with this problem as they are too large to ignore, but I also doubt everyone is as fortunate.

So please understand, when I say morally bankrupt, I mean not invested in the truth. Which is morally bankrupt.

I'll share my experience on this: I used to work for a small local website that focused on PS3 games. We had deals with all major distributors and they would send us review copies of their games. So far so standard.

The main issue we had was the owner and editor of the site. First of all, he also owned an eshop from which he sold his games so he ended up using the website as more of an advertising joint than anything. This reflected in the demands he put on us, reviewers. For instance, when he sent us review copies, he would also tell us the minimum score he wanted us to give the game. Yeah. Let that thought go through your head. I'll wait.

There were two major things hat made him behave like this. A> he was afraid that if we gave the games low scores, they wouldn't sell on his eshop. B> he was terrified of the idea of publishers blacklisting him. see, the thing about this entire 'blacklist' boogeyman is, that publishes generally avoid doing it; especially if the reasons are as petty as a low review score on one game - it's not good business. And there's nothing stopping the reviewer from buying the game with their own money and then tearing it another asshole anyway. But he was unable to process that, no matter how much you'd explain it to him. It got quite extreme at times.

This completely absurd stance of his would show time and time again. He would call me up telling me that he was sending me a copy of God of War HD collection vol. 2 because no other reviewers were available and that he expected a score higher than 7.5/10. He would make sure that a colleague's review of Resident Evil 6 would go from 4/10 to 6/10 in between the first draft and the published review; obsessing about the score so much that he kind of forgot to proofread the rest of the review (which was kind of his job). But the worst experience I've had with the guy, the one that made me quit working for him, was my Ghost Recon: Future Soldier review. I gave it an 8/10. I acknowledged it as a great, refreshingly tactical and mechanically rock-solid game, but with a lot of issues that needed to be addressed, from the terrible, TERRIBLE story that didn't even have the common courtesy to not shove itself down the player's throat, the schizophrenic, homogenised mess of a presentation and big dramatic setpieces that didn't make any sense and interfered with the core gameplay. That shitstorm wouldn't calm down for weeks. I stayed up well past midnight that Saturday, finishing up, proofreading and publishing the review... then on the Sunday morning at 8, he would wake me up with a call, just to let me know that some of the users didn't agree with the review in the comments. He would then proceed to keep nagging about it non-stop for the next few weeks. Turns out he really liked the game and expected a 10/10.

The moral of the story? Blame the editor. Always blame the editor.

I'm kidding.

The thing is, there are always loads of factors affecting the review scores and causing the disparity between them and user reviews. You can't single out one cause because there's going to be a different one with every single review. Maybe the reviewer liked the game enough to be able to look past the flaws. Maybe the fans are just throwing a hissie fit because the game isn't enough like the previous installments. maybe the reviewer is being pressured into giving the game a high score or maybe they are, as you said, morally bankrupt. Or maybe it's something completely different. You can never really know for sure.

 

[Blaster395]

Angry Joe gave it a 6/10 and now with the second patch (which is a huge improvement) perhaps a 7/10. He is closer to the review scores than the user scores.

I believe the problem isn't the review scores. It lays with the user scores. The vast majority of people don't waste time going to metacritic to vote a 9/10 or 7/10 or any reasonable score. Those that do go and vote, a very small proportion of those who play the game, will usually either give it a 0/10 or 10/10. Even if the number of people who think it's worth 0/10 or 10/10 is equal, people who think that it deserves 0/10 are more likely to vote on metacritic because the other people are too busy playing the game.

Basically, it's a downvote brigade. If you check this: http://www.metacritic.com/game/pc/total-war-rome-ii/user-reviews you will see that about half the reviews simply say "0". That's a score that realistically should only be given to things like Big Rigs over the road racing. I think we can all establish that anyone who gives this game a 0 or 1 /10 just doesn't give a shit about giving reasonable scores and is simply brigading.

Furthermore metacritic doesn't even require the person voting to have played the game. All it takes is an article on a major website describing the launch as flawed and people will set sail in their 0/10 brigades.

I decided to do a chart of one page of review scores sorted by date.

This shows a clear gap in the middle, starting after 3 and ending at 9, where nobody gave it those scores. The absurd number of 0 and 1 scores indicates downvote brigades, and the clear peaks at both 0 and 10 proves my point about how the vast majority will only pick the highest or lowest score. The near complete lack of average scores (of the likes that Angry Joe would give) probably is indicative of a lack of analysis.

If the voting was done in a sensible, calculated and analyzed way and the average was 4 it would probably look more like this:

 

[MetalDooley]

It should be said that I do not consider IGN trustworthy though. That's one of those sites I would assume gets the review scores written out before they receive the game they are reviewing. I remember someone getting fired for not writing a positive review of Kane and Lynch 2...

That was Gamespot that fired Jeff Gerstmann for giving the first Kane & Lynch game a poor review

 

[Gearhead mk2]

Reviewers can't really do this for enjoyment, they have to power through a game in the shortest time possible then report on their impressions of it. This naturally leads to inaccuracies and shoddy work, plus some issues that don't really crop up until you give it some thought. Reviews aren't some be-all-end-all, they're just one of many tools that allow consumers to make an informed decision. That said, when IGN is blatantly selling out, Gamespot is firing reviewers for calling a game OK and Famitsu gave FF XIII a perfect score, that's a sign there is definitely something wrong. Nowadays I don't really read "professional" reviews, I just stick to word-of-mouth and TGWTG style comedy reviewers, because no matter what faults and problems they might have, I know that they're at least honest instead of being afraid to score anything lower than 7 for fear of getting fired.

 

[ron1n]

Gaming reviewers aren't 'morally bankrupt' they're just ceasing to be relevant due to the following factors:

1. Review scores are a broken mechanic. They attempt quantify something that simply isn't quantifiable, have no frame of reference or organisation (how can 2 games in two completely different genres, different development costs etc. possibly be compared using the same scoring system?).

They create a dissonance between the reader and the authors real impression. (Do we still trust in a score when the written portion of a review doesn't 'read' the same way or appears to draw a differing conclusion?)

Lastly, there is no proper curve to speak of.

Ideally, 1/10 = Terrible, 2/10 = Very flawed, 3/10 = Very Poor 4/10 = Poor, 5/10 = Average 6/10 = Above Average 7/10 = Good, 8/10 = Very Good, 9/10 = Excellent, 10/10 = Perfect.

INSTEAD, the curve is more along the lines of: 1-6/10 = Shithouse. 7 = Average. 8-10 = Great.


2. Games are not static. Why do we review games in the same way we review movies? More so now than ever, games are constantly changing via updates and dlc. A game that was shit on release might end up blossoming. Likewise lots of games start out strong but are brought undone via terrible decisions on behalf of the developers.

With these changes occurring, often on a weekly basis, how can a 'review' possibly be relevant for longer than a few days? The product has already changed.


3. People already know what they think. Once upon a time, reviewers were our window into upcoming releases. Then the internet happened. And then video streaming.

Now we've reached a point where not only do gamer's have unprecedented access to gameplay footage, open betas, early access releases and leaked footage etc. But games (AAA titles in particular) are more often than not, either highly Iterative, or part of a yearly franchise.

Thus, a lot of people now already know what they like and don't like. Therefore, instead of treating reviews as a source of information, people use them to seek reinforcement of their own pre-established opinions. (Not all people certainly, but enough to create negative shit storms against reviewers when they have a differing opinion)



So what's the solution?

Ideally, it would be great if every single gaming reviewer agreed to do away with scores entirely. Not only would it put the focus back on ideas rather than scores, it would completely cripple the evil behemoth that is Metacritic.

Secondly, I think we need to redefine 'review' as 'first impressions'.

It's too unfair and unrealistic to expect journalists to make structured/in-depth calls on games when they have to plow through countless reviews a year, often with (likely) only a few hours to commit to certain titles.

But a first impression? Yeah sure, why not. I'd much rather see them shoot straight from the hip and focus soley on what they have in front of them, minus any stupid scoring system.

Say what you will about Totalbiscuit, but there's a reason the guy stresses he only gives 'first impressions'. It's because he knows the limited time he has to form an opinion on a game, simply isn't enough to come up with anything so authoritative as a 'review'.

 

[Abomination]

Rome II is a really strange game. Are the reviewers looking at how it WILL be when it gets patched? The problem is we don't know if they'll ever iron out the current problems with the game.

Under all the crap is still a great turn-based "grand (lite)" strategy with a good real-time strategy combat... but there's still a LOT of crap covering it.

Medieval II: Total War has a better "faction politics" system than Rome II and that's just a family tree - but at least you can understand what your option are.

Do I think the reviews of 8 are inaccurate? Hell yes. The game is worth no more than a 6 in its current state and that's just with recent patches. I can't imagine what the game was like for the review copies.

The thing is Metacritic needs to allow for "patched reviews" or something. Empire: Total War at release was horrid but now? With all the changes and mod support available? I would recommend it in a heartbeat.

 

[In Search of Username]

2. Games are not static. Why do we review games in the same way we review movies? More so now than ever, games are constantly changing via updates and dlc. A game that was shit on release might end up blossoming. Likewise lots of games start out strong but are brought undone via terrible decisions on behalf of the developers.

With these changes occurring, often on a weekly basis, how can a 'review' possibly be relevant for longer than a few days? The product has already changed.

Absolutely this. I feel like the recent SimCity was a victim of this; while it wasn't a perfect game anyway, a lot of reviews seemed to score it largely based on the launch issues, which now a few months later are entirely irrelevant to someone wanting to buy the game - but the scores remain.

I think ideally reviewers should mention if there are problems at launch, but always with the disclaimer that this stuff might not be the case in a few months' time at all. That or check in on the game a while after for a 'finalised' review that takes into account all the updates that have been made.

 

[MysticSlayer]

The problem with this is that it requires a fundamental shift in the way we view grades. When someone receives a 50% on a test in school, they aren't thinking "Well, I did average." No, they are thinking "I am so screwed right now!" Pretty much everything below 70% is a sign that you need serious improvement, with the exact score only indicating how much improvement there needs to be. If you want people to accept 50% as average, then you need to change their thinking on what those grades mean, but that would be very hard considering most people grow up viewing a grading scale as only being acceptable if their grade falls within the 70-100% range. Even in college you won't be able to move on to the next class if your grade is below 70% at the end.

That's not a universal thing man. At my university you have to get below 40% to fail, and anything above 70 is the highest possible grade. I got 85% once and was fucking ecstatic. So it might be like that in most cases but don't assume it's universal.

I was speaking general terms, not universal ones. It doesn't have to be universal, though. If the general population has that view, which comes from the "most cases" you've already acknowledged, then it will require a shift in thinking before most people accept a different system in video game grading.

 

[Lieju]

Angry Joe gave it a 6/10 and now with the second patch (which is a huge improvement) perhaps a 7/10.

So, on his scale it's better than average.

Hardly lambasting.

 

[Blaster395]

Angry Joe gave it a 6/10 and now with the second patch (which is a huge improvement) perhaps a 7/10.

So, on his scale it's better than average.

Hardly lambasting.

Just giving an example of a very low review score still being far higher than the user review score on metacritic.

 

[In Search of Username]

I was speaking general terms, not universal ones. It doesn't have to be universal, though. If the general population has that view, which comes from the "most cases" you've already acknowledged, then it will require a shift in thinking before most people accept a different system in video game grading.

Hmm, well it seems to be the case for most people I know (at various different universities). Maybe it's a UK thing? Because the general population here doesn't seem to have that view, it seems to be pretty generally accepted that above 70 is very good. I'm assuming it's different wherever you're from and that's the difference in mentality here.