Poll: Critics, how can they get it so wrong (not on all occasions but)?

[IKWerewolf]

Now before we go any further I believe that the Escapist does a good job as they do not rely on Scores alone and are critical allowing players to make their own decisions (they don'treat us as 4 year olds)... although I could be an escapist fanboy and have rose tinted glasses.

There are times when the critics are at odds with the public, the most recent example being SimCity.

Just to quote metacritic (not a reliable source but...) the Critics 64% average compared to User's Score of 19% (and the majority of Most helpful being in the Red score band)

Now the obvious ones spring to mind after the Alien's Colonial Marines fiasco and the Kane N Lynch stupidity where there was an intention to Deceive and Intimidate respectively.

There are also two potential other reasons. Firstly they could be too close to the development of the game, they see elements in their best form, there's no active DRM on the games when they get their review they are tainted by what they have seen before so even if the DRM causes problems they can still technically review it.

Another problem is that they don't necessarily see what is game breaking as what we see as game breaking suggesting Critics out of touch with an audience. This is imperitve as the DRM effectively breaks the game so in thoery if they agree with the audience; EA will be banning a lot of Critics from a lot of parties but they don't clearly with higher ratings.

I'm just curious to know what everyone else thinks to why Critics get it wrong and (just as important) does it shake the faith in any of the parties involved (exclude EA in this case as we all know the general opinion of EA)?














Escapist Rules.

 

[IKWerewolf]

PS I couldn't correct the poll, now what I'm about to say in reality is probably non-existent (maybe rare) but its perception that counts not the facts. In my opinion this is completely wrong but you have to put all options in.

The second to last should read "they don't care" and is based on the perception that because they are being paid then to them its just a job so they don't really care.

 

[Delerien]

There are times when the critics are at odds with the public, the most recent example being SimCity.

Just to quote metacritic (not a reliable source but...) the Critics 64% average compared to User's Score of 19% (and the majority of Most helpful being in the Red score band)

In the case of SimCity many critics ignored the DRM fiasco and without that SimCity would indeed deserve more than 19%.

Many Critics probably also focus too much on the mechanical quality of the game because it is most easily measured. So games with great graphics/sound etc. often get more praise from critics than from the general public.
Being somewhat dependent on Publishers and Developers probably doesn't help stay fair, but i have enough trust in most people to believe that this isn't the main reason for most discrepancies. In the end if i disagree with a critic it's often because we just value different things in a game or have a different approach to the game.

 

[SSJBlastoise]

@TheKasp is pretty much spot on. People judge by their own experiences. Most of the time I would like to think (yes I am kind of naive) that critics try to rate it more on the actual game play and story. Of course this isn't the case with some sites but I usually take my reviews from Good Game (and Australia game review show) and I feel they judge games fairly accurately and they gave SimCity a 7 out of 10 but also mentioned that the game was going through some struggles with connection issues and whatnot. They rated the game on what they actually played (when they could) rather than just try say it's crap because it had a rough launch.

 

[tippy2k2]

Because gamers are morons who don't know how to judge for shit.

- ME3, even despite its all problems is not a 0/10.
- SimCity, despite its all problems is not a 0/10.

Basically, the audience overreacts. Since the reviewbombing is a thing now and people tend to bomb games for the most stupid reasons (oh no, Portal 2 had some minor cosmetics at launch, it clearly deserved to be reviewbombed! I should not have listened to the critics when I bought it but to the morons who wrote metacritic user reviews) and they are not capable of putting the bad things in relation to the good bits and judge based on that.

In the end, I would take even the Dorito guys 'opinion' of a game more serious than the sum of 1000 metacritic users (or gamers).

I had a nice, long, rambling and a slightly-drenched in Haterade filled rant about why Metacritic's system is completely broken and I consider users as a trusted review source as much as a monkey that banged it's hands against the keyboard to make words come out. You just put it in a much better way

So I will throw out the second part to my opinion, which combined with your post, I feel accurately describes the situation.

People forget that game reviewers are people with opinions. That's ALL a review is; a professional opinion. They are professionals at it so they (should at least) have the ability to recognize what's good but not their cup of tea (or what's bad but they recognize that they have a super-crush on a bad game). A good reviewer should be able to explain why their game got the score that it got but there will always be drops and spikes in their reviews based on their own personal opinion.

 

[Thoric485]

Looking at the huge score-oriented magazines with not a single distinctive writer in them, the whole thing is just a hype mill. Being pleasant, letting everyone with the right friends go through and pinning them a gold star. And as a result on Metacritic green scores for movies start at 60, while scores below 50 for games are red.

Everything high-profile is cramped in 3 grades of a 10-grade scale. How in the hell is a studio supposed to defend years of development and alienating the mass audience with a complex and original product, when the so coveted critical success consists of a 0.5 score difference to some bi-yearly released crock of shit that pushes millions to 13-year-olds?

So of course when a publisher gets pitched something in a niche genre, or dangerously innovative, or simply with a female protagonist, the studio will be denied, because even on the off-chance they produce something original and memorable, there's not enough value in critical success to make up for losing the masses. Journalists and critics are supposed to create that value, but they haven't.

 

[Rickin10]

Because gamers are morons who don't know how to judge for shit.

- ME3, even despite its all problems is not a 0/10.
- SimCity, despite its all problems is not a 0/10.

Basically, the audience overreacts. Since the reviewbombing is a thing now and people tend to bomb games for the most stupid reasons (oh no, Portal 2 had some minor cosmetics at launch, it clearly deserved to be reviewbombed! I should not have listened to the critics when I bought it but to the morons who wrote metacritic user reviews) and they are not capable of putting the bad things in relation to the good bits and judge based on that.

In the end, I would take even the Dorito guys 'opinion' of a game more serious than the sum of 1000 metacritic users (or gamers).

Whilst I agree with much of what you say, you forget the other side: the equally retarded fanboy 'reviews' that amount to 'OMG!!!! amazin cooooooooooolllll GOTY 10/10!!!!! I read one 'review' from a guy who described some FPS (I forget which one) as the greatest game he'd ever played. The reason? 'the grafics are amazin!'

Metacritic is a great source of info IF you only read the reviews rating between say 4-8. Even if you don't necessarily agree, You're generally getting some reasoned views.

 

[sethisjimmy]

I'd say being out of touch with the audience. Ignoring giant reviewers who could possibly be pressured by publishers, the biggest thing I see is not understanding the typical gamer.

Reviewers have entirely different play styles from regular gamers - generally they're trying to plow their way through a game to see all they can before the review. A lot of them don't even have the free time to play the games they want to, and end up not being able to properly compare games because of this. Plus they don't have time to let a game really "sink in". Personally I don't really know my true opinion on a game until I've had some time to mull it over even after playing. Reviewers typically have to get the review out quickly, and thus have to review only the very obvious features and mechanics.

Some of the blame rests on the gamers themselves though too, and even the pressure they put on certain reviewers. I mean if I want to piss myself off I could just go and look at the comment section of pretty much any IGN review. Say a great game gets a 9.5. Guaranteed in the comments you'll find a large number of people bitching that it didn't get a 10, whereas all the rest will be complaining they scored it too high. Both sides will complain that IGN is corrupt because of this score. This is what is known as "completely fucking nonconstructive feedback".

The concept of scores hurts the overall reception of the review as well. I feel like way too many people just scroll past the writing portion of the review so they can cry about the score. That and the fact that review scores have become so inflated that anything below an 8 is considered not worth playing.

In fact, I'd argue that the large majority of the disconnect between reviewers and gamers is at the fault of the gamer. Gamers need to learn to either find reviewers they generally match opinions with, or use other methods of determining a game's quality entirely. It's the standard "don't like it don't watch it" cop-out, but it's pretty much true. Reviews simply aren't the be-all-end-all of quality determination.
Also, gamers tend to be pretty unprofessional with their reviews. Take Metacritic for example. Where reviewers are able to be professional, and see that while a game may have a major flaw, it still has redeeming qualities. Metacritic fanboys however, are not able to do this.

 

[CannibalCorpses]

The problem with the reviewing system is all the hype. You get shown snippets of information and your imagination fills in all the blanks. When the game arrives and you get to play it you have already decided what it is that you want from it and when things are slightly different your hopes are dashed and your opinion is jaded as a result.

Then, for no good reason people rush out and tell the world about it, trying to outdo each other on how bad they experienced it. Can't be outdone by some spotty 14 year old with an advanced sense of disgust...it wasn't just bad, it felt like rape to play it. No! It was like watching a baby being raped. No! It was like a baby raping another baby etc.

The chain goes on and on and then for some really strange reason, other people read it and base their opinion on it. It's staggering how often i play a less than AAA game and mention it to my friends and without having seen or heard anything but a score review from some fucktard they tell me it is shit. A review from someone who has played maybe 50 games in their lifetime and they ignore my 1000+ game experience. No amount of explaining can diminish their first impression.

So, in conclusion i guess critics get it wrong so often because they aren't really critics at all but overly expressive morons who want the world to listen to them and get their own way because people are basically sheep and lack the intelligence to do much more than parrot the words of others like in some way that validates anything. Thankfully i have my own mind and a distinct lack of trust in the opinions of others and so i make my own mind up and generally find good in the bad games im told of and find bad where i have been told there is none.

3 cheers for self determination

 

[DoPo]

I'll go a step beyond blaming gamers or publishers or reviewers, etc. OK, they aren't entirely not to blame, but there is something else at work - the system for ratings is...not good. OK, actually it's not even that, but the concept of it eludes people and that is not entirely their fault. Heck, the whole thing isn't terribly well defined, either.

So, let's take a step back and look at it - a X out of 10 (and X out of 100 is pretty much the same with some fine graining) - what does it mean? First of all, it's an ordinal scale, so it measures stuff and the answers do have an order towards them - some would be better, some worse, or maybe they'll just lean to two different ends of a spectrum. At any rate, you can use them to measure stuff. However, and here is where errors creep in - you've seen that many, many times - on a survey you'll most probably find something answers like "Agree", "Neutral", and "Disagree" (or throw in strongly agree/disagree, too). You can also encode the answers as letters or numbers - whatever[footnote]random fact of the day the agree-disagree stuff is called a Likert scale. It's a type of an ordinal scale.[/footnote] but the bottom line is you have a set of answers. You can see how many of them are, say, positive or not but you cannot take the average of them. If 50 people voted they disagree with something and 100 voted they strongly agree you can take the average (assign the answers values from 1 to 5 starting with "strongly disagree", for example) and say that overall people agree. Yeah, that is sort of true. More people agree than not, for sure. However, let's take another example - you have 100 votes for "strongly agree" but also 100 for "strongly disagree". Now the average response is "neutral". But looking at the data, it's anything BUT neutral.

So we hit the first problem with the scale - you cannot average it meaningfully. And this stems from the following reason - an ordinal scale is not proportional. The distance between two answers is not the same. Or let's call it the weight - the weight of answers is not the same. A "strongly agree" answer is not twice the "agree", not every time.


Right, boring stuff aside, how does this relate to the ratings. Simples, the scale is not used properly. As was explained, you don't really average ratings like that. But also, there are no standards for ratings - one person can rate it based on personal opinion - it sucks, thus it gets a 1, it's great, thus it gets a 10, for example. But others base it on merit - reviewers certainly do that - they may not like it but chances are, they'll give it some points for the enjoyable parts. This creates a divide between opinions - the opinions themselves are not equal - "I don't like it - 1" is not the same as "The game BARELY FUNCTIONS AS A GAME AT ALL - 1". And then there is the difference in opinions, too - what is "I don't like it" may be a 1 or a 5 across different people.

So...that means that ratings are not fit for averaging on two accounts - the scale doesn't lend itself to such a manipulation but also even if it did, the data collected is not consistent. But there is more. It's more minor...so it's not a fundamental reason for the ratings to not be useful but...actually it is, but it's less fundamental than the two fundamental ones: people's perception on scores. Taken just by itself, just a 1-10 (or 1-5 - whatever) scale would have a problem - people tend to avoid extremes. Yeah, it's actually a thing - people are less likely to give an extreme answer - so the absolute minimum or the absolute maximum. I'm not sure on the psychology behind it (as in, how it's called) but it's present. But there are exceptions - review bombings. People just swooping in and tipping the scales by rating the lowest possible. It's easy to do, because a lot of the other responses would hover around the average, so there is nothing to redeem the rating. A single 1 needs many a 7s or 8s to be redeemed into a more or less coherent average rating.

TL;DR version - few paragraphs of mostly boring stufff explaining why the scale is fundamentally flawed on few accounts - the scale itself is inconsistent, the ratings are also inconsistent, and it's easy to break, which people do.

 

[Phlakes]

Because gamers are morons who don't know how to judge for shit.

- ME3, even despite its all problems is not a 0/10.
- SimCity, despite its all problems is not a 0/10.

Basically, the audience overreacts. Since the reviewbombing is a thing now and people tend to bomb games for the most stupid reasons (oh no, Portal 2 had some minor cosmetics at launch, it clearly deserved to be reviewbombed! I should not have listened to the critics when I bought it but to the morons who wrote metacritic user reviews) and they are not capable of putting the bad things in relation to the good bits and judge based on that.

In the end, I would take even the Dorito guys 'opinion' of a game more serious than the sum of 1000 metacritic users (or gamers).

/thread.

The divide between critic and user scores is there because the critics' job is to be fair and objective (at least as much as possible). They don't hand out zeros to every game that pisses them off because that would be irresponsible and frankly pretty dumb. The public doesn't have to worry about that so they don't bother to keep their bias in check, and you end up with a bunch of people zero-bombing games that at least have some redeeming qualities.

Basically it's the user scores that are wrong. And that really shouldn't be a surprise.

EDIT: To clarify-

-Critic scores are good for measuring actual quality
-User scores are good for measuring public opinion

And those can be two very different things.

 

[Rariow]

Because gamers are morons who don't know how to judge for shit.

- ME3, even despite its all problems is not a 0/10.
- SimCity, despite its all problems is not a 0/10.

This, this, and all of this. It's fun to ***** on the internet, it really is. I spend most of the free time I have that I don't spend gaming bitching about gaming on the internet. We just love do it way too much.

Besides, gamers tend to "follow suit" with each other. I think that had SimCity not had its always-online DRM, very few people would've complained about the actual gameplay itself. But SimCity DOES have always-online DRM. Now it's trendy to complain about it, so we'll tear the aspects of the gameplay that aren't up to snuff apart. Same with any BioWare game after EA bought them, any re-make of an old beloved franchise, and anything with a bit of history behind it. The few cases where the gap between critics and players is as gargantuan as the examples cited above are usually games that belong to old beloved franchises or had some sort of dodgy policy behind them, simply because some people like to go on metacritic and give games zeroes when they clearly don't deserve them.

I'm just going to say: No game I've played in the last ten years deserved a zero, and I've played some horrible, horrible games. No game has no redeeming factors at all behind it.

 

[Auron]

Review bombing on metacritic is not a good source of critical acclaim among gamers, there's a dedicated populace of trolls who haven't really even played the game but go there just to give a game a 0. It happens with lots of games especially ones with EA's name on it. A vocal part(minority or not I don't really know.) of current gamer population seems to hate very very easily and that's a serious problem. If every review started to follow the same trend I'd stop giving them any credit whatsoever, In fact in the last years when I was in doubt I relied much more on say, serious let's plays than simple reviews which gives me a broader understanding of the game and allows me a much more informed purchase. Though I do often like and agree with the Escapist's reviewers it doesn't fly for a large margin of them.

And lastly I'd like a reviewer to not rail a game based on DRM I might not have a problem with or technical issues that are going to be solved*, it's nonsensical I want them to judge the game content. I can find out if it was fixed or not in the future, if all I hear about the game during a review is "it sucks for no reason other than drm I disagree with." it's not really being critical of the content.

*Maxis and Blizzard(who also had always online troubles though very reduced in scale.) have made a big deal of assuring everyone it was going to be fixed asap and it actually was so it's different from a bug ridden game that might not see a patch.

 

[Pebkio]

...because they're people? With their own opinions? And tastes? And continued desire to get early releases from developers by avoiding the reputation that they're honest about games they dislike? No! Bad Pebkio! End sentence being anew!

I voted "other", because aside from the lip-wrapping ones who never seem to have anything bad to say about any game, most reviewers really do have their own tastes and opinions and obsessions. Reviews are, technically, op-ed pieces. What you have to do is find yourself one or two reviewers that you find yourself agreeing with a good 85% of the time. Sometimes they'll let you down by liking a game you don't, because they aren't you. When that happens, don't post stupid, pretty, comments like "you've lost all my respect because you like a game that I didn't".Thpppppt! I mean, keep score when that happens; keep a tally to make sure that your favored reviewers are still closely, at least somewhat, aligned with how you think.

 

[CommanderL]

game reviewers from game websites tend to have a weird rating system and review higher i.e call of duty 72 getting a ten/9 go to youtube people like angryjoes and Jeremy Jahns' tend to be more fair and balenced

 

[blackdwarf]

Comparing the ratings on Metacritic is not a good indication of the comparison between gamer and reviewers. user reviews are mostly people who hate the publisher/developer/franchise for something unrelated to actual quality of the game itself. I can post a user review on a game I never played, but my score is still thrown in the equation for the overall user score. Most people also think pretty black and white concerning their opinions, if you don't enjoy it, then it must be bad game, so you give it a 0/10. I dislike Fifa, but I still consider that a quality product. And the biggest issue with meta critic is that people have a different idea about which ratings they should give to a certain degree of quality. Some say that "above average" should be given an 8/10, while would give a game like that a 6/10. That is why the ratings on Metacritic are mostly meaningless.

And in the case of Simcity, the Reviewers played at a time where the were few players on the servers, meaning they had a different experience than most players after launch. So it is not weird that they give the game a good rating, because it worked for them.