A Review Scoring System That Would Work

[veloper]

There's a solid scientific basis to subjective numerical scoring systems, what Daniel Kahneman referred to as "Intensity Matching" in his book "Thinking Fast and Slow".

From the book:
"An underlying scale of intensity allows matching across diverse dimensions.
If crimes were colors, murder would be a deeper shade of red than theft.
If crimes were expressed as music, mass murder would be played fortissimo while accumulating unpaid parking tickets would be a faint pianissimo.
And of course you have similar feelings about the intensity of punishments. [...] If you heard two notes, one for the crime and one for the punishment, you would feel a sense of injustice if one tone was much louder than the other".

I see no reason not to intensity match my experiences with numbers (among other things), or allow myself to feel emphatic enough to read another person's numerical intensity matching of a video game and draw my own conclusions.

On a strictly personal note, the scores I give games in Metacritic are almost always close to the cumulative user scores (critic scores are almost always off, for obvious reasons).

Why? Because the human brain kicks ass at intensity matching.

So why limit the tools at our disposal, when we have this ability just waiting to be used with the calculative speed and force of an infinitely creative supercomputer?

That's the most interesting post I've seen over here in a long time. Welcome new guy!

The like-o-meter could be something like intensity matching.

 

[Silent Protagonist]

One thing that I like about Yahtzee's scoring system:

Something like GTA V or Skyrim would be "0% functional" games, because the larger and more complicated a game is, the more bugs it's going to have.

Meanwhile, a cowclicker could easily get a 100% under the same system, since with very little interaction and simple mechanics, it's much easier to make it be bug-free.

Interesting point. Perhaps a bugs/crashes per minute stat would be more helpful than deducting a percentage. Of course that could be flawed in that it may not always reflect the severity or game breaking nature of the bugs. It also could be skewed depending on the play style if, for example, bows had a consistent repeating bug but magic did not, so a reviewer that played as a magic user would have a lower BPM than one that used ranged.

It almost seems like coming up with a absolutely accurate and objective review score may be futile.

 

[StreamerDarkly]

Why do you guys say the author is on to something, as if he invented the idea of assigning scores to multiple facets of the game?

Christ Centered Gamer (of all places) has been doing this for awhile.

 

[Maze1125]

I find this article to be full of terrible logic that attempts to support a preconceived conclusion, which, ironically, is something the author complains about in attempting to explain why review scores are bad.

The worst flaw is probably the idea that aggregate statistics, such as an average score, "merely flatten things out and become meaningless". To address this problem, it is claimed that we really need to see every single data point from some 20 or 25 independent reviewers to derive any meaning. Clearly not the case, as data sets of any size can often be well characterized by just two parameters: mean and variance of the sample.

Actually, the initial claim that 'averages are useless' because things get 'flattened out' is the sort of cringeworthy insight I'd expect from a 2nd year general arts student. Averages are great in certain types of analysis, particularly in the case of video games where the little niggle that would cause one player to dock marks is completely irrelevant to 90% of the gaming population. More precisely stated, every observation (individual review score) in the data set is contaminated by noise (personal preference that doesn't transfer to the majority of other players), and the only way to extract what we might call the signal (more objective assessment of the game's quality) is to compute the average rating. Obviously not everyone will agree that the average score reflects the game's quality, but it is the single number that MOST players will agree with.

To think you can just say "art is subjective" and be done with it is the height of ignorance.

I'll tell you what the height of ignorance is.
Complaining about part of an article when you didn't even bother to read it properly.

That part was about technical issues and is absolutely correct.
A game that is 50% functional on every PC is very different to one that is 100% on half of PCs and 0% on the other half and the distinction is important to know.
Including the variance would help but how many people even understand what variance even is, nevermind how many reviewers know how to calculate it?

 

[Hyperstorm]

A very valid point. I remember when Skyrim came out every review did everything short of giving it a rim job as if it were the second coming.

But for all it's 90%'s and 9/10's I hated the game and binned it (not literally, but almost) after a day.

Now days I go with my gut, reviews be damned and am a lot happier for it.

 

[sageoftruth]

One problem I see is, couldn't games get high technical scores just for being short? The shorter a game is, the fewer bugs you're likely to come across.

 

[Therumancer]

My basic attitude is that unless reviewers and critics can be detached entirely from the industry creating the games any kind of score is meaningless since at the end of the day they have to worry about their jobs and the industry being willing to send them more games to review.

That said the most important point I think is for reviewers to be consistent with all their reviews. Meaning that in reviewing a game now, it should be viewed not in the context of only the games coming out right them, or on the market recently, but in the context of every game the reviewer has played. Something increasingly important as gamers themselves have been at this for increasing amounts of time and can have very wide perspectives. Let's say your reviewing a first person shooter or dungeon crawling RPG, in assigning a review number you should consider it in comparison of other games of the same type. Does this game your playing now measure up to how good a game of the same type you played 10 years ago? If say you think Wolfenstein 3D was better for it's day (important qualification) than the current Wolfenstein is for today that should be in the review, if you happened to think say "Shadowcaster" or "Cybermage: Darklight Awakening" innovated more than a current first person shooter that should also figure. In short, especially in this stagnant industry, I think it's become increasingly important that developers compare themselves against all the games that were as opposed to entirely what's being done now. This means that some old grognard actually qualified to be a reviewer might wind up calling all games today crap compared to what he played in the 1990s because the innovation and a lot of the depth is gone, replaced entirely by graphics and sound in many cases. Being called on this is important. What's more the odd thing is I think a lot of graphics today are lacking compared to the 1990s where people did the best they could usually and the games looked great for their time, today it seems like even in terms of graphics and sound we're seeing them cut back due to things like Ubisoft's 30fps cap and things like that, basically you didn't see games in the 1990s saying "well yeah, we could do better graphics but it's more cinematic to release in 16 color EGA as opposed to this new fangled 256 color VGA stuff... and forget Sound Blaster, the PC speaker might make your ears bleed but it's more immersive that way".

 

[Therumancer]

I find this article to be full of terrible logic that attempts to support a preconceived conclusion, which, ironically, is something the author complains about in attempting to explain why review scores are bad.

The worst flaw is probably the idea that aggregate statistics, such as an average score, "merely flatten things out and become meaningless". To address this problem, it is claimed that we really need to see every single data point from some 20 or 25 independent reviewers to derive any meaning. Clearly not the case, as data sets of any size can often be well characterized by just two parameters: mean and variance of the sample.

Actually, the initial claim that 'averages are useless' because things get 'flattened out' is the sort of cringeworthy insight I'd expect from a 2nd year general arts student. Averages are great in certain types of analysis, particularly in the case of video games where the little niggle that would cause one player to dock marks is completely irrelevant to 90% of the gaming population. More precisely stated, every observation (individual review score) in the data set is contaminated by noise (personal preference that doesn't transfer to the majority of other players), and the only way to extract what we might call the signal (more objective assessment of the game's quality) is to compute the average rating. Obviously not everyone will agree that the average score reflects the game's quality, but it is the single number that MOST players will agree with.

To think you can just say "art is subjective" and be done with it is the height of ignorance.

I tend to agree with you here, when everything is art then nothing is. I tend to be a real snob and believe society should far no often be willing to say things are not art, music, etc... even if people happen to like some of those things. We've become so politically correct we've lost the ability to apply standards... and I think that applies to video game reviews as well, but of course a big part of the problem with video games is the rather incestuous relationship video game publishers and reviewers have. Reviewers cannot properly review or criticize a game when they are dependent on the publishers of those game for their continued livelihood. On some levels the way someone like "Total Biscuit" reviews is cowardly as it's a fancy way of trying to get around the difficulty of a big name personality who relies to some extent on the game industry getting cut down by alienating his own patrons, and at the end of the day part of the reason Total Biscuit does what he does is that he gets sent review copies of games ahead of time, if they stopped coming he wouldn't be able to review the way he does anymore, and while he'd still have a show, he would lose part of his popularity.

That said if I wasn't such a wreck I actually thought it might be fun to do a series called "Creepy Game Reviews" and use a Ouija board or Tarot cards to do a reading to see what the spirits think of a game and how it rates on the karmic scale of the universe. After all mortal reviews can be flawed, but the spirits of the great beyond? I can actually do a Tarot reading (though I'd of course have to brush up) but lack the presence for it. As an extension of it I might even do "reviews" of games that haven't come out yet due to umm... divination.

 

[Therumancer]

I agree with the other people in this thread. Scores are fine. After all, we can't expect people to understand there are nuances and subtleties that might affect people's gameplay experiences in a way that's not easily categorized into a numerical ranking system. And that small details are just as important as large ones if they affect people in big ways.

So yes. Let's keep a scoring system. Because if I want to disagree with a criticism/review, I want to be able to do it by dismissing an arbitrary number, not actually engaging the criticisms.

I can't tell if that last sentence is a sad truth or pure cynicism...

Or humor, it can be all three as well.

That said, he does hit upon something, critics and reviewers rarely actually engage their own critics. Heck, that's true of the gaming industry in general which tends to dismiss everyone they don't like as "toxic" and listen to only what they want to hear. Then of course you've got the whole Sarkeesian brigade who are people who like to dish it out but then prevent comments in return even being attached to their performance never mind addressing them. Now, I understand on something like the internet reviewers can't engage with every individual person making a comment, but the point is that it's pretty rare when they even try, they mostly do their thing and move on, and if they respond they try and sidestep the really hard counter points especially if it's likely to see them lose, or get into a protracted debate.

Engaging the number doesn't help either, but it's not like you generally have anything to engage with.

 

[Tracinya]

Haha yeah, the Functionality score seems like a brilliant idea. A game that is five minutes long and has you clicking one button would get max score for it since there's no room for bugs at all, and MMOs would automatically score 0 since they have no end, and inevitably gather up 100 bugs (usually pretty damn quickly). Conclusion: short games are better than long games.

 

[RolandOfGilead]

I always liked X-Play's five star system.
Before they changed it to giving out half stars, and then the show being canceled.

 

[Frederf]

The problem with functionality scores is how quickly game-breaking bugs get patched. It might CTD every 5 minutes on the reviewer's copy but be 100% stable by the time the customer can buy it. It's very perishable information.

Objective scores are largely a filtering mechanism for me which is very dangerous since I have tastes which might really like a general public 6 but hate a generate public 9. Also if it's IGN or something the numbers more reflect how in bed the game maker is with IGN than anything.

The review itself however should never ever ever ever tell me "you like this game." That's not a reviews position. What I want out of a review is condensed information. With better compact information I can decide for myself if I'm interested.

I'm the same way with "watch this movie, it's good." I ignore that crap. Good doesn't tell me about the movie.

 

[normalguycap]

It's a good idea. And it would give people jobs. Economics!