Broken Online Review Systems: Through the lens of Ghostbusters

[Lacedaemonius]

http://fivethirtyeight.com/features/ghostbusters-is-a-perfect-example-of-how-internet-ratings-are-broken/

?Ghostbusters,? a revival of the 1984 original, hits theaters nationwide on Friday. As a reboot of a beloved, male-led science fiction film from the 1980s with a female-led cast, the reboot has proved somewhat controversial in the circles you would expect. But regardless of the quality of the film, it serves as a perfect demonstration of why internet movie ratings are inherently a problem.

Most fundamentally, single-number aggregations ? like those used by sites such as Rotten Tomatoes, Metacritic and IMDb ? are a pitiful way of explaining the diverse views of critics. More specifically, a vocal portion of men on the internet ? shall we say ? go out of their way to make their voices heard when it comes to judging entertainment aimed at women, and that appears to be happening with the new ?Ghostbusters.?

So we have that, which is pretty much where every discussion I've seen of reviews and Ghostbusters comes to a screeching halt, online. There is much more though.

But let?s back up. Last year, as part of an investigation into the inflated ratings on Fandango?s website, I looked at the world of online movie ratings in general. The moral of this story: Each site that aggregates ratings and reviews has its own skew one way or another, and it?s up to the user to determine which heuristic most accurately matches what they?d consider an ideal rating. (Also, don?t trust always-positive movie reviews from sites trying to use that review to sell you movie tickets. That, too.)

Spoiler: Here were the ratings curves for some of the most popular aggregators when I did this analysis in October -Hickey

That's ugly, but probably not a surprise.

Earlier this year, I also looked at IMDb?s user rating skew for television shows. Essentially, male users were more likely to rate television shows with a female-heavy audience lower than female users would rate male-centric television lower. Men were tanking the ratings of shows aimed at women.

But this ?Ghostbusters? thing? It lays bare so, so much of what we?re investigating when it comes to the provenance and reliability of internet ratings.1 Namely, they?re inconsistent, easily manipulated and probably not worth half the stock we put in them.2 Here are a few stats I collected early Thursday for the new ?Ghostbusters? movie:

IMDb average user rating: 4.1 out of 10, of 12,921 reviewers
IMDb average user rating among men: 3.6 out of 10, of 7,547 reviewers
IMDb average user rating among women: 7.7 out of 10, of 1,564 reviewers

The movie isn?t even out in theaters as I?m writing this, but over 12,000 people have made their judgment. Male reviewers outnumber female reviewers nearly 5 to 1 and rate ?Ghostbusters? 4 points lower, on average.

But it?s not just IMDb, there are serious disagreements on ?Ghostbusters? across the whole universe of ratings aggregation sites:

Metacritic score of 61 out of 100, based on 41 critics.
Of the reviews aggregated by Metacritic, 24 were positive, 16 were mixed, 1 was negative.
Tomatometer score of 74 percent ?fresh? based on 138 reviews.
Of those reviews, the average rating was 6.5 out of 10.
Looking only at top critics, it had a Tomatometer score of 54 percent ?fresh? based on 35 reviews.
Of top reviewers, average rating was 6.3 out of 10.

Now we get to the bottom line:

So even though 74 percent of critics enjoyed the film, according to Rotten Tomatoes, ?Ghostbusters? has an average score of 6.5 out of 10, so reviewers clearly didn?t love the film. A similar issue, only in the opposite direction, is apparent in the site?s top critics score. ?Top critics? is a somewhat exclusive category on Rotten Tomatoes, encompassing prolific and experienced critics at top publications with high circulations. Contrary to the hoi polloi of the Tomatometer, about half of reviews of ?Ghostbusters? from ?top critics? were negative, but it earned a middling-but-positive 6.3 out of 10, on average.3
...
, like the vast majority of people on earth, have not yet seen the new ?Ghostbusters? film. And to be honest, I don?t have a lot of skin in the game ? based on my birth date alone, ?Pokemon Go? is a bit more in line with manipulating my nostalgia for money than busting ghosts is.

The point is that this is a hugely instructive case for why internet ratings need to be approached with way more nuance than they currently are.4 People put far too much faith in numbers that are preliminary, decontextualized and, in the end, oversimplified.

This case demonstrates one way that ratings can be skewed, but in the context of how terribly skewed they already are. Does anyone here actually use review scores to determine what they buy though? I use trusted people who have similar taste to mine, so it's not an issue of broad recommendations. With Youtube... who doesn't do that?

 

[Avnger]

This case demonstrates one way that ratings can be skewed, but in the context of how terribly skewed they already are. Does anyone here actually use review scores to determine what they buy though?

1. Online polls of any type are roughly the same. You get awful results because idiots have a tendency to vote based on politics/trolling/making themselves look better rather than their actual thoughts or situation.

2. Over the years, I've come to the conclusion that gamers in general use review scores less as recommendations and more as validation for their already decided view (see death threats to reviewers who put too low a score and/or use the "wrong" criteria when scoring). They don't want to hear someone else's point of view, they want their own parroted back at them.


As for me, I almost never care about the actual "score" of a review. Reading the article or text or watching the video is much more informing about whether I'll like it.

 

[Zontar]

For a moment I thought this was going to talk about how its middling review outlook is skewing the more polarized reviews it's getting, or why critics call it middling while top critics call it trash, and a possible discussion as to why that was the case.

Instead I get an article talking about how men and women like different things and because unlike the original the new Ghostbusters is only meant for women to find entertaining that skews things negative.

 

[Cap'nPipsqueak]

Okay guys; cut it out.

Two topics concerning this have already been canned. Give it a rest, huh?

 

[Avnger]

For a moment I thought this was going to talk about how its middling review outlook is skewing the more polarized reviews it's getting, or why critics call it middling while top critics call it trash, and a possible discussion as to why that was the case.

Um did you take a look at the statistics provided?

Looking only at top critics, it had a Tomatometer score of 54 percent "fresh" based on 35 reviews.
Of top reviewers, average rating was 6.3 out of 10.

Top critics didn't "call it trash." They very obviously called it "middling" (to use your word).

Instead I get an article talking about how men and women like different things and because unlike the original the new Ghostbusters is only meant for women to find entertaining that skews things negative.

Love this sentence. To avoid putting words in your mouth, please explain exactly how the movie was "only meant for women to find entertaining." Was it the all female leads cast? Was it the rather plain slap-stick comedy? Was it the clearly making a movie to appeal to nostalgia? The uninspired writing?

Because out of all those I listed, only one could possibly strike me as something that could make a person think the movie was "only meant for women," and it would say a rather large amount about anyone who get stuck on that point. The other aspects are found in a sadly large amount of modern remakes/sequels/reboots/etc, and I rarely hear such things said about them. Take the example of Adam Sandler's Pixels movie; it hits all of those points, but no one was even trying to say that was "only meant for women to enjoy."

 

[Johnny Novgorod]

The fact that fans and haters alike are using online reviews to further their causes should tell you reviews have been fairly mixed.

 

[Silentpony_v1legacy]

I won't pass judgement until the Nostalgia Critic review.

 

[Parasondox]

Another thread about Ghostbusters????

Can we just give it up and let it go already?

Why are some of you so addicted and attracted to controversy like a fly to a bright light?

The horse is dead. Leave it be.

 

[Thaluikhain]

Eh, if the thread as supposed to be about online reviews using a well-known example, I can't see anything wrong with it, excepting that it'd go don in flames without much discussion of online reviews.

 

[deleted+75193]

DELETED

 

[Chanticoblues]

I won't pass judgement until the Nostalgia Critic review.

That guy's still around?

 

[Wintermute_v1legacy]

This is the movie event of the decade.

 

[Bilious Green_v1legacy]

If Ghostbusters proves once and for all that online review aggregation is bunk, then it's all been worth it.

 

[Mangod]

Much to my surprise, I actually really enjoyed it. I think that may be in part because my expectations were so astronomically low, but it's easily one of the better movie remakes in recent memory.

So, is it on par with Mad Max: Fury Road [https://www.youtube.com/watch?v=hEJnMQG9ev8] where remakes are concerned?

 

[sXeth]

Just as a point of note. I'd like to mention that its reiterated multiple times in the article that people have rated a movie before its out and they can't have possibly seen it. I consider posting a "Welcome to the Internet", but that would kind of vague. Let's just leave it at, you shouldn't be reporting on the internet if you think people can't find ways to see movies that aren't released yet.

Much to my surprise, I actually really enjoyed it. I think that may be in part because my expectations were so astronomically low, but it's easily one of the better movie remakes in recent memory.

So, is it on par with Mad Max: Fury Road [https://www.youtube.com/watch?v=hEJnMQG9ev8] where remakes are concerned?

Fury Road wasn't a remake. But I'd say no. On a cinematographic level alone, Fury Road would win that comparison by miles. There's a lot of weird edits and oddball shots in Ghostbusters. I'm not really on the educated level to do more then notice they're weird, but you can find lots of video reviewers mentioning the problem and covering it in more depth.

 

[BloatedGuppy]

Okay guys; cut it out.

Two topics concerning this have already been canned. Give it a rest, huh?

Another thread about Ghostbusters????

Can we just give it up and let it go already?

Why are some of you so addicted and attracted to controversy like a fly to a bright light?

The horse is dead. Leave it be.

Feels like it's a thread about critical interpretation of review aggregate sites. I'm not sure that subject should be verboten simply because some of the residents can't have a civil conversation about "Ghostbusters". We do tend to throw around aggregate numbers as evidence, and sometimes a lot more goes into those numbers than just "the number"...as detailed in the OP.

Also this...

Earlier this year, I also looked at IMDb's user rating skew for television shows. Essentially, male users were more likely to rate television shows with a female-heavy audience lower than female users would rate male-centric television lower. Men were tanking the ratings of shows aimed at women.

...was interesting, if depressing. Unfortunately, time has demonstrated that we cannot have adult discussion of stuff like this around here, so I get that it's perceived as flame bait.

 

[Worgen]

I won't pass judgement until the Nostalgia Critic review.

Double Toasted did a review.

They do good movie reviews, their Zootopia review is one of the best around.

 

[Pirate Of PC Master race]

You mean it is working as exactly as it is intended to be?

So no problems then. Moving on.

 

[Areloch]

Also this...

Earlier this year, I also looked at IMDb's user rating skew for television shows. Essentially, male users were more likely to rate television shows with a female-heavy audience lower than female users would rate male-centric television lower. Men were tanking the ratings of shows aimed at women.

...was interesting, if depressing. Unfortunately, time has demonstrated that we cannot have adult discussion of stuff like this around here, so I get that it's perceived as flame bait.

This makes me wonder what causes this to occur. I have extreme doubts it's just huge, broad-scale "Ew, girls".

Are men more inclined to be harsh in reviews of stuff that doesn't appeal to them?