Follow Up: Rotten Bias

09 Feb 2011 One of the things that stuck in my mind from the last blog's analysis was 2007
The top two films for RT Score and Audience Score were Juno and No Country for Old Men. No Country got the Oscar and the highest RT score, but Juno got the highest Audience Score - this was one of the instances where the hypothesis of the last blog failed.

Now there was something I thought about in passing when I was writing the last blog, but I didn't bother mentioning it. But I thought I'd revisit and look in to it.

So that 'thing' is sample bias - that is, the people voting on films on RT (and creating the audience score) may not be representative of the general population.

I was originally concerned that the users of RT may be predominantly younger people; so then, their opinion and taste would be over-represented in the score. But I dismissed that idea as being a stereotyped view of internet users.

But realistically, it's best not to make assumptions either way if you can get actual details.


Rotten Audience

The are two sites I look at for site demographics. The first is Alexa, which you may have heard of. It has some benefits, but unfortunately for more detailed analysis, they want your money.

The other is Quantcast, which is a lot better, but sometimes their numbers are only estimates (not the case for RT). This is the one I'm using here. Here's what the demographics look like
NB/ this is US only demographics, which make up ~60% of the total visitors to RT. But the results are very similar for other countries.

Percentages to the left of the graphs are the actual distributions of the users. The 'index' numbers to the right show how those distributions compare to the internet as a whole.


Misrepresented Judges

So the 'hypothesis', or at least one the the hypotheses, of the last blog was that the Audience score could in some way be used as a predictor of Oscar winners, in so much as it reflects the 'wide appeal' of the films.

Of course, this only works if the taste of the voting comity for the Best Picture matches well the tastes of the people visiting Rotten Tomatoes and contributing to the Audience Score.

The membership of the Oscar comity is fairly secret, or at least, undisclosed. But we do know it's made up of ~6,000 industry professionals, so it seems safe to assume they're likely predominantly in the 35-60 age group, with maybe an even male/female split. And this could be a source of the weak correlation, since RT users are predominantly in the 18-49 age group - skewed slightly towards younger ages.

But do the different demographics have different tastes in film?

I suppose not necessarily. There are certainly some overlaps between these two groups. But on the other hand, maybe the difference between which films RT users vote highest, and which films gets Oscars is a reflection of these differences in tastes.

As it is, there's no way to manipulate RT Audience scores to better reflect the different demographic make up, and no way to improve on their predictive power.

So just some thoughts.


Oatzy.