An in-depth analysis has revealed there simply is no evidence to justify the claim that South African referees are biased towards local teams.

Ed’s note: This analysis was conducted by Nicholas Caine, and supplied to SA Rugby magazine with permission – it was first published on The Sport Scientist

In February 2020, an Australian rugby fan called ‘Rebels3’ produced a study, claiming to show how South African rugby referees were exhibiting favourable bias towards South African home teams. The study did not consider how other countries’ referees treat South African home teams, an important comparison that needs to made before any conclusion can be drawn.

Yet, the unvetted study was reported by sports journalists from all over the southern hemisphere, leading to calls for official inquiries. The study’s conclusions are shown to be unfounded. Yes, this is about six months late; we’ve all been a little preoccupied.

At least one brazenly worded open letter was sent to the CEO of Sanzaar. The hype around the unfounded conclusions prompted Australia’s rugby franchise executives to call for an official inquiry. All of this for a fan’s unvetted spreadsheet.

For context, Vodacom Super Rugby’s professional referees are are not required to be neutral, a matter at the heart of the alleged problem.

Original fan study
The table below shows how the fan study sampled different groups of matches.

 

 

 

 

 

 

 

 

 

 

 

 

The bar chart below shows each group’s average penalty difference, the metric used in the fan study. The penalty difference is simply the difference between the penalties awarded against the home and away teams.

The chart shows an Australian home team with an Australian referee, on average, was penalised 0.5 fewer times than their foreign opposition between 2017 and 2019. A New Zealand home team with a compatriot referee received 0.1 fewer penalties.

These two figures are in stark contrast to the fact that South African home teams refereed by a South African were penalised 3.5 fewer times than their overseas rivals. This large difference is what drives the idea that South African referees have a demonstrable bias toward South African teams.

The fan study only considered the years 2017 to 2019, but if we expand the range from 2009 to 2019 (there is no Super Rugby penalty count data before 2009) we see a similar picture:

Again, this is a sound preliminary study, or maybe the first step of a preliminary study, and Rebels3 is clearly on to something. But, to know what that something is, one needs to dig deeper.

Fan study ignores how other referees treat South African teams

This is where the misunderstanding happened. People saw the dissimilarity in average penalty differences and immediately jumped to the conclusion that South African referees unfairly favour South African teams.

However, the study only makes comparisons between Australian home matches officiated by Australians, New Zealand home matches officiated by New Zealanders, and South African home matches officiated by South Africans. What it doesn’t do, is compare South African home matches officiated by different groups of referees, for example. This is a huge oversight.

Obviously, we cannot say South African referees favour South African home teams if we haven’t even looked at how other referees treat South African home teams. It’s completely possible that referees from other countries favour South African home teams to a similar or even greater extent (foreshadowing).

The problem can be more formally expressed in terms of the study’s variables.

 

 

 

 

 

 

 

 

 

 

 

In each of Rebels3’s groupings, not one variable is kept constant. With such distinct groups and uncontrolled variables, it would be incredible if the fan study didn’t show a difference between the groups.

How can one conclude that referee nationality, and not any of the other variables, is responsible for the dissimilarity in penalty difference? Such a conclusion would indeed be highly premature and unfounded, given the limited information we have so far.

In any experiment or study, to test any single variable, we need to keep the others as constant as possible, a sentiment voiced in the forum where this study first appeared.

So, to test if a referee’s nationality really is a factor, we need to break our referees into different nationality groups, much like Rebels3 did. But, we need to control the variables we aren’t testing, as in the below table. The set-up in the table below isn’t in conflict with or even an alternative to Rebels3’s study, but rather its natural next step.

 

 

 

 

 

 

 

 

 

 

 

 

In controlling the variables we aren’t testing, we are able to say with more surety that the independent variable, referee nationality, is or is not a significant factor in the dissimilarity between each groups’ penalty difference.

In other words, as previously said, before we can know if South African referees are treating South African home teams differently, we need to see how other groups of referees treat South African home teams.

Further analysis
Let’s do just that. Let us keep the home team South African and the away team not South African, while breaking the referees into five clearly defined groups:

• neutral Australian
• neutral New Zealand
• non-neutral Australian
• non-neutral New Zealand
• non-neutral South African

Since the home team is always South African in this case, there can be no neutral South African referees.

Looking at the chart below, we see that all groups of referees favour the South African home team, albeit marginally in the case of non-neutral New Zealand referees. However, there is one group that favours South African home teams more than any other. And it’s not the South African referees.

On average from 2009 to 2019, South African referees penalised South African home teams 3.1 times fewer then the non-South African away teams (2017-2019: 3.5). However, the non-neutral Australian referees’ penalty difference was 4.1 in favour of the South African home team (2017-2019: 6).

It’s simply impossible to conclude that the South African referees’ penalty difference is proof or even evidence of unfair bias when the Australian referees penalty difference favours South African teams even more.

Taking that a step further, using the groups from the above chart, the South African referees’ average penalty difference was closest to that of the neutral referees (see bar charts above and below).

So, not only is it unfounded to claim that South African referees favour South African teams, but there is evidence that South African referees may be the least biased when officiating non-neutral matches in South Africa.

 

Conclusion

Blaming the ref is as old as sports officiating itself. As is usual in such cases, there simply is no evidence to justify the claim that South African referees are biased towards South African teams.

To reiterate, there is no problem with the original stats presented by Rebels3, if taken for what they are: part of a preliminary study, drawing attention to a potential issue. The real problem with this scenario is how the story was blown out of proportion without being properly examined.