Evidentiary standards are slipping

Over the past month, there have been a number of instances in which a politician or journalist has made a bold claim, and then ignored or been unable to provide any evidence to support those claims.

For example, Fraser Nelson claimed that being in the EU had been a net detriment to the UK’s trade, and that the evidence he had seen supports that view. However, when provided with evidence that contradicted his claim, and when challenged to provide the evidence to which he referred, Nelson did not provide any sort of response. Likewise, Michael Gove claimed that there was evidence to indicate that leaving the EU would provide the UK with a “net dividend”. However, when pressed to provide the evidence that he claimed existed, Gove did not do so; nor did he respond to the provision of evidence that contradicted his view.

This is not just a problem for right-leaning opinion makers either; it affects left-leaning ones just as much. For example, despite copious evidence (from the Low Pay Commission) that increasing the minimum wage too high would be detrimental to the employment rate of low-income earners, Jeremy Corbyn claimed that increasing the minimum wage to £10 per hour would raise their living standards.  Again, Corbyn provided no evidence to support his claim.

This seems to be part of a wider, and long-running, malaise, in which policymakers can make a bold claim without any evidence to support it, yet said claim is taken at face value and isn’t challenged by the media nearly as often as it should be. Even worse (and a point made by Jonathan Portes in his recent discussion with Michael Gove), when challenged to provide evidence to support their views many in the media and political sphere tend to rely on a single statistic or anecdote even if copious evidence exists that contradicts their claim.

That’s assuming that the personalities concerned respond at all. Much of the time, they remain meekly silent, failing to respond, yet letting their original claim stand as though it hadn’t been challenged at all.

This isn’t just a point of pedantry – quite clearly, claims made by those covering and participating in campaign trails have real implications. For example, Vote Leave’s claim that Turkey would join the EU (despite all evidence to the contrary) likely played on some voters’ desires to reduce immigration (according to Ashcroft immigration was a major concern for roughly one third of voters), despite the fact that immigration has continually been proven to benefit the UK and everyone in it.  Similar points can be levied against various claims that the current level of trade between the EU and the UK could easily be replaced by trade with Commonwealth countries (despite the fact that the well-proven gravity model of trade directly contradicts this). And it seems likely that the upcoming election will be rife with claims and counter-claims that are (un)supported with evidence to varying degrees.

In essence, it is at least plausible that false claims made by opinion formers were taken to be true by some members of the voting public who based their decisions accordingly, and might have voted differently had they been informed of the actual evidence.

Now, what can be done to ensure that voters (and the general public as a whole) have actual evidence available rather than simply the claims of journalists and politicians?

Well, for a start, the press regulators (IPSO and Impress), the Electoral Commission, and the likes of the Office for National Statistics need to take on a much more proactive role. They should not wait for complaints to be submitted to them by the general public, but should take it upon themselves to investigate and penalise those in the public eye that make misleading or unsupported claims, with those punishments being far more severe than those currently used (for example, newspapers cannot continue to be allowed to get away with publishing retractions in the bottom corner of some page in the middle of their publication).

Second, political programmes like Newsnight, Question Time, and the Daily Politics should do far more to challenge politicians and journalists to support any claims they might make with sufficient evidence (i.e. more than just a single anecdote or statistic).  In other words, any journalist or politician appearing on such shows must be able to demonstrate that their claims are valid. The presenters on such shows should spend far more effort researching the actual evidence as well as questioning their guests on the basis of any claims that they might make.

Third, the Parliamentary Standards Committee needs to realise that their role in holding MPs accountable extends to claims made by MPs that are not supported by any evidence. Such claims are in violation of the MPs’ Code of Conduct and should be treated as such, with the necessary punishments for these violations being far more than the usual slap on the wrist.

Finally, and a much more long-term remedy, the general public should be provided with far greater training in the use and abuse of statistics. This should start from an early age and not only train people in how to calculate various (simple) statistics, but also provide information concerning how to spot when a commenter is using misleading figures or is relying solely on anecdotes to try to substantiate their points.

Once these suggestions have been implemented, the ability of journalists and politicians to deliberately obfuscate and mislead would be markedly reduced. That can only be a good thing.


Statistics, state benefits, and reproduction: some sort of unholy trinity

Over the past week, there has been some discussion regarding a book written by Adam Perkins (a Lecturer in the Neurobiology of Personality at King’s College London) regarding the impact of state benefits on personal outcomes (the title of the book – “The Welfare Trait: how state benefits affect personality” – probably gives that away).

Of particular focus has been one table that Perkins claims demonstrates that being on benefits is associated with a higher number of children in that household – i.e. that being on state benefits encourages reproduction (according to Perkins). The table, reproduced from here below, appears to indicate that households in which no-one is employed have higher numbers of children than do households with one or two workers.


Perkins’ conclusions were criticised by Jonathan Portes (a Senior Fellow at the NIESR) for a number of reasons, to which Perkins provided a response – see here.

However, this response is entirely unsatisfactory as it fails to acknowledge the actual problems with using the table above to make meaningful inferences. First, it does nothing to demonstrate that the results in the table are statistically significant – that is to say that the results in the are not just due to “random chance” (i.e. an artefact of the data used) and are, in fact, a “true” result. Without testing the results for statistical significance, there is no way to determine whether or not there is actually a meaningful difference between the number of children in working households as compared to workless household. Indeed, Perkins does not appear to even acknowledge this as an issue of his reliance on this table.

Second, Perkins’ numbers deliberately exclude households in which there are no children. This is an egregious decision that Perkins has tried to justify using baseless arguments. The issue is that Perkins could have deliberately introduced a bias into the numbers in the table that directly affects the inferences drawn from the results in the table. To see this, suppose that a higher proportion of workless households have no children than the proportion of working households that have no children.

Indeed, suppose that 500,000 working households do not have children, but 1,000,000 workless households do not have children. The table below, shows the impact of this hypothetical example – including households that do not have children actually reverses the direction. Once childless households are included in this hypothetical example, workless households actually have fewer children that do working households. Hence, it is essential that the number of childless households are included when trying to see if state benefits affect reproduction.

Hypothetical childless

In fact, when one uses the actual total number of households available from the ONS (rather than the hypothetical example above), we actually find a mixed set of results. The number of children per workless household is indeed lower than the number of children in working and mixed households, but the number of children per mixed household is higher than that in working households.

Actual childless

Hence, Perkins’ conclusions regarding the impact of benefits on reproduction are incorrect. The actual results suggest that there does not appear to be a systematic relationship between the employment status of a household and the number of children in that household. This goes to show how excluding certain groups from a dataset can introduce bias in any results obtained from analysing that dataset.