How to assess the effectiveness of the government’s Prevent counter-extremism programme

In the aftermath of the Manchester bombing, many questions were raised regarding the effectiveness of the government’s counter-extremism programmes, with particular focus on the Prevent arm of that strategy. For example, the FT reported that Salman Abedi was referred to Prevent, but that report was not followed up (although note that the police state that they cannot find any record of Abedi’s referral to Prevent). More generally, the Prevent programme has been called “toxic” by the now mayor of Manchester Andy Burnham and the Home Affairs Select Committee.

However, there are a number of examples of the Prevent programme also having done a lot of good. For example, two teenagers were stopped from travelling to Syria after being referred to Prevent by their parents in 2015, and it is credited with helping to stop 150 people (including 50 children) from going to fight in Syria in 2016.

Importantly, much of the discussion regarding Prevent’s effectiveness has focused on anecdotal evidence, the odd stylised fact here and there, a couple of case stories. Most criticisms or praise of Prevent focus on a few examples where it has not worked, been implemented badly, or succeeded. There are even calls to expand or to shut down Prevent without any evidence of whether or not it is actually an effective programme.

Indeed, as far as I’m aware, there has been no (publicly available) rigorous or systematic assessment of Prevent’s effectiveness. (Note that although the recently-launched book “De-Radicalisation in the UK Prevent Strategy: Security, Identity and Religion” by M. S. Elshimi claims to constitute such an assessment, its results are based on an absurdly small sample of only 27 people and therefore cannot be considered a systematic analysis.) However, conducting a systematic assessment could be a relatively simple procedure.

In particular, if data are available on the number of extremist / terror convictions and/or number of people successfully and unsuccessfully “treated” by Prevent at the level of individual local authorities, then it would be possible to use variations across those local authorities to assess Prevent’s effectiveness.

Put simply, the “outcome” variable (i.e. metric that assesses Prevent’s success) could be the number of extremist / terror convictions or the proportion of people referred to Prevent that are successfully treated. (Obviously if the number of convictions is used, it would be important to allocate those convictions to the local authority in which the extremist grew up and/or resided rather than where the extremist activity was carried out.) Of course, there would also be technical considerations regarding whether the outcome variable is a “count” variable, is bounded due to being expressed in percentage terms etc., but those can be dealt with relatively easily.

The explanatory “variable of interest” that would then measure the actual effectiveness of spending on the Prevent strategy would be each local authority’s annual budget for Prevent. If Prevent was effective, one would expect this variable to be negatively related to the number of convictions (since a successful Prevent would stop people before they committed a crime) and positively related to the proportion of successfully treated people. Alternative variables of interest could include the number of Prevent-dedicated personnel in each local authority or the amount of Prevent training that is provided to practitioners – each of these could be investigated to try to identify the most effective/important aspect of the Prevent strategy.

Note that it is unlikely that there would be any simultaneity between the Prevent budget (or other variable of interest)  and the outcome variable – although it is plausible that current Prevent spending would be based on past extremist activity in a local area (i.e. local areas with higher extremist activity get more money for Prevent), it is unlikely to be the case that current Prevent spending reacts quickly enough to be affected by current extremist activity. Nonetheless, this could be investigated by using lags of the Prevent budget variable as instruments or as the variables of interest themselves (since it could well be the case that Prevent takes time to have an impact).

As Prevent has been running since 2003, and there are roughly 400 local authorities in the UK, that should give a sizeable panel of data on which to conduct some relatively simple regression analyses. Of course, a number of other factors would need to be taken into account – for example, the population of each local authority, the average income within it, any changes to Prevent guidelines and/or the introduction or suspension of other counter-extremism strategies. The “identification” of the impact of Prevent would therefore come through variation in Prevent spending (or other Prevent-related variables of interest) and outcomes across local authorities and across time.

Now, I don’t have the data to conduct this analysis. However, I suspect that the data are out there – I understand some organisations have created databases containing details of all extremist-related convictions over a reasonably lengthy period of time (for example, The Henry Jackson Society has a dataset on all Islamist-related convictions from 1998-2015, but this would also need to be supplemented with data on other forms of extremism covered by Prevent).  Moreover, local authorities / the Home Office / the relevant government authority no doubt have records of the amounts that were spent on Prevent by local authorities (as well as the number of Prevent-related personnel etc.) on an annual basis . As such, combining the two together (along with the various controls) should provide a useable dataset fairly easily.

Hence, if the government and/or organisations with an interest in Prevent really do want to assess how effective is the Prevent strategy, then it actually isn’t very difficult to do so.

Gravity, Gobbledygook, and Government Reports

Mark Reckless is an idiot. Now, if I stopped there, it wouldn’t make much of a blog post (although it would, as usual, be factually accurate). So, you might be asking, “Why is Mark Reckless an idiot?” But that is the wrong question. Instead, what you should be asking is “How has Mark Reckless demonstrated his idiocy this time?” And that would be a very good question indeed.

The answer to that question rests in the tweet below. This was Reckless’ comment regarding a description of one part of the methodology that the Treasury used to estimate the costs of Brexit – the equation in question was contained in a technical annex (i.e. where one would expect to find a detailed explanation of the approach used).

Reckless appears to be claiming that the equation in the picture he posted is equivalent to the fraudulent claims of a fortune teller. That could not be further from the truth.

Instead, the equation posted by Reckless is an algebraic representation of the “Gravity Equation” as applied to international trade. Emanating from Newtonian physics, this equation relates trade between two countries to the relative sizes of those countries (in terms of output and population) and the distance between them, plus some other controls for whether or not the countries in question share a common border / language / colonial history.

This is not a controversial method to estimate the impact of those factors on trade between two countries. In fact, the use of gravity equations is widespread in the assessment of international trade. A priori, one would expect larger countries to trade more with each other, but countries that are further away to trade less with each other and this is indeed reflected in the Treasury’s results.

The main point of this exercise, however, was to estimate the impact of being in the EU on the UK’s trade, and the Treasury does this by including a variable to capture that. The main result is that being in the EU increases trade in goods by about 100% (i.e. leaving the EU would result in a decrease in trade in goods of 53%) and increases trade in services by about 22%. Hence, being in the EU increases trade in goods and services overall by about 75%.

However, although the main approach used by the Treasury is reasonable, there are some areas in which it could be refined further. First, the Treasury’s analysis uses data covering the period 1948-2013, yet does not really try to control for factors that change over time (other than GDP and population). For example, there have been substantial changes to exchange rates and barriers to trade during the period covered by the Treasury’s data, both of which would have had substantial impacts on trade between two countries. The Treasury’s attempt to control for these changes over time consists solely of using dummy variables for each year (that do not vary across countries), which cannot even begin to capture the changes in exchange rates, trade barriers etc that would have occurred over the time period. This means that the estimated impact of being in the EU could well be incorrect.

Second, the Treasury’s approach assumes that the impact of being in the EU is the same for all countries. However, it is possible that the EU has an heterogeneous impact across countries – for some countries the impact of being in the EU might be larger than it is for other countries. By assuming away this possibility, the Treasury is likely to have under or over-estimated the impact of Brexit on trade.

Third, and on a more technical note, the Treasury does not specify what standard errors it has used. If the Treasury has used incorrect standard errors (for example, ones that do not correct for serial correlation or heteroscedasticity), that means that the statistical significance of its estimates is incorrect and, more importantly, that the error bounds (i.e. the upper and lower ends of their estimate) are likely to be incorrect.

Nonetheless, these minor potential refinements of the Treasury’s approach do not detract from the fact that Mark Reckless has been remarkably foolhardy in his response to the Treasury’s assessment of the impact of Brexit.