What prevents government agencies from making a greater use of randomised controlled trials (RCTs), as well as evidence to inform their policies? Last summer we set out to answer this question, and to try and tease out which barrier is the most important.
To this end, we launched a survey, aimed at policymakers, practitioners and researchers. We would like to thank all of those who helped us by responding to this survey, and provide a summary of the key findings.
The respondents
We received over 170 responses from a very heterogeneous group. About 21% of answers came from academics, 35% from the public sector and 37% from the third sector (foundations, think-tanks, etc.). Half of all respondents reported being involved in evaluating a programme; among them, roughly 70% had also delivered or designed one, such that a third of the total had been involved in all three activities. Only 6% said they had not taken part in any of those activities. Overall, 31% had been involved in an RCT in some way, although only 12% wrote that the organisation they currently work for make use of RCTs.
The answers: barriers ranked by importance
One of the main objectives of the survey was to identify which barriers people thought were more important. We were surprised to find that respondents attributed a high degree of significance to most of the options we presented.
The table below reproduces the percentage of respondents answering ‘Important’ or ‘Very important’ to challenges in conducting RCTs. Most barriers received answers above 50%. The three barriers that stand out, each above 70%, are ‘Limited resources and time constraints’, ‘Programme design might not lend itself to RCTs’ and ‘Programme has already been rolled out’. On the other side of the spectrum, answers that received relatively lower percentages include ‘Legal or regulatory difficulties’ and ‘Difficulty in finding appropriate research/delivery partners’, at around or below 40%.
As for barriers to the greater use of evidence to inform policymaking, all options were rated highly (above 65%), with ‘Pressure to make policy decisions before rigorous evidence emerges’ topping the ranking at 82%.
What is the main barrier?
At first sight, this does not tell us very much. That is why we asked respondents to write in what was, in their view, the single most important barrier to running RCTs and to using more evidence (not necessarily from amongst our options). The answers were quite varied, especially when it came to RCTs; we classified them and provide the top answers in the tables below.
For barriers to running RCTs, the most common answer was ‘Lack of resources and/or time’, followed closely by a number of answers related to a lack of knowledge, such as ‘Lack of understanding of RCTs among senior officials’ or ‘Lack of knowledge or skills to implement RCTs.
For barriers to the use of evidence, the most recurrent item was ‘Competing priorities’ or political demands, including the short nature of the policy cycle (i.e. results need to be in faster than evidence can be gathered, analysed and incorporated). Also important were the answers related to the lack of skills (both in terms of evidence generation and consumption), as well as ‘knowledge inertia’, a term we used to categorise answers pointing to the fact that decision-makers often believe their opinion is correct and do not demand better evidence of their programmes’ impact.
Breakdown of answers by group
Were there any differences among groups in terms of answers? Although the number of responses does not allow us to do an in-depth group analysis, there are a few things that stand out:
- A majority (72%) of respondents from the public sector said that the main barrier to the greater use of evidence was political (e.g. relating to the policy cycle, or competing political priorities, etc.).
- Conversely, a majority (55%) of answers from the non-profit/think-tank sector pointed to policymakers’ lack of understanding of evidence as the main barrier to the use of evidence. For academics, this number was even higher (80%).
- Regarding barriers to RCTs, the largest group of answers among policymakers and public sector practitioners revolved around concerns with public reactions and fear of failure (24% of all answers), followed by concerns of the delivery partner organisations.
- For those in the non-profit sector, however, the main concern seemed to be cost, time constraints and complexity, with 38% of all answers; this number was 14% among public sector respondents.
- Finally, the main difference between respondents whose organisation was already involved in RCTs and those who were not was their concerns for ethics. In fact, ethical concerns were the main worry for 16% of respondents in organisations that had not been involved in RCTs; for those whose organisations had run an RCT, this issue was never mentioned. Moreover, more answers from the latter group related to securing delivery partners (14%) and costs and resources (19%); for those not involved in an RCT, the other main worries were public reactions (12.5%) and lack of skills to conduct an RCT (16%).
What can we learn from the survey?
One of the take-aways from this exercise is that policy organisations require a significant change in mind-set with respect to the use of evidence and rigorous methods like RCTs. Some of the recurrent answers that appeared as main barriers include things that are inherent to the way these organisations work, such as the short nature of policy-cycles and the perennial lack of resources to do evaluation and gather evidence on what works. Many also pointed to a lack of understanding of the benefits of using more evidence or running RCTs, especially by senior management and politicians.
There is no easy fix for this. The change in internal culture needed to overcome these barriers is not something that can happen overnight, and there are many obstacles. Moreover, there are certain things – such as the need to listen to what the public wants – that cannot be sidestepped.
Nevertheless, the results from the survey point to solutions that we can work towards. An important barrier that came up repeatedly was the lack of expertise among practitioners and policymakers, coupled with a lack of understanding of the benefits of using evidence and rigorous evaluations like RCTs. IGL, alongside many other organisations, has been working to tackle this. The fact that respondents whose organisation had run an RCT were not as likely to mention ethical concerns or public reactions to RCTs shows that exposure to trials can go a long way in demonstrating the value of well-designed and ethical RCTs, as well as their feasibility. Moreover, there are a number of groups, such as the Alliance for Useful Evidence and the What Works centres, that are helping translate evidence so that it can be used in public policy.
Finally, with regards to the cost and resource constraints to RCTs, IGL and others have been promoting the use of smaller, lower cost trials that can produce a number of interesting results. Some government agencies, such as the UK’s Department for Business, have recently been successful in conducting small messaging trials that have helped them achieve policy objectives.
In sum, these answers point to the areas where more work can be done – promoting the benefits of evidence and RCTs, as well as building capacity within policy organisations to overcome these barriers.
We welcome further comments or questions on this project at [email protected]. We thank the individuals who took the time to answer our survey.