Why is experimental evidence not used more and what can we do about it?

Candid reflections on what learnings from randomised controlled trials can and cannot do for policymakers, and why we still need them to inform decision making

Five years ago, the Innovation Growth Lab launched a survey to find out what was preventing policymakers from using evidence coming from randomised controlled trials (RCTs) to inform their decisions. As part of a project to bridge the gap between research and practice, we recently revisited this question through a series of small workshops with key actors from the United States’ business support ecosystem. The key question we asked was: What prevented access and use of scientific evidence for decision makers today? 

What we found was that the majority of the barriers identified - limited resources, time pressures or a lack of internal skills to interpret studies - aligned with what we had found five years prior. However, other reasons raised seemed to run a little deeper. Issues around the transportability of evidence from one place to another and representative biases in existing pools of studies were some of the barriers we found harder to argue against (although these criticisms stretch beyond experimental research alone). Nonetheless, we consider these challenges to be a good thing as they indicate that policymakers and business support providers care deeply about really improving the programmes they provide.

At the Innovation Growth Lab, we also believe that another issue is at play when it comes to barriers to using experimental research. Often evidence will be considered in ways it wasn’t intended to be. Traditionally, evidence may have been viewed as the gateway to the silver bullet solution; a policymaker wanting to convince a Ministry to spend public money in one way as opposed to another will often be seeking ‘proof’. However, in a fast changing and more complex world, ‘proof’ becomes harder to attain and sustain prior to trying something new first. Instead, we believe that evidence should be used to inspire new ideas for interventions that are then tested and further refined or modified to fit a local context. 

What prevents policymakers and practitioners from using experimental evidence?

Internal barriers to evidence use

After talking to a group of policymakers in the United States’ entrepreneurship and business support ecosystem last summer, we found that the majority of the barriers preventing them from using evidence had come up with a similar group we engaged with previously. Barriers we had heard of before included a lack of time to identify and distil existing evidence, lack of expertise to discern reliable evidence from bad, and difficulties slotting experimental evidence into existing decision-making processes.

Staying on top of the field is resource intensive. As one workshop participant said: “There is always more and newer evidence available”. 

Not only that, the quality and rigour of available evidence varies greatly, and results from one study to the next are not always easy to reconcile. So trying to keep up with the evidence can be a daunting task for policymakers and practitioners, which can easily get deprioritised in favour of more urgent calls.

Transportability of the experimental research

Most experimental evidence will come from a different context and a lot of evidence about entrepreneurship and business support schemes comes from developing countries. We heard that policymakers and practitioners in the United States, and other OECD countries, will often question how transferable the learnings arising from these studies are, given fundamental differences in how their systems operate. 

This is especially true when findings suggested by existing evidence do not quite align with the day-to-day experiences of these same policymakers and practitioners. It can be easy in these instances to discard insights from the evidence entirely. 

Representativeness of the evidence base

Another barrier is that available experimental evidence fails to represent all the policies and support schemes that policymakers and practitioners in the entrepreneurship and business support space care about. 

This particular barrier has many levels to it. To start with, among all the instruments that could potentially achieve a desired goal, like encouraging business growth, only a few end up being funded and implemented. Of those which are funded, few to date have looked to meet the needs of traditionally underserved entrepreneurs and even fewer target the more recent societal challenges we face today. On top of that, not all policies are well suited for an RCT. Most of the time, experiments (and other robust impact evaluations) look at well defined interventions where it is easy to exclude a group of potential beneficiaries who can act as a control group. Rarely do we see robust evaluations looking at the impact of system-wide interventions or public goods. 

Finally, not all robust evaluations are likely to get published. Studies indicating that an intervention works are more likely to get published than those that show inconclusive or null effects. All of this makes the pool of evidence we can use both small and narrow.

What can we do about this?

Improve how we consider transportability of evidence

Worries about the transportability of the results are fair. Results obtained from randomised controlled trials (RCTs) are inevitably influenced by the conditions under which the given intervention was tested. A policy that has worked amazingly in one specific context might not be effective in another context. However, when we do not have rigorous evidence on the effectiveness of a specific intervention in the context we are working in, knowing that it has worked somewhere else can be an important starting point, even if the original context does not look that similar to ours at  first sight.

What’s important is, first, to understand the conditions and mechanisms that allowed for impact in the original context: Who participated? What challenges or barriers were participants facing? What exact support was provided? In what macroeconomic context were they operating? And what factors drove the positive impact? That is, not only what has or hasn’t worked, but also why, for whom and under what circumstances. Researchers and institutions doing evidence translation work are in a good position to provide this input.

Second, it is important to have practical and localised knowledge about the destination context (where the evidence will be transported to) to judge whether the same conditions and mechanisms are likely to hold there. Local policymakers and practitioners are in the best position to adapt promising approaches suggested by evidence in one context, to ensure impactful delivery in their environment.

What we need to remember is that evidence does have the potential to be transported across contexts but it requires researchers and policymakers to work collaboratively, with a clear framework that can guide them through the whole process (with resources and expert knowledge dedicated to it).

Build a culture around evidence use

With many factors to consider to make better use of existing experimental evidence we need more dedicated practitioners seeking and translating scientific findings for business support programs. Unfortunately, it is still rare to find business support organisations with dedicated staff who are trained to seek and make use of existing evidence, and whose roles prioritise this task above others. In addition, it isn’t common to see organisations using experimental methodologies to assess the effectiveness of their schemes. And when they do, time pressures to make decisions quickly make it more likely for them to focus on short-term outcomes, like business knowledge or adoption of better business practices, rather than medium-term outcomes that policymakers usually care more about, like business survival, performance and growth. 

We need to see more resources to help build a culture around making use of evidence that stretches from top to the bottom. 

Keep expanding the evidence base

Although we’ve seen more experimental studies happening over the past years, in part thanks to the IGL Grants Programme, INNOSUP and Business Basics, the experimental evidence in the entrepreneurship and business support space is still scarce. On top of this, existing evidence does not cover the richness of schemes that are currently being implemented to support entrepreneurs and businesses.

To address this challenge, policymakers can also be part of the solution by starting to use experimental methodologies more often to evaluate the effectiveness of their support schemes - particularly when evaluating programmes we know are underrepresented in the existing evidence base. Examples of this can be business support schemes taking place in developed countries, programmes targeted and tailored to traditionally underserved populations, interventions aiming at helping start-ups in early stages, or greening SMEs programmes, among others. Initiatives to support the implementation of RCTs, like experimentation funds and capacity building activities, will be important catalysts for this to actually happen.

Final remarks

While we recognise the number of barriers to using evidence to inform business and entrepreneurship support, at the Innovation Growth Lab we also believe that not all hope is lost.

If researchers, practitioners and policymakers find more opportunities to have more open conversations around concerns surrounding experimental research, we believe that there could be an increased understanding of where scientific evidence can help policy the most. 

Secondly, if we train more staff working in business support ecosystems to acquire and retain the right skills to translate research, with dedicated time to practice these skills regularly, we can build a new generation of policymakers and practitioners who are confident and capable of transporting results from one context to another.

By cultivating a culture around evidence, we should also start to see more rigorous evaluations taking place and an expansion of the scope and diversity of the existing evidence base. 

If you are keen to move your organisation forward in any of these dimensions, do get in touch with us. We can support you to make your organisation more experimental and your policies and programmes more evidence-informed.