The IGL Winter Research Meeting, held on 13th December in London this year, brought together a group of researchers to discuss experimental research aimed at improving our understanding of the drivers of innovation, entrepreneurship and growth, and some potential interventions to accelerate these.
Researchers around the world presented design-stage, ongoing and completed studies. These ranged from looking at the effects of business coaching on new technology-based firms to estimating the impact of interventions encouraging SME adoption of cutting-edge but tried and tested technologies, to stimulating innovation and productivity improvements more broadly (see the full programme here).
Sharing the wealth of knowledge and experience of running trials in this field formed a key aspect of the meeting. The researchers presented their randomised controlled trials (RCTs) followed by feedback on the strengths and limitations of the studies from the discussants and the wider audience. Drawing on the studies presented and the feedback, the following key takeaways emerged:
- Trial management and monitoring form a crucial part of a trial success. A rigorous trial design alone is not a key to credible results. Appropriate planning before the trial and adequate oversight and monitoring during the trial will help to ensure high quality implementation and data, but also ensure that reporting of results and conclusions drawn at the end of the study are accurate. Therefore, any risks and mitigating actions need to be identified as early as possible.
- The value of feasibility studies/pilot RCTs cannot be underestimated. Pilots are not a way to run trials to justify a small N. Instead, they provide a great opportunity to identify and address any feasibility issues prior to the main RCT. This way we can avoid wasting resources on trials that will never generate sufficiently rigorous evidence but still put a burden on trial participants.
- More thought needs to be given about where a trial might land on the efficacy-effectiveness continuum in the design stage and throughout the study duration. Most trials in this field tend to be designed as effectiveness (pragmatic) trials though there appears to be an appetite to test the efficacy of interventions as well. Some trials start out more as efficacy trials but end up being ‘pragmatic’ trials due to the compromises made during the course of the trial. Nevertheless, we shouldn’t lose sight of what we truly set out to test in the design phase, as it will affect the choice of a sample, implementation features, analytical choices and the interpretation of the results.
- Don't underestimate the role of implementation and process evaluation. This evaluation strand helps us to understand how a project is implemented on the ground and the elements of successful delivery. Increasingly, it also helps us to explain what 'business as usual' means for trials. Process evaluation can help us to identify interventions that are inherently faulty (theory failure) and those that are badly delivered (implementation failure). Overall, implementation and process evaluation has a crucial role in the interpretation of trial outcomes.
- Lessons need to be learned when trials don’t go as planned. There are a range of reasons why trials can go wrong, from delivery challenges and recruitment failures to a high level of participant and/or outcome data attrition. Sometimes trials are discontinued for these reasons. That doesn’t mean, however, that learning can’t be generated from these studies or that these studies can’t be ‘re-scoped’ or extended to look into the specific aspects that led to the failure (for an example see this article from EEF). We need to make the most of opportunities to learn from our experiences, and share our learnings with other audiences to ensure we do better in the future.
The IGL Winter Research Meeting created a forum to generate insights, reflect on lessons learnt and share the experiences of a range of researchers working on trials in this field. It’ll be hugely exciting to see the results of the studies presented at the meeting being published. Meanwhile, the next IGL Research Meeting will take place during 21-23 May 2019 in Berlin as part of the IGL2019 Global Conference. Find out more about this event and register your interest.