In 2018, the European Commission introduced a new EU Horizon 2020 programme - INNOSUP-06-2018 - to encourage innovation agencies across Europe to experiment with their policy programmes. Here at the Innovation Growth Lab, we’ve been supporting both the EU and innovation agencies to succeed. This piece explores the journey of a representative from Innovate UK’s Economic and Insights team, who is currently partaking in the INNOSUP programme.
We recently sat down with a representative from Innovate UK’s Economics and Insights team, and discussed their EU funded project ‘RCT4MANU: Testing an innovative support scheme for manufacturing SMEs and accelerating the use of RCTs in innovation agencies’. Innovate UK co-designed this trial with Innovate UK KTN, with the aim of evaluating the effectiveness and impact of the 4Manufacturing tool - an innovative support programme designed by Innovate UK KTN, based on diagnosis and one-to-one consultant advice, which aims to accelerate the adoption of industrial digital technologies for manufacturing SMEs.
The need for more evidence
The intervention sounds straightforward: Manufacturing businesses interested in understanding the potential of digital technologies through one-to-one business support from an Innovate UK KTN advisor were to sign up for a 1-2 day session. In this session, the advisor would use the 4Manufacturing framework to help the business identify the challenges they face and how best to start tackling them, or whether to take a step back and address an entirely different but more pressing issue. The advisor would then recommend technology areas to explore.
With a treatment group receiving the intervention first, and a control group receiving the same intervention a year later, the team were hoping to see SMEs adopt new technologies that would lead to either greater productivity or environmental sustainability. As a member of the project team explained, though experimentation has been an ambition of Innovate UK, historically, evaluations were planned as an afterthought, making it difficult to draw robust conclusions. ‘You’re under constant pressure to deliver things faster and better’, they explained, ‘there isn’t really the appetite to do experimentation because it adds costs.’ The INNOSUP programme, however, gave the RCT4MANU project team an opportunity to be deliberate about their expectations and outcome measures from the outset.
From theory to practice
Starting work in September 2019, the team embraced the importance of upfront planning and dedicated many months to perfecting the intervention and planning the design of the trial in detail, considering what options to test and whether to perhaps test more than one type of intervention. Despite their careful, thought-out approach, they still encountered later issues; some of these were due to unprecedented circumstances and others were roadblocks the team may have hit regardless - the theory may have been sound but the policy plumbing hadn’t been tested.
Dealing with the unexpected: Covid-19 strikes
With six months upfront planning up their sleeve, the project team were ready to launch in March 2020, and unfortunately, they hit their first unexpected wall: the onset of the Covid-19 pandemic. As a result, the team faced their first tough decision: Was it worth continuing as planned or delaying the intervention until the impacts of the pandemic became clearer? With so many businesses having to suspend activity at the time, the decision was made to pause the intervention for six months to wait and see what would happen. The team tentatively picked back up where they had left off in the autumn of 2020. Recruitment began in November in the hope that they would have around 350 sign ups by late December and could begin implementation in early January 2021.
Unfortunately, yet again, things did not run as smoothly as planned. With the Covid pandemic escalating, and with regional lockdowns in the UK shifting towards a full national lockdown, the project team did not manage to recruit as many businesses as they had initially hoped. As a member of the project team explained, their ability to sign up only 90 businesses was not just down to Covid, but also partly to do with some of their assumptions when approaching recruitment. They initially felt confident in relying heavily upon their existing contact list and then only extended their approach later when the results of this tactic were disappointing. Looking back now, the Innovate UK representative acknowledges that this overestimation of how readily the targeted businesses would respond to their offer was a part of the problem.
Navigating the unexpected
As a result of all these issues combined, including changes coming for many of their targeted manufacturers as a result of Brexit, the team had to go out to different populations in order to get more businesses to register their interest, and had to keep their baseline survey open for longer than expected as businesses were not readily completing this part of the process. Additionally, they realised with a new research partner on board that they needed further ethics approval; this meant that they had to retroactively ask businesses for their consent, again fuelling further drop outs.The outcome of these changes was that the team faced their second big hurdle. Since they were waiting until recruitment was complete to begin implementing the programme: the longer they extended their recruitment process, the more of those that had already signed up dropped out. The catch-22 of extending recruitment can be that by drawing out the time spent trying to bring in new sign-ups, you lose the attention and the interest of those who joined at the start.
The team realised that they faced a constant trade off between how they wanted to achieve the best research results possible that demanded a larger sample, and what was practically achievable. In the end, they stuck by their original plans not to keep recruitment open any longer and get ready to start delivering support between January and April 2021. They hoped that in doing so, they would be able to prioritise and hold on to their initial sign ups, and not delay the treatment group’s support too much. At this point, there was one final negative impact from Covid: while the post-lockdown recovery was good news for their recruited businesses, many of them were now too busy to take up support.
The effects of Covid, however, were not all negative. Initially, with RCT4MANU being a versatile framework, the team had expected to scale up the project through other parties (e.g. Local Enterprise Partnerships and those with business support knowledge). However, due to the lower than expected sample size, Innovate UK KTN ended up delivering the intervention entirely in-house. Though they weren’t able to test the scalability of the framework in the same way, this gave them greater fidelity and consistency of delivery, and ultimately more control over how it was delivered. This also allowed them to develop a detailed understanding of implementation that is useful when they talk to other parties interested in adopting the 4Manufacturing framework.
What can be learned?
Ultimately, the onset of Covid meant that despite the team’s considered trial design, many aspects of this intervention could not be planned for, and many tough decisions had to be made when it came to prioritisation and holding on to recruited businesses.
It is hard to draw a fixed line, however, between impacts caused by Covid, and those which may have cropped up regardless. As an Innovate UK team member points out, some of the challenges that the Innovate UK and Innovate UK KTN team faced were around recruitment and the oversight they had that businesses do not always want to wait around too long for support - an assumption which formulated the basis of their trial design. Whilst Covid certainly contributed to the willingness of businesses to sign up for support, as they faced graver issues at the time, it certainly wasn’t the only reason that the team faced recruitment challenges. Ultimately, those working on the RCT4MANU project learned a similar lesson to those in the KEPA team: piloting is always a good idea in order to test out the weaker parts of your trial design and implementation plans before you scale up.
Despite these challenges, the team representative that we spoke to was a proponent of the overall experimentation journey.
‘What we’ve learned through the process has been really valuable’, they shared, as for one, ‘there are so many elements of the RCT development that you can apply outside of an RCT context to properly evaluate your programmes’.
Overall though, they remain keen on continuing with an experimental approach but note that using an RCT may not always be the ideal approach, as they added that ‘a well run RCT [can be] the gold standard, not a poorly run one’. By instilling best practice from the beginning, and adopting a rigorous evaluative mindset to the way you make experimental decisions, you are not only less likely to run into hurdles along the way, but you may be better prepared to navigate the unforeseen circumstances which may crop up.