Researchers around the world are constantly producing new discoveries. But for this knowledge to lead to innovation and growth, they often need to work with industry partners. So promoting effective collaboration between researchers and businesses is of critical importance. However, there has been surprisingly little robust testing to optimise and increase the impact of interventions in this space. While the experimental method is used to develop new scientific knowledge, it is ironic that the same method is not being used to create new policy knowledge about what works in advancing that science to society. This is something the Innovation Growth Lab – together with researchers from the Barcelona School of Economics and Esade – sought to change with the ATTRACT NEXT project.
As part of the project and a practical test of the feasibility of using experimental approaches (and randomised experiments in particular) to boost university-industry collaboration, we set up the University-Industry Impact Accelerator. Over six months, teams from the Portuguese Innovation Agency ANI, the applied research institute CCG and a team of researchers from Northumbria, Ulster, and the Manchester Metropolitan (MMU) Universities took part in a structured training programme on the foundations of experimentation. Through a series of seven workshops and one-to-one clinic sessions with IGL researchers, the project teams designed experiments to address key questions or challenges they were facing. The participants developed the essential elements of their experiments, including the problem definition, the theory of change, the details of the experimental design, data collection approaches and risk management strategies. This process culminated in the teams piloting different elements of their experiments.
What did we learn from running the Accelerator? Four takeaways
First, the projects have demonstrated that experimentation in university-industry collaboration is possible. There are numerous untested initiatives in this field, many of which target significant numbers of researchers or businesses, so there are reasonable sample sizes to work with. In addition, there is often good consensus on what success looks like, which helps in coming up with consistent and comparable outcome measures. Since there is little robust evidence about the effects of most interventions, concerns about fairness in being allocated to a specific treatment or control group are also minimised.
Small-scale piloting has also once again proven to be crucial before trying to run a full-scale randomised study. The pilots carried out for the Accelerator have led to important learning about the levels of demand for the interventions in question, the practicalities of implementation, and the details of how to collect data. These insights will all feed into the design of the full-scale experiments and help to make them successful. For example, the team from Northumbria, Ulster and MMU tested a training programme on knowledge exchange for academic researchers. The pilot involved offering this programme to researchers at three universities, which showed that demand was significantly lower than expected. This prompted the implementers to consider how to stimulate more interest in this area before scaling up the offer.
And even if an intervention is only being tested at a small scale – meaning that a full-scale experiment is not yet possible – experimentation is still useful to learn how to optimise the way the intervention is delivered. The CCG project involved an integrated proof-of-concept programme of funding, training and mentoring for researchers to enable them to work on commercialising their outputs. As a new initiative, it was clear that small-scale testing was essential before being gradually scaled up over the coming years. But the limited number of eligible technologies meant that carrying out a randomised evaluation of the overall programme would not be feasible in the near future. Instead, we worked with the implementers to design an experiment assessing the best method of reaching out to potential private-sector partners. This will generate valuable learnings for the specific programme while also allowing the implementing organisation to consider the feasibility of an experiment of the overall programme.
Finally, the Accelerator experience has reinforced that there are important side benefits to taking an experimental approach. Since it is not possible to randomise in retrospect, carrying out a randomised experiment requires thinking through the design, implementation and evaluation of an intervention before running any of it. This prompts organisations to consider ahead of time some elements that can sometimes be omitted in the rush to launch an initiative – such as the theory of change, the key assumptions that are being made, the questions that remain, how to measure the outcomes of the programme, and so on. As we saw in the Accelerator, discussing these elements can have positive consequences for the design and delivery of the programme, on top of facilitating the experiment.
The experimental road ahead
The University–Industry Impact Accelerator project has demonstrated the feasibility and value of experimentation in this area. All three teams were able to set up practical experiments, with only limited support and in a relatively short amount of time – leading to trial designs and the development of internal capabilities within the participating organisations. In each case, the experiments will be valuable not only for the insights they generate directly but also for demonstrating the value of experimentation within the implementing organisations and more widely. Given the success of the format, we have already started applying it to other areas of our work and will continue to do so in the future.
We’re also compiling a Handbook on Experimentation in Science Commercialisation, which will set out a framework of challenges in this policy area and proposals for experiments to address them. The handbook will be published in late 2024: sign up here to be notified when it’s available, and to have the chance to join us at the launch event!
Sara Garcia Arteagoitia
Sara García Arteagoitia joined IGL as a Senior Researcher in 2022. In her role, she helps identify experimental research opportunities with our global government partners and bring them to fruition.
Before joining IGL, Sara worked as a consultant and instructor in behavioural design with private companies and government agencies in Spain and Latin America, helping them improve their structures and programmes to achieve higher impact. Prior to that, she was the Senior Assistant Director of the Center for Decision Research at the University of Chicago, where she worked on developing partnerships and research structures to propel and promulgate experimental behavioural research.
She started her career in experimental research as a research assistant for different projects at the University of Chicago and the Max-Planck Institute for Collective Goods. Sara holds a BA in Political Science and a MA in Political Psychology from the University of Heidelberg, Germany.
Rob Fuller
Rob is the Evaluation Manager at the Innovation Growth Lab.
He is responsible for working with project teams, funders and researchers to deliver high-quality evaluations that can be used to inform policy decisions.
Prior to joining Nesta, Rob worked on randomised controlled trials and quasi-experimental impact evaluations in the international development field. Most recently he worked with an impact investor, AgDevCo, managing the monitoring and evaluation activities of a programme providing support to agricultural SMEs in sub-Saharan Africa. Rob has an MSc in Economics from the University of Edinburgh and a BSc in Mathematics and Statistics from Birkbeck, University of London.