Taking the first steps in business policy experimentation

Embarking on a journey of policy experimentation might be easier with just a small first step…

Keen to encourage a culture of experimentation amongst policy makers, IGL has been examining the barriers that prevent its adoptions. Finding that these include a reluctance to disrupt the status quo, fears of a backlash if ‘lotteries’ are used to allocate support or simply that evaluation is considered too late.

At IGL we believe that these barriers can and should be overcome, but the first step in experimentation doesn’t have to be a giant leap in order to yield valuable insights.

Over the last few years the UK’s Department for Business Energy and Industrial Strategy (BEIS) has launched a number of large scale RCTs testing the economic impact of business advice programmes (eg Growth Vouchers). Its first foray in experimentation was much more modest, and it offers useful lessons on how other governments can begin introducing policy experimentation.1

Even a small experiment can help achieve an important goal…

As part of a wider effort2 to promote business mentoring BEIS (then BIS) launched the ‘Get Mentoring’ campaign, which had the aim of training 15,000 new volunteer mentors by December 2012. Despite lots of people registering to become mentors, relatively few were completing the necessary training.

To rectify this, BEIS decided to test whether behavioural insights could be applied to email reminders sent out to volunteers in order to ‘nudge’ them into completing their training. For the first time in this policy area, RCTs were used to learn what worked best. The process of testing and improving proved to be instrumental in the programme meeting its goal. By the end of December 2012 15,305 people had completed their Get Mentoring training.3 The trials directly generated an additional 778 mentors4 and perhaps more than double5 that once wider application of lessons learnt are included.

…and provide insights that are unlikely to get through from other approaches

The elements that made the messages most effective6 were not those that we expected prior to the trial. For example, one of the most successful was to prime altruism by including the following at the start of the emails:

Adam Smith, the great British social philosopher once said: “To feel much for others, and little for ourselves; to restrain our selfish, and exercise our benevolent affections, constitutes the perfection of human nature”.

The approach also uncovered other interesting differences that are likely to have otherwise gone unknown. For women, the inclusion of messages on anticipated pride only had the same impact as their inclusion for men when combined with an emotional norm (ie others also feel this pride). For men, the addition of emotional norm made no statistically significant difference.

Small experiments could be used to test and improve almost all aspects of programme design and delivery…

BEIS later explored other areas where smaller trials might provide insights to improve policy.

Policy design: Research to inform efforts to raise demand for business mentoring

This research7 found that asking businesses to reflect on their growth ambitions, and then telling them that mentors help to spot growth opportunities, was the most effective approach - increasing both their attitude toward mentoring and motivation to get a mentor.

Raising awareness: Identifying effective ways to raise awareness of policies to support enterprise

A series of trials8 tested simple tweaks to improve engagement with an email newsletter on available support. For example, opening rates increased by 3.9% just by inviting businesses to reflect on their growth ambitions in the subject line (“Realise your hopes and aspirations”, but highlighting that support was “free” completely undermined this. Possibly due to a lack of perceived ‘value from free support or perhaps simply as the term triggered spam filters (human or automated).

Boost applications: For the Growth Vouchers Programme9 trials were run to find ways of encouraging applications.

A common challenge for business support programmes is making sure that the right businesses are aware of the programme and motivated to apply – not easy given demands on their attention and time. As part of the Growth Vouchers Programme (a joint project with the Behavioural Insights Team) trials were conducted to explore how emails could be used to boost applications. Overall the email led to 9,000 additional applications 10 to the Growth Vouchers Programme and provided BEIS with valuable information on how we can most effectively communicate with small firms.

…whilst very beneficial, these rapid trials typically come up short when it comes to proving actual impact – so we recommend they aren’t the only step taken into experimental policy

As outlined above small rapid experiments can provide quick and actionable insights. They will however, typically come up short when it comes to providing evidence of more tangible economic benefits.

Returning to the initial mentoring example, are those mentors ‘nudged’ into completing their training as active and effective as those who would always have completed their training. Similarly, once a business is tipped into applying for a programme by a change to marketing messages will it actually go on and use, implement and benefit from the support to the same extent as other applicants.

This is where fully developed RCTs come into play… but travelling that far (even with the support of IGL) may be easier if start with a small step.

• 1. In the interests of disclosure, I led the analytical team in BEIS responsible for delivering these trials. For many trials BEIS was support by Leandro Galli.
• 2.  Galli (2012), as found in BIS (2012) ’Mentoring Policy – the action plan‘
• 3.  Hudson and Sear (2013), ‘Get Mentoring final evaluation’.
• 4. Galli (2013), as found in BIS (2013), ‘SMEs: The Key Enablers of Business Success and the Economic Rationale for Government Intervention’
• 5.  Galli (2013), as found in BIS (2013), ‘Demand for Mentoring among SMEs’
• 6.  Galli (2013), as found in BIS (2013), ‘SMEs: The Key Enablers of Business Success and the Economic Rationale for Government Intervention’
• 7.  Galli (2013), as found in BIS (2013), ‘Demand for Mentoring among SMEs’
• 8.  Galli (2013) as reported in BIS (2013), ‘SMEs: The Key Enablers of Business Success and the Economic Rationale for Government Intervention’
• 9. The Growth Vouchers Programme was a joint project between BEIS and the Behavioural Insights Team.
• 10.  http://www.behaviouralinsights.co.uk/trial-results/you-have-been-selecte...