How can outsiders nudge businesses to make better use of digital technologies? We tested a simple low-cost intervention - providing information about how the performance of their website compared to their peers - and saw companies making clear improvements as a result.
The economic uncertainty and technological complexity faced by businesses were on the rise before the COVID-19 pandemic, but both accelerated further during this challenging time. Digital adoption had been deepening across the entire spectrum of business life in recent years and data-driven decision making had become part and parcel of high-performance businesses. As Marc Benioff - founder and Co-CEO of Salesforce - once put it, “every digital transformation is going to begin and end with the customer, and I can see that in the minds of every CEO I talk to.” The speed at which businesses must now react, and the costs associated with not doing so in this wired world, are what have profoundly changed – businesses now have little time to reflect upon the reactions from customers to this digitally enhanced experience.
Against the backdrop of this pandemic-fuelled digital adoption, policymakers have responded by creating a new range of policies, including the UK government's Help to Grow: Digital, the Scottish Government's DigitalBoost programme, and so forth. But the first-time adoption of new software applications is not the end of the digital journey since businesses must also put in place appropriate management practices to harness the benefits of these new digital solutions.
In this project funded by the ESRC through the PrOPEL Hub, we explore what the key differences were to better manage existing digital assets. To do so we focused on a digital technology that plays a pivotal role in a company’s interaction with its customers – its website, which is often powered by scores of software applications. A business website is a window to a company’s ‘soul’, acting as the second, or increasingly, the only, shopfront it presents to customers.
If companies use web technologies inefficiently, this affects the experience that customers receive and reduces the benefits that companies can realise from their adoption. There are many factors that could lead to inefficient adoption, but information barriers about how best to use the technology and managerial inattention are often cited as the most important.
To test whether improvement in digital performance could be generated by providing more information, in this project, we designed a field experiment involving over 800 firms from the UK distilling and brewing industry. Our treatment was designed to be minimally invasive, delivered in the form of a bespoke Digital Benchmark Report tailored to each company. It contained information on its website performance, which was benchmarked against the average and best performance amongst its industry peers. It also informed companies of the key components of a website performance management system and the need to utilise digital technologies more effectively. To the extent that this was information that the business already had, then our information provided a nudge to re-focus their attention and potentially improve the experience of customers using their website.
Half of all these firms were randomly selected to receive our business support treatment and the other half simply being monitored as a control group, allowing us to quantify how important this information or the reminder was.
Our benchmarking exercise focused on a comparison across seven website performance metrics that are often used by web developers and considered to influence customer experience and ultimately conversions into sales. Five metrics measured the loading time of a website on a digital device and two additional metrics measured search engine optimisation (SEO) which affects the website’s Google ranking.
We found gaps in performance across firms within the baseline data that were substantial. The best performing websites loaded some three to six times faster than the worst and contained 38 times the number of keywords picked up by a search engine such as Google. There were numerous things that businesses with under-performing websites could do to improve. Throughout the trial, our main objective was to observe any behavioural changes amongst businesses in response to the information given to them about how efficiently they had been using technology and their relative ranking.
In a nutshell, we found causal evidence indicating that the information we sent to businesses was important, helping to generate improved website performance amongst UK companies. These effects happened quickly, within a month or so, with further improvements at three months and six months. They also occurred for performance metrics that are difficult to analyse without sound internal monitoring processes, consistent with the absence of a performance management process for digital technologies among UK companies in the baseline period. In particular, the most notable impact was seen in the website speed where our intervention led to a roughly 0.3 seconds increase on the speed index metric within the first month (i.e., a gain of 13% of the standard deviation in the baseline data), with this effect intensifying in three- and six-month time (i.e., a gain of over 20% of the standard deviation). Similar improvement also occurred on the time-to-interactive speed metric. These performance gains were induced in addition to the common improvements seen across the sample during the pandemic period.
Digging deeper we tried to explore some of the mechanisms behind the changes that we helped generate. Overall, we found evidence that the information worked by providing knowledge to some and serving as a reminder to others. We found, for example, evidence that companies improved on dimensions of performance that were not included in the information sent to them. Website loading times can differ according to the type of device they are viewed on, with loading speeds typically much slower on mobile versus desktop devices. We informed businesses of their desktop loading times only but found evidence that mobile loading times also improved amongst treated firms. That is good news and suggests a deeper improvement in management processes from the intervention.
However, whereas the number of keywords remained broadly constant for treated firms, they increased amongst the counterfactual group, such that the effect of the treatment worsened relative performance. These ranking keywords are terms that frequently appear in online searches for their products. The apparent contradiction in the results for keywords with those for speed is consistent with the idea the treatment worked by discouraging additions to the website, such as more text, that might slow it down. But why the improved process suggested by the speed results would not lead the firm to try to improve both is not obvious.
We also surveyed treated firms to ask why they adjusted their website in response to the information we sent. About half of the firms gave answers which suggested an increase in monitoring and frequency of tasks to actively improve performance, but the other half suggested it simply reminded them of the need to act – in part because they had performed below average.
How much of the effect of the treatment is due to a lack of knowledge about what superior performance management processes might include - and how much is simply due to managers not paying sufficient attention - is important, as they suggest different policy responses. If it is about knowledge, then this would suggest management training could play a key role. If it is about inattention, then a periodic reminder might well suffice. We will have to leave this assessment for future work.
Overall, our project answers why it is important for businesses to look beyond the adoption of digital technologies to find their digital ‘souls’ and manage them better to appropriate such benefits. The digital adaptation that has taken place will require significant experimentation and learning by managers. But it also affords a new opportunity to build more robust internal systems for managing digital assets by reflecting on best managerial practices for enabling these transitions.
In addition to providing causal evidence on what works in influencing companies to extract more value from existing investments in digital technologies, this project also sheds light on which information mechanisms are most important in given contexts. Insights from this trial can thus effectively inform more targeted mechanism policies to promote better management of digital technologies. In our ongoing work, we will continue to explore what sort of information feedback is most effective in nudging companies to improve, and whether companies already collecting performance data respond differently to future interventions vis-à-vis those not capable of self-monitoring.
Cher Li
