The UK government’s M365 Copilot trail found very little evidence of any productivity gains or time savings, though users reported high satisfaction with the AI technology.

The UK Department for Business and Trade trialled Microsoft’s AI assistant offering for three months in 2024 with its employees, aiming to understand the risks and benefits of deploying M365 Copilot across the department.

The trial revealed that only small time savings were observed, with written tasks representing the largest time savings.

Some tasks, however, actually incurred additional time from users, such as scheduling and image generation. Extra time was required due to the tech not being able to produce high enough quality outputs or the extra tasks being created from the availability of the Copilot.

Interestingly, the study found that the small amount of time saved did not lead to improved productivity, with participants not reporting any productivity improvements in the pilot. However, in their own roles they were more likely to report time savings.

Despite this lacklustre productivity gain, 72% of participants said they were satisfied or very satisfied with M356 Copilot during the trial.

However, satisfaction ranged with tasks – users were more likely to be satisfied with the tech when it came to reviewing and summarising information, while they were less impressed with it came to scheduling meetings or producing presentations.

The study also revealed differences in satisfaction among various user groups – neurodiverse colleagues tended to be more satisfied with the AI than their neurotypical colleagues. Non-native English speakers also reported significant benefits, including better wellbeing, enhanced technical abilities, and helping their future career ambitions.

The way people were trained to use the AI Copilot also correlated to their satisfaction levels, with self-led training being more effective in improving satisfaction that formal training led by the department.

The study did reveal a range of issues with the potential rollout of the Microsoft tool – there was a lack of knowledge from some of the employees who were not part of the pilot. The study also identified inconsistencies when it came to quality assuring outputs, with some respondents seeing hallucinations throughout the trial period.

Recommended reading

• 95% of AI Pilots Deliver Zero ROI, Finds MIT Study
• AI Is Driving Data Breaches, IBM Finds
• Organisations Struggle to Cope with Shadow AI

Further, ethical concerns over energy consumption and the environmental impact of large language models limited usage. Users were influenced by their colleagues and line managers hesitancy about the tech and were less likely to use it.

Overall, while satisfaction with the technology may have been reportedly high, the actual time saving capability and productivity gains from M356 Copilot appeared to me lacklustre.