After concluding its own internal investigation into the UK-wide disruption to 999 call services on Sunday 25 June, BT — which manages the 999 phone system — has provisionally identified that 11,470 unique calls failed to connect.

According to the telecoms company, a “complex software issue which had never been seen through our continuous testing regime” affected the 999 phone system. The issue had apparently caused a caching issue, resulting in impacted calls not being routed correctly and the user’s call being disconnected.

A “robust temporary fix” is currently in place while a permanent fix for the issue is being tested. The system is said to be stable and running as normal.

The telecoms company also said that it had taken until Wednesday morning to get in touch with the unconnected callers to establish whether further help was needed. In the instances where callers couldn’t be reached, BT has passed details onto the police to investigate.

In general terms of what happened on the technical side last Sunday, BT said that 999 call handling agents began to experience problems with some emergency calls being cut off on connection to the emergency services at 6:24am. By 7:02am, an internal incident was raised.

BT runs three primary network clusters for its 999 phone system, each of which can handle 999 call traffic. However, the cluster which was affected was initially unclear, leading BT to decide to switch to a backup system.

The transfer to the 999 backup system was attempted at 7:31am — however, it had proven unsuccessful. “While the backup system itself was ready to handle calls, the complex transfer process had not been completed successfully. We have since put in place steps to simplify this process,” stated BT.

During attempts to restore the system to full service, BT returned to one of the primary network clusters. But as it became apparent later on, the cluster that had been selected was the one which had the underlying issue. “This resulted in callers being unable to connect to the 999 service between 07:32 and 08:50,” BT said.

The transfer to the backup system was successfully initiated at 8:37am for calls from a landline, and 8:50am for calls from a mobile.

At 11:54am, BT began to bring over non-emergency traffic back to the non-affected primary network clusters, while the impacted cluster was isolated. After extended monitoring, and from 2:52pm, BT started moving emergency calls back to the primary network clusters. By 4:56pm, all emergency calls were being handled by the primary 999 system once again, BT said.

Diagnostics and a temporary fix on the impacted network cluster meant that it was reintroduced at 8:50pm for non-emergency traffic first. Emergency calls were also brought over at 9:29pm after no issues were experienced.

Recommended

• Diverse AI Inspires at its Scottish Launch Event
• “No Confidence”: House of Lords Issues New Digital Exclusion Report
• CMA Pauses Adobe-Figma Deal as it Could ‘Harm UK Digital Economy’

In addition to BT’s internal investigation, Ofcom — the communications regulator — has also launched an investigation.

“Our investigation will seek to establish the facts surrounding the incident and examine whether there are reasonable grounds to believe that BT has failed to comply with its regulatory obligations,” the regulator said on Wednesday.

BT said that it “will fully cooperate with Ofcom’s investigation.”