A pioneering new system capable of reading lips could herald a new generation of smart hearing aids, according to researchers from the University of Glasgow.

Developed by an international team of engineers and computing scientists, the technology pairs radio-frequency sensing with artificial intelligence to identify lip movements.

Researchers say that when integrated with conventional hearing aid technology, the system could help tackle the ‘cocktail party effect’.

At present, hearing aids support hearing-impaired people by amplifying the ambient sounds around them.

However, in noisy situations or busy environments, this broad spectrum of amplification can make it difficult for users to focus on specific sounds – like conversation with a single person.

Dr Qammer Abbasi of the University of Glasgow’s James Watt School of Engineering is the lead author of the paper.

Commenting on the study, Dr Abbasi said the findings were “highly encouraging” and could support the development of smart hearing aids in the future.

He said: “Around five percent of the world’s population – about 430 million people – have some kind of hearing impairment.

“Hearing aids have provided transformative benefits for many hearing-impaired people. A new generation of technology which collects a wide spectrum of data to augment and enhance the amplification of sound could be another major step in improving hearing-impaired people’s quality of life.”

In a paper published in the journal Nature Communications, the Glasgow Uni-led team outlined how they harnessed cutting-edge sensing technology.

To develop the system, the team asked male and female volunteers to repeat the five vowel sounds – first while unmasked and then later while wearing a surgical mask.

As volunteers repeated the vowel sounds, their faces were scanned using radio-frequency signals from a radar sensor and WiFi transmitter.

Participants’ faces were also scanned while their lips remained still.

In total, the study collected more than 3,600 samples of data, which were then used to teach machine learning and deep learning algorithms on how to identify the tell-tale lip and mouth movements associated with each vowel sound.

Recommended

• Leader Insights | What Scotland can learn from Silicon Valley
• Elizabeth II: A pioneering monarch at the dawn of the digital age
• Fintech Summit | Three days to go!

According to the research team, the system could correctly read volunteers’ lips “most of the time”.

The study found that the machine learning algorithms interpreted WiFi data up to 95% of the time for unmasked lips, and 80% for masked.

Similarly, radar data was interpreted correctly up to 91% without a mask and 83% of the time with a mask.

Dr Abbasi commented: “With this research, we have shown that radio-frequency signals can be used to accurately read vowel sounds on people’s lips, even when their mouths are covered.

“While the results of lip-reading with radar signals are slightly more accurate, the WiFi signals also demonstrated impressive accuracy.

“Given the ubiquity and affordability of WiFi technologies, the results are highly encouraging which suggests that this technique has value both as a standalone technology and as a component in future multimodal hearing aids.”

Get the latest news from DIGIT direct to your inbox

Our newsletter covers the latest technology and IT news from Scotland and beyond, as well as in-depth features and exclusive interviews with leading figures and rising stars.

To subscribe, click here.