A new study from Glasgow University has explored how computers can be more accurately controlled using facial expressions, and could help make augmented and virtual tech more accessible to people living with disabilities.

Working with researchers from the University of St. Gallen in Switzerland, the Glasgow team discovered that seven simple facial movements could be reliably recognised by Meta’s Quest Pro headset, enabling users to control a virtual reality (VR) game, as well as navigate web pages in an augmented reality (AR) environment.

The results could allow those living with disabilities new ways to interact with computers using widely available VR headsets, as well as broaden the options for hands-free access to virtual environments for every user.

During the study, the researchers asked 20 non-disabled volunteers to move their faces to match the fifty-three expressions already recognised by the Quest Pro’s software.

These expressions, called Facial Action Units or FAUs, are picked up by the headset’s onboard cameras to translate real-world facial expressions into virtual ones in online environments.

Following testing, seven FAUs were found to provide the optimum balance between being reliably recognised by the headset and being comfortable for users to regularly repeat, including gestures like opening the mouth, squinting the eyes, and pulling at the edges of the mouth.

The researchers then built a neural network model to read these expressions captured by the Quest Pro with 97% accuracy, with volunteers asked to use only facial expressions to turn, select options and shoot weapons in a VR game, then use their faces to scroll, select and navigate web pages in AR. 

“With this study, we were keen to explore whether the functions of a commercially-available VR headset could be adapted to help users accurately control software using only their face,” said Dr Graham Wilson, of the University of Glasgow’s School of Computing Science.

“Some have suggested that VR is an inherently ableist technology because it often requires users to perform dextrous hand motions with controllers or make full-body movements which not everyone can perform comfortably. 

“That means that, at the moment, there are barriers preventing widespread access to VR and AR experiences.”

Recommended reading

• UK Gov Funds VR, AI Tech Projects For Reducing Drug Deaths
• Scots Developers Create VR Experience For Football Fans With Dementia
• Scotland’s Emergency Workers To Get AI-Powered Trauma App

According to the university, the potential applications of the study go much further than expanding accessibility, potentially allowing for users to control AR glasses while walking with hands full, while cooking, or performing tasks requiring clean hands like surgery. 

These VR face controls could also help users more discreet control of devices when in public spaces, and help in the development of more intuitive tech requiring less physical input.

“We plan to work with people with disabilities like motor impairments or muscular disorders in further studies to provide developers and XR platforms with new suggestions of how they can expand their palette of accessibility options, and help them break free of the constraints of hand and controller-based input,” said Dr Mark McGill, co-author of the paper and also from Glasogw’s School of Computing Science.