Researchers at the University of Glasgow http://gla.ac.uk have developed a precise lip-reading system for wearers of protective masks that paves the way for a new generation of hearing aids. The approach combines radio frequency sensing with artificial intelligence for the first time to identify lip movements. It could, when integrated with conventional hearing aid technology, help combat the “cocktail party effect”. Hearing aid users often only understand “stationary” in a clamour of voices because they cannot identify the directions from which the individual voices are coming.
Privacy at risk

The problem could be solved if cameras were used to assist lip-reading, but collecting video footage of people without their explicit consent raises concerns about individual privacy. Cameras are also unable to read lips through masks, an everyday challenge for people who wear face coverings for cultural or religious purposes, and a broader problem in the age of COVID-19. For the development, researchers asked male and female volunteers to pronounce the five vowel sounds A, E, I, O and U with their lips uncovered and then repeat them while wearing a surgical mask.

In the process, radar sensors, which operate in the high-frequency range and therefore easily penetrate the masks, recorded the faces and lip movements. For comparison, they were also scanned while the lips remained immobile. The 3,600 data samples were used to teach machine learning and deep learning algorithms to recognise characteristic facial movements associated with each vowel sound. The hit rate achieved by the scientists was up to 95 per cent for uncovered lips and 83 per cent for covered lips.