These face-reading glasses track physical and mental health

Charles Nduka is a plastic surgeon who mends the bodies of his patients with a scalpel and rebuilds their minds with a combination of face-reading technology and virtual reality.

Nduka’s interest in VR dates back more than two decades, to when he was a medical student. He is now co-founder and chief science officer of the Brighton-based startup Emteq, which makes wearable devices that read facial movements to give feedback on muscle activity. In the future, the same technology could be used to track the wearer’s emotions and even monitor their mental health.

Nduka started experimenting with sensor technologies while creating his facial palsy centre at Queen Victoria Hospital, East Grinstead, for the treatment of paralysis. Patients often need to practise facial exercises to improve control over their muscles. But Nduka found that many of his patients were disturbed by looking at their reflection while doing so. He realised that, paradoxically, the lack of reality in virtual reality could help.

In 2015 he set up Emteq with the data security entrepreneur and AI graduate Graeme Cox, funded by a £850,000 grant from the National Institute for Health Research and in collaboration with Phil Breedon, who leads the medical design research group at Nottingham Trent University.

Emteq has developed an insert for VR headsets that can track the wearer’s facial expressions and project them onto a VR avatar in real time. “People with facial paralysis can have limited awareness or control of abnormal facial movements,” explains Nduka. “Without proper feedback, their condition may worsen. By wearing expression-sensing glasses to provide real-time muscle feedback, patients can practise their exercises without having to stare at themselves in the mirror.”

The headset contains 10 electrodes that monitor the minute electrical changes that occur when muscles move in the face. From a flash of anger to a beaming smile, a telltale wave of electrical activity washes over the skin. This is detected using the sensors, and processed in the headset.

Emteq's sensor technology has now been shrunk for use in a pair of glasses to collect facial expression, body movement and voice intonation data from people with Parkinson’s disease. Parkinson’s affects 6 million people worldwide, can take decades to develop and comes in a dozen subtypes. Its symptoms can wax and wane, yet doctors have to make treatment decisions based only on a thin slice of a patient's life. This can make it hard to find the right drug dosage to stop a patient having either frozen muscles or severe tremors.

Nduka’s Parkinson’s project, which is funded by Innovate UK and run in collaboration with the German voice analytics startup audEERING, aims to give doctors a more nuanced, long-term view of the progression of the disease. The first part of the study involves 50 Parkinson’s patients and control group members wearing the glasses in laboratory conditions. A second study in 2019 will see 20 people wear the glasses at home.

Nduka hopes that data gathered could be used to create a “digital phenotype” for Parkinson’s patients. A phenotype is a description of an individual’s observable characteristics and behaviour – similar to how a genotype describes genetic traits. Digital phenotyping uses devices such as smartphones – or, in this case, smart glasses – to collect data and build a profile. Tracking how the digital phenotype changes over time, and in response to medication, could help doctors fine-tune dosages more accurately.

Next Nduka wants to see if tracking facial expressions could also monitor emotion and help to produce a digital phenotype for mental health conditions. Prompted by a personal interest, after the suicide of his brother 18 years ago, he wants to develop phenotyping technology that could track a person’s mental health and flag up early warning signs before they reach a crisis. “We will be looking at depression next,” he says.

Other companies have already started work in this area. The psychiatrist and neuroscientist Tom Insel, co-founder of Mindstrong Health and a former director of the US National Institute of Mental Health, says that smartphones offer the opportunity to gather digital phenotypes as people go about their daily lives. Mindstrong Health is working with Takeda Pharmaceutical in Tokyo to develop digital biomarkers to track mental health conditions including schizophrenia and treatment-resistant depression.

Nduka hopes that digital phenotyping could one day bring a more preventive approach to mental health. If we ran transport like today’s mental health services, he says, “we would spend all our money on accident repair shops and recovery vehicles”.