The face of the digital avatar was created using the same animation technology as in The Last of Us II.
Researchers at the University of California, San Francisco and the University of California, Berkeley, in partnership with Edinburgh-based Speech Graphics, have developed a pioneering communication system that allows a woman paralyzed by a stroke to communicate through a digital avatar she controls using a brain-computer interface (BCI).
BCI devices allow you to track analog signals created by the brain and convert them into digital signals that can be understood by a computer.
Researchers at the Department of Neurological Surgery implanted a 253-pin array of electrodes in the patient’s speech center, which monitors and records signals that normally drive the muscles of the jaw, lips and tongue – and in this case, transmit them through a cable port in her skull to a computer. (The computing stack is powered by artificial intelligence, which spent several weeks learning the patient’s electrical signal patterns in more than 1000 words.)
Using the interface, the patient can write her answers or speak using an artificial voice trained on recordings of her own made before her paralysis.
The rest of the work (creating the digital avatar) was done by Speech Graphics, the same company that developed the photorealistic facial animation technology for Halo Infinite and The Last of Us Part II. The tool extracts the necessary facial movements based on the analysis of the incoming audio, and then transmits this data in real time to the game engine to animate them into an avatar without delay.
“The creation of a digital avatar that can speak, express emotions, and articulate in real time, connected directly to the subject’s brain, demonstrates the potential for using AI-controlled faces far beyond video games,” said Michael Berger, CTO and co-founder of Speech Graphics.
BCI technology was introduced in the early 1970s and slowly developed in the following decades. However, advances in data processing and computing systems have helped revitalize the industry, and several startups are now vying to be the first to go through the U.S. Food and Drug Administration (FDA) approval process.
Brooklyn-based Synchron made headlines last year when it became the first company to successfully implant a BCI in a patient. Elon Musk’s Neuralink received approval for human trials earlier this year after it was revealed that the company had killed dozens of test animals in previous rounds of testing.
