Researchers at UCSF implanted an array of electrodes over the part of her brain that controls speech, then built software that decoded her attempted speech movements into text, a synthesized voice built from recordings of her before the stroke, and the expressions of a digital facial avatar.
The system decoded her attempted speech at a median rate of 78 words per minute, close to natural conversational speed, with the decoder reaching that performance after less than two weeks of training on her brain signals. The avatar moved its mouth and face in sync with her attempted words as the system spoke them aloud in her own voice.
It was the first system to combine text output, personalized voice synthesis and facial avatar control from the same brain signals at once, rather than restoring just one channel of communication. The word error rate was still around 25 percent, so the system is not yet a flawless translator, and researchers are working to shrink that gap in future versions.
Read the original study
Metzger et al., Nature, 2023 · doi.org/10.1038/s41586-023-06443-4
Her face and voice spoke again, decades later.
Source
- Metzger, S. L., Littlejohn, K. T., Silva, A. B., Moses, D. A., Seaton, M. P., Wang, R., Dougherty, M. E., Liu, J. R., Wu, P., Berger, M. A., Zhuravleva, I., Tu-Chan, A., Ganguly, K., Anumanchipalli, G. K., & Chang, E. F. (2023). A high-performance neuroprosthesis for speech decoding and avatar control. Nature, 620(7976), 1037–1046. https://doi.org/10.1038/s41586-023-06443-4