Direct Speech Reconstruction from Sensorimotor Brain Activity with Optimized Deep Learning Models [article]

Julia Berezutskaya, Zachary V. Freudenburg, Mariska J. Vansteensel, Erik J. Aarnoutse, Nick F. Ramsey, Marcel A. J. van Gerven
2022 bioRxiv   pre-print
Development of brain-computer interface (BCI) technology is key for enabling communication in individuals who have lost the faculty of speech due to severe motor paralysis. A BCI control strategy that is gaining attention employs speech decoding from neural data. Recent studies have shown that a combination of direct neural recordings and advanced computational models can provide promising results. Understanding which decoding strategies deliver best and directly applicable results is crucial
more » ... r advancing the field. In this paper, we optimized and validated a decoding approach based on speech reconstruction directly from high-density electrocorticography recordings from sensorimotor cortex during a speech production task. We show that 1) dedicated machine learning optimization of reconstruction models is key for achieving the best reconstruction performance; 2) individual word decoding in reconstructed speech achieves 92-100\% accuracy (chance level is 8\%); 3) direct reconstruction from sensorimotor brain activity produces intelligible speech. These results underline the need for model optimization in achieving best speech decoding results and highlight the potential that reconstruction-based speech decoding from sensorimotor cortex can offer for development of next-generation BCI technology for communication.
doi:10.1101/2022.08.02.502503 fatcat:oz327qpv4zeefkrs26pleveaau