Posted on by Dr. Francis Collins
Credit: Adapted from Nima Mesgarani, Columbia University’s Zuckerman Institute, New York
Computers have learned to do some amazing things, from beating the world’s ranking chess masters to providing the equivalent of feeling in prosthetic limbs. Now, as heard in this brief audio clip counting from zero to nine, an NIH-supported team has combined innovative speech synthesis technology and artificial intelligence to teach a computer to read a person’s thoughts and translate them into intelligible speech.
Turning brain waves into speech isn’t just fascinating science. It might also prove life changing for people who have lost the ability to speak from conditions such as amyotrophic lateral sclerosis (ALS) or a debilitating stroke.
When people speak or even think about talking, their brains fire off distinctive, but previously poorly decoded, patterns of neural activity. Nima Mesgarani and his team at Columbia University’s Zuckerman Institute, New York, wanted to learn how to decode this neural activity.
Mesgarani and his team started out with a vocoder, a voice synthesizer that produces sounds based on an analysis of speech. It’s the very same technology used by Amazon’s Alexa, Apple’s Siri, or other similar devices to listen and respond appropriately to everyday commands.
As reported in Scientific Reports, the first task was to train a vocoder to produce synthesized sounds in response to brain waves instead of speech . To do it, Mesgarani teamed up with neurosurgeon Ashesh Mehta, Hofstra Northwell School of Medicine, Manhasset, NY, who frequently performs brain mapping in people with epilepsy to pinpoint the sources of seizures before performing surgery to remove them.
In five patients already undergoing brain mapping, the researchers monitored activity in the auditory cortex, where the brain processes sound. The patients listened to recordings of short stories read by four speakers. In the first test, eight different sentences were repeated multiple times. In the next test, participants heard four new speakers repeat numbers from zero to nine.
From these exercises, the researchers reconstructed the words that people heard from their brain activity alone. Then the researchers tried various methods to reproduce intelligible speech from the recorded brain activity. They found it worked best to combine the vocoder technology with a form of computer artificial intelligence known as deep learning.
Deep learning is inspired by how our own brain’s neural networks process information, learning to focus on some details but not others. In deep learning, computers look for patterns in data. As they begin to “see” complex relationships, some connections in the network are strengthened while others are weakened.
In this case, the researchers used the deep learning networks to interpret the sounds produced by the vocoder in response to the brain activity patterns. When the vocoder-produced sounds were processed and “cleaned up” by those neural networks, it made the reconstructed sounds easier for a listener to understand as recognizable words, though this first attempt still sounds pretty robotic.
The researchers will continue testing their system with more complicated words and sentences. They also want to run the same tests on brain activity, comparing what happens when a person speaks or just imagines speaking. They ultimately envision an implant, similar to those already worn by some patients with epilepsy, that will translate a person’s thoughts into spoken words. That might open up all sorts of awkward moments if some of those thoughts weren’t intended for transmission!
Along with recently highlighted new ways to catch irregular heartbeats and cervical cancers, it’s yet another remarkable example of the many ways in which computers and artificial intelligence promise to transform the future of medicine.
 Towards reconstructing intelligible speech from the human auditory cortex. Akbari H, Khalighinejad B, Herrero JL, Mehta AD, Mesgarani N. Sci Rep. 2019 Jan 29;9(1):874.
Advances in Neuroprosthetic Learning and Control. Carmena JM. PLoS Biol. 2013;11(5):e1001561.
Nima Mesgarani (Columbia University, New York)
NIH Support: National Institute on Deafness and Other Communication Disorders; National Institute of Mental Health
Posted on by Dr. Francis Collins
Over the past couple of weeks, we’ve lost two legendary scientists who made major contributions to our world: Sir John Sulston and Stephen Hawking. Although they worked in very different areas of science—biology and physics—both have left us with an enduring legacy through their brilliant work that unlocked fundamental mysteries of life and the universe.
I had the privilege of working closely with John as part of the international Human Genome Project (HGP), a historic endeavor that successfully produced the first reference sequence of the human genetic blueprint nearly 15 years ago, in April 2003. As founding director of the Sanger Centre (now the Sanger Institute) in Cambridge, England, John oversaw the British contributions to this publicly funded effort. Throughout our many planning meetings and sometimes stormy weekly conference calls about progress of this intense and all-consuming enterprise, John stood out for his keen intellect and high ethical standards.
Posted on by Dr. Francis Collins
Matthew Disney grew up in a large family in Baltimore in the 1980s. While his mother worked nights, Disney and his younger brother often tagged along with their father in these pre-Internet days on calls to fix the microfilm machines used to view important records at hospitals, banks, and other places of business. Watching his father take apart the machines made Disney want to work with his hands one day. Seeing his father work tirelessly for the sake of his family also made him want to help others.
Disney found a profession that satisfied both requirements when he fell in love with chemistry as an undergraduate at the University of Maryland, College Park. Now a chemistry professor at The Scripps Research Institute, Jupiter, FL, Disney is applying his hands and brains to develop a treatment strategy that aims to control the progression of a long list of devastating disorders that includes Huntington’s disease, amyotrophic lateral sclerosis (ALS), and various forms of muscular dystrophy.
The 30 or so health conditions on Disney’s list have something in common. They are caused by genetic glitches in which repetitive DNA letters (CAGCAGCAG, for example) in transcribed regions of the genome cause some of the body’s cells and tissues to produce unwieldy messenger RNA molecules that interfere with normal cellular activities, either by binding other intracellular components or serving as templates for the production of toxic proteins.
The diseases on Disney’s list also have often been considered “undruggable,” in part because the compounds capable of disabling the lengthy, disease-causing RNA molecules are generally too large to cross cell membranes. Disney has found an ingenious way around that problem . Instead of delivering the finished drug, he delivers smaller building blocks. He then uses the cell and its own machinery, including the very aberrant RNA molecules he aims to target, as his drug factory to produce those larger compounds.
Disney has received an NIH Director’s 2015 Pioneer Award to develop this innovative drug-delivery strategy further. He will apply his investigational approach initially to treat a common form of muscular dystrophy, first using human cells in culture and then in animal models. Once he gets that working well, he’ll move on to other conditions including ALS.
What’s appealing about Disney’s approach is that it makes it possible to treat disease-affected cells without affecting healthy cells. That’s because his drugs can only be assembled into their active forms in cells after they are templated by those aberrant RNA molecules.
Interestingly, Disney never intended to study human diseases. His lab was set up to study the structure and function of RNA molecules and their interactions with other small molecules. In the process, he stumbled across a small molecule that targets an RNA implicated in a rare form of muscular dystrophy. His niece also has a rare incurable disease, and Disney saw a chance to make a difference for others like her. It’s a healthy reminder that the pursuit of basic scientific questions often can lead to new and unexpectedly important medical discoveries that have the potential to touch the lives of many.
 A toxic RNA catalyzes the in cellulo synthesis of its own inhibitor. Rzuczek SG, Park H, Disney MD. Angew Chem Int Ed Engl. 2014 Oct 6;53(41):10956-10959.
Disney Lab (The Scripps Research Institute, Jupiter, FL)
Disney NIH Project Information (NIH RePORTER)
NIH Support: Common Fund; National Institute of Neurological Disorders and Stroke