video
From Brain Waves to Real-Time Text Messaging
Posted on by Lawrence Tabak, D.D.S., Ph.D.
For people who have lost the ability to speak due to a severe disability, they want to get the words out. They just can’t physically do it. But in our digital age, there is now a fascinating way to overcome such profound physical limitations. Computers are being taught to decode brain waves as a person tries to speak and then interactively translate them onto a computer screen in real time.
The latest progress, demonstrated in the video above, establishes that it’s quite possible for computers trained with the help of current artificial intelligence (AI) methods to restore a vocabulary of more than a 1,000 words for people with the mental but not physical ability to speak. That covers more than 85 percent of most day-to-day communication in English. With further refinements, the researchers say a 9,000-word vocabulary is well within reach.
The findings published in the journal Nature Communications come from a team led by Edward Chang, University of California, San Francisco [1]. Earlier, Chang and colleagues established that this AI-enabled system could directly decode 50 full words in real time from brain waves alone in a person with paralysis trying to speak [2]. The study is known as BRAVO, short for Brain-computer interface Restoration Of Arm and Voice.
In the latest BRAVO study, the team wanted to figure out how to condense the English language into compact units for easier decoding and expand that 50-word vocabulary. They did it in the same way we all do: by focusing not on complete words, but on the 26-letter alphabet.
The study involved a 36-year-old male with severe limb and vocal paralysis. The team designed a sentence-spelling pipeline for this individual, which enabled him to silently spell out messages using code words corresponding to each of the 26 letters in his head. As he did so, a high-density array of electrodes implanted over the brain’s sensorimotor cortex, part of the cerebral cortex, recorded his brain waves.
A sophisticated system including signal processing, speech detection, word classification, and language modeling then translated those thoughts into coherent words and complete sentences on a computer screen. This so-called speech neuroprosthesis system allows those who have lost their speech to perform roughly the equivalent of text messaging.
Chang’s team put their spelling system to the test first by asking the participant to silently reproduce a sentence displayed on a screen. They then moved on to conversations, in which the participant was asked a question and could answer freely. For instance, as in the video above, when the computer asked, “How are you today?” he responded, “I am very good.” When asked about his favorite time of year, he answered, “summertime.” An attempted hand movement signaled the computer when he was done speaking.
The computer didn’t get it exactly right every time. For instance, in the initial trials with the target sentence, “good morning,” the computer got it exactly right in one case and in another came up with “good for legs.” But, overall, their tests show that their AI device could decode with a high degree of accuracy silently spoken letters to produce sentences from a 1,152-word vocabulary at a speed of about 29 characters per minute.
On average, the spelling system got it wrong 6 percent of the time. That’s really good when you consider how common it is for errors to arise with dictation software or in any text message conversation.
Of course, much more work is needed to test this approach in many more people. They don’t yet know how individual differences or specific medical conditions might affect the outcomes. They suspect that this general approach will work for anyone so long as they remain mentally capable of thinking through and attempting to speak.
They also envision future improvements as part of their BRAVO study. For instance, it may be possible to develop a system capable of more rapid decoding of many commonly used words or phrases. Such a system could then reserve the slower spelling method for other, less common words.
But, as these results clearly demonstrate, this combination of artificial intelligence and silently controlled speech neuroprostheses to restore not just speech but meaningful communication and authentic connection between individuals who’ve lost the ability to speak and their loved ones holds fantastic potential. For that, I say BRAVO.
References:
[1] Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis. Metzger SL, Liu JR, Moses DA, Dougherty ME, Seaton MP, Littlejohn KT, Chartier J, Anumanchipalli GK, Tu-CHan A, Gangly K, Chang, EF. Nature Communications (2022) 13: 6510.
[2] Neuroprosthesis for decoding speech in a paralyzed person with anarthria. Moses DA, Metzger SL, Liu JR, Tu-Chan A, Ganguly K, Chang EF, et al. N Engl J Med. 2021 Jul 15;385(3):217-227.
Links:
Voice, Speech, and Language (National Institute on Deafness and Other Communication Disorders/NIH)
ECoG BMI for Motor and Speech Control (BRAVO) (ClinicalTrials.gov)
Chang Lab (University of California, San Francisco)
NIH Support: National Institute on Deafness and Other Communication Disorders
Immune Macrophages Use Their Own ‘Morse Code’
Posted on by Dr. Francis Collins
In the language of Morse code, the letter “S” is three short sounds and the letter “O” is three longer sounds. Put them together in the right order and you have a cry for help: S.O.S. Now an NIH-funded team of researchers has cracked a comparable code that specialized immune cells called macrophages use to signal and respond to a threat.
In fact, by “listening in” on thousands of macrophages over time, one by one, the researchers have identified not just a lone distress signal, or “word,” but a vocabulary of six words. Their studies show that macrophages use these six words at different times to launch an appropriate response. What’s more, they have evidence that autoimmune conditions can arise when immune cells misuse certain words in this vocabulary. This bad communication can cause them incorrectly to attack substances produced by the immune system itself as if they were a foreign invaders.
The findings, published recently in the journal Immunity, come from a University of California, Los Angeles (UCLA) team led by Alexander Hoffmann and Adewunmi Adelaja. As an example of this language of immunity, the video above shows in both frames many immune macrophages (blue and red). You may need to watch the video four times to see what’s happening (I did). Each time you run the video, focus on one of the highlighted cells (outlined in white or green), and note how its nuclear signal intensity varies over time. That signal intensity is plotted in the rectangular box at the bottom.
The macrophages come from a mouse engineered in such a way that cells throughout its body light up to reveal the internal dynamics of an important immune signaling protein called nuclear NFκB. With the cells illuminated, the researchers could watch, or “listen in,” on this important immune signal within hundreds of individual macrophages over time to attempt to recognize and begin to interpret potentially meaningful patterns.
On the left side, macrophages are responding to an immune activating molecule called TNF. On the right, they’re responding to a bacterial toxin called LPS. While the researchers could listen to hundreds of cells at once, in the video they’ve randomly selected two cells (outlined in white or green) on each side to focus on in this example.
As shown in the box in the lower portion of each frame, the cells didn’t respond in precisely the same way to the same threat, just like two people might pronounce the same word slightly differently. But their responses nevertheless show distinct and recognizable patterns. Each of those distinct patterns could be decomposed into six code words. Together these six code words serve as a previously unrecognized immune language!
Overall, the researchers analyzed how more than 12,000 macrophage cells communicated in response to 27 different immune threats. Based on the possible arrangement of temporal nuclear NFκB dynamics, they then generated a list of more than 900 pattern features that could be potential “code words.”
Using an algorithm developed decades ago for the telecommunications industry, they then monitored which of the potential words showed up reliably when macrophages responded to a particular threatening stimulus, such as a bacterial or viral toxin. This narrowed their list to six specific features, or “words,” that correlated with a particular response.
To confirm that these pattern features contained meaning, the team turned to machine learning. If they taught a computer just those six words, they asked, could it distinguish the external threats to which the computerized cells were responding? The answer was yes.
But what if the computer had five words available, instead of six? The researchers found that the computer made more mistakes in recognizing the stimulus, leading the team to conclude that all six words are indeed needed for reliable cellular communication.
To begin to explore the implications of their findings for understanding autoimmune diseases, the researchers conducted similar studies in macrophages from a mouse model of Sjögren’s syndrome, a systemic condition in which the immune system often misguidedly attacks cells that produce saliva and tears. When they listened in on these cells, they found that they used two of the six words incorrectly. As a result, they activated the wrong responses, causing the body to mistakenly perceive a serious threat and attack itself.
While previous studies have proposed that immune cells employ a language, this is the first to identify words in that language, and to show what can happen when those words are misused. Now that researchers have a list of words, the next step is to figure out their precise definitions and interpretations [2] and, ultimately, how their misuse may be corrected to treat immunological diseases.
References:
[1] Six distinct NFκB signaling codons convey discrete information to distinguish stimuli and enable appropriate macrophage responses. Adelaja A, Taylor B, Sheu KM, Liu Y, Luecke S, Hoffmann A. Immunity. 2021 May 11;54(5):916-930.e7.
[2] NF-κB dynamics determine the stimulus specificity of epigenomic reprogramming in macrophages. Cheng QJ, Ohta S, Sheu KM, Spreafico R, Adelaja A, Taylor B, Hoffmann A. Science. 2021 Jun 18;372(6548):1349-1353.
Links:
Overview of the Immune System (National Institute of Allergy and Infectious Diseases/NIH)
Sjögren’s Syndrome (National Institute of Dental and Craniofacial Research/NIH)
Alexander Hoffmann (UCLA)
NIH Support: National Institute of General Medical Sciences; National Institute of Allergy and Infectious Diseases
Welcoming First Lady Jill Biden to NIH!
Posted on by Dr. Francis Collins
It was wonderful to have First Lady Jill Biden pay a virtual visit to NIH on February 3, 2021, on the eve of World Cancer Day. Dr. Biden joined me, National Cancer Institute (NCI) Director Ned Sharpless, and several NCI scientists to discuss recent advances in fighting cancer. On behalf of the entire NIH community, I thanked the First Lady for her decades of advocacy on behalf of cancer education, prevention, and research. To view the event, go to 53:20 in this video. Credit: Adapted from White House video.
NIH at 80: Sharing a Timeless Message from President Roosevelt
Posted on by Dr. Francis Collins
This Saturday, October 31, marks an important milestone in American public health: the 80th anniversary of President Franklin Delano Roosevelt’s dedication of the campus of the National Institutes of Health (NIH) in Bethesda, MD. The President’s stirring speech, delivered from the steps of NIH’s brand-new Administration Building (now called Building 1), was much more than a ribbon-cutting ceremony. It gave voice to NIH’s commitment to using the power of science “to do infinitely more” for the health of all people with “no distinctions of race, of creed, or of color.”
“We cannot be a strong nation unless we are a healthy nation. And so, we must recruit not only men and materials, but also knowledge and science in the service of national strength,” Roosevelt told the crowd of about 3,000. To get a sense of what it was like to be there on that historic day, I encourage you to check out the archival video footage above from the National Archives and Records Administration (NARA).
These words from our 32nd President are especially worth revisiting for their enduring wisdom during a time of national crisis. In October 1940, with World War II raging overseas, the United States faced the prospect of defending its shores and territories from foreign forces. Yet, at the same time as he was bolstering U.S. military capacity, Roosevelt emphasized that it was also essential to use biomedical research to shore up our nation’s defenses against the threats of infectious disease. In a particularly prescient section of the speech, he said: “Now that we are less than a day by plane from the jungle-type yellow fever of South America, less than two days from the sleeping sickness of equatorial Africa, less than three days from cholera and bubonic plague, the ramparts we watch must be civilian in addition to military.”
Today, in the midst of another national crisis—the COVID-19 pandemic—a similar vision is inspiring the work of NIH. With the aim of defending the health of all populations, we are supporting science to understand the novel coronavirus that causes COVID-19 and to develop tests, treatments, and vaccines for this disease that has already killed more than 225,000 Americans and infected more than 8.6 million.
As part of the dedication ceremony, Roosevelt thanked the Luke and Helen Wilson family for donating their 70-acre estate, “Tree Tops,” to serve as a new home for NIH. (Visitors to Wilson Hall in Building 1 will see portraits of the Wilsons.) Founded in 1887, NIH had previously been housed in a small lab on Staten Island, and then in two cramped lab buildings in downtown Washington, D.C. The move to Bethesda, with NIH’s first six buildings already dotting the landscape as Roosevelt spoke, gave the small agency room to evolve into what today is the world’s largest supporter of biomedical research.
Yet, as FDR gazed out over our fledging campus on that autumn day so long ago, he knew that NIH’s true mission would extend far beyond simply conducting science to providing much-needed hope to humans around the world. As he put it in his closing remarks: “I voice for America and for the stricken world, our hopes, our prayers, our faith, in the power of man’s humanity to man.”
On the 80th anniversary of NIH’s move to Bethesda, I could not agree more. Our science—and our humanity—will get us through this pandemic and show the path forward to brighter days ahead.
Links:
Who We Are: History (NIH)
Office of NIH History and Stetten Museum (NIH)
“70 Acres of Science” (Office of NIH History)
Coronavirus (COVID-19) (NIH)
Next Page
