Skip to main content

AI

From Brain Waves to Real-Time Text Messaging

Posted on by

For people who have lost the ability to speak due to a severe disability, they want to get the words out. They just can’t physically do it. But in our digital age, there is now a fascinating way to overcome such profound physical limitations. Computers are being taught to decode brain waves as a person tries to speak and then interactively translate them onto a computer screen in real time.

The latest progress, demonstrated in the video above, establishes that it’s quite possible for computers trained with the help of current artificial intelligence (AI) methods to restore a vocabulary of more than a 1,000 words for people with the mental but not physical ability to speak. That covers more than 85 percent of most day-to-day communication in English. With further refinements, the researchers say a 9,000-word vocabulary is well within reach.

The findings published in the journal Nature Communications come from a team led by Edward Chang, University of California, San Francisco [1]. Earlier, Chang and colleagues established that this AI-enabled system could directly decode 50 full words in real time from brain waves alone in a person with paralysis trying to speak [2]. The study is known as BRAVO, short for Brain-computer interface Restoration Of Arm and Voice.

In the latest BRAVO study, the team wanted to figure out how to condense the English language into compact units for easier decoding and expand that 50-word vocabulary. They did it in the same way we all do: by focusing not on complete words, but on the 26-letter alphabet.

The study involved a 36-year-old male with severe limb and vocal paralysis. The team designed a sentence-spelling pipeline for this individual, which enabled him to silently spell out messages using code words corresponding to each of the 26 letters in his head. As he did so, a high-density array of electrodes implanted over the brain’s sensorimotor cortex, part of the cerebral cortex, recorded his brain waves.

A sophisticated system including signal processing, speech detection, word classification, and language modeling then translated those thoughts into coherent words and complete sentences on a computer screen. This so-called speech neuroprosthesis system allows those who have lost their speech to perform roughly the equivalent of text messaging.

Chang’s team put their spelling system to the test first by asking the participant to silently reproduce a sentence displayed on a screen. They then moved on to conversations, in which the participant was asked a question and could answer freely. For instance, as in the video above, when the computer asked, “How are you today?” he responded, “I am very good.” When asked about his favorite time of year, he answered, “summertime.” An attempted hand movement signaled the computer when he was done speaking.

The computer didn’t get it exactly right every time. For instance, in the initial trials with the target sentence, “good morning,” the computer got it exactly right in one case and in another came up with “good for legs.” But, overall, their tests show that their AI device could decode with a high degree of accuracy silently spoken letters to produce sentences from a 1,152-word vocabulary at a speed of about 29 characters per minute.

On average, the spelling system got it wrong 6 percent of the time. That’s really good when you consider how common it is for errors to arise with dictation software or in any text message conversation.

Of course, much more work is needed to test this approach in many more people. They don’t yet know how individual differences or specific medical conditions might affect the outcomes. They suspect that this general approach will work for anyone so long as they remain mentally capable of thinking through and attempting to speak.

They also envision future improvements as part of their BRAVO study. For instance, it may be possible to develop a system capable of more rapid decoding of many commonly used words or phrases. Such a system could then reserve the slower spelling method for other, less common words.

But, as these results clearly demonstrate, this combination of artificial intelligence and silently controlled speech neuroprostheses to restore not just speech but meaningful communication and authentic connection between individuals who’ve lost the ability to speak and their loved ones holds fantastic potential. For that, I say BRAVO.

References:

[1] Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis. Metzger SL, Liu JR, Moses DA, Dougherty ME, Seaton MP, Littlejohn KT, Chartier J, Anumanchipalli GK, Tu-CHan A, Gangly K, Chang, EF. Nature Communications (2022) 13: 6510.

[2] Neuroprosthesis for decoding speech in a paralyzed person with anarthria. Moses DA, Metzger SL, Liu JR, Tu-Chan A, Ganguly K, Chang EF, et al. N Engl J Med. 2021 Jul 15;385(3):217-227.

Links:

Voice, Speech, and Language (National Institute on Deafness and Other Communication Disorders/NIH)

ECoG BMI for Motor and Speech Control (BRAVO) (ClinicalTrials.gov)

Chang Lab (University of California, San Francisco)

NIH Support: National Institute on Deafness and Other Communication Disorders


National Library of Medicine Helps Lead the Way in AI Research

Posted on by

NIH, National Library of Medicine. The earth surrounded by a ring of data
Credit: National Library of Medicine, NIH

Did you know that the NIH’s National Library of Medicine (NLM) has been serving science and society since 1836? From its humble beginning as a small collection of books in the library of the U.S. Army Surgeon General’s office, NLM has grown not only to become the world’s largest biomedical library, but a leader in biomedical informatics and computational health data science research.

Think of NLM as a door through which you pass to connect with health data, literature, medical and scientific information, expertise, and sophisticated mathematical models or images that describe a clinical problem. This intersection of information, people, and technology allows NLM to foster discovery. NLM does so by ensuring that scientists, clinicians, librarians, patients, and the public have access to biomedical information 24 hours a day, 7 days a week.

The NLM also supports two research efforts: the Division of Extramural Programs (EP) and Intramural Research Program (IRP). Both programs are accelerating advances in biomedical informatics, data science, computational biology, and computational health. One of EP’s notable investments is focused on advancing artificial intelligence (AI) methods and reimagining how health care is delivered with the power of AI.

How to teach machines, showing for different piles of pills.
Credit: National Library of Medicine, NIH

With support from NLM, Corey Lester and his colleagues at the University of Michigan College of Pharmacy, Ann Arbor, MI, are using AI to assist in pill verification, a standard procedure in pharmacies across the land. They want to help pharmacists avoid dangerous and costly dispensing errors. To do so, Lester is using AI to develop a real-time computer vision model. It views pills inside of a medication bottle, accurately identifies them, and determines that they are the correct or incorrect contents.

The IRP develops and applies computational methods and approaches to a broad range of information problems in biology, biomedicine, and human health. The IRP also offers intramural training opportunities and supports other training aimed at pre-baccalaureate to postdoctoral students and professionals.

The NLM principal investigators use biological data to advance computer algorithms and connect relationships between any level of biological organization and health conditions. They also use computational health sciences to focus on clinical information processing and analyze clinical data, assess clinical outcomes, and set health data standards.

Four chest x-rays
Credit: National Library of Medicine, NIH

NLM investigator Sameer Antani is collaborating with researchers in other NIH institutes to explore how AI can help us understand oral cancer, echocardiography, and pediatric tuberculosis. His research also is examining how images can be mined for data to predict the causes and outcomes of conditions. Examples of Antani’s work can be found in mobile radiology vehicles, which allow professionals to take chest X-rays (right) and screen for HIV and tuberculosis using software containing algorithms developed in his lab.

For AI to have its full impact, more algorithms and approaches that harness the power of data are needed. That’s why NLM supports hundreds of other intramural and extramural scientists who are addressing challenging health and biomedical problems. The NLM-funded research is focused on how AI can help people stay healthy through early disease detection, disease management, and clinical and treatment decision-making—all leading to the ultimate goal of helping people live healthier and happier lives.

The NLM is proud to lead the way in the use of AI to accelerate discovery and transform health care. Want to learn more? Follow me on Twitter. Or, you can follow my blog, NLM Musings from the Mezzanine and receive periodic NLM research updates.

I would like to thank Valerie Florance, Acting Scientific Director of NLM IRP, and Richard Palmer, Acting Director of NLM Division of EP, for their assistance with this post.

Links:

National Library of Medicine (National Library of Medicine/NIH)

Video: Using Machine Intelligence to Prevent Medication Dispensing Errors (NLM Funding Spotlight)

Video: Sameer Antani and Artificial Intelligence (NLM)

NLM Division of Extramural Programs (NLM)

NLM Intramural Research Program (NLM)

NLM Intramural Training Opportunities (NLM)

Principal Investigators (NLM)

NLM Musings from the Mezzanine (NLM)

Note: Dr. Lawrence Tabak, who performs the duties of the NIH Director, has asked the heads of NIH’s Institutes and Centers (ICs) to contribute occasional guest posts to the blog to highlight some of the interesting science that they support and conduct. This is the 20th in the series of NIH IC guest posts that will run until a new permanent NIH director is in place.


Understanding Long-Term COVID-19 Symptoms and Enhancing Recovery

Posted on by

RECOVER: Researching COVID to Enhance Recovery. An Initiative Funded by the National Institutes of Health

We are in the third year of the COVID-19 pandemic, and across the world, most restrictions have lifted, and society is trying to get back to “normal.” But for many people—potentially millions globally—there is no getting back to normal just yet.

They are still living with the long-term effects of a COVID-19 infection, known as the post-acute sequelae of SARS-CoV-2 infection (PASC), including Long COVID. These people continue to experience debilitating fatigue, shortness of breath, pain, difficulty sleeping, racing heart rate, exercise intolerance, gastrointestinal and other symptoms, as well as cognitive problems that make it difficult to perform at work or school.

This is a public health issue that is in desperate need of answers. Research is essential to address the many puzzling aspects of Long COVID and guide us to effective responses that protect the nation’s long-term health.

For the past two years, NIH’s National Heart, Lung, and Blood Institute (NHLBI), the National Institute of Allergy and Infectious Diseases (NIAID), and my National Institute of Neurological Disorders and Stroke (NINDS) along with several other NIH institutes and the office of the NIH Director, have been leading NIH’s Researching COVID to Enhance Recovery (RECOVER) initiative, a national research program to understand PASC.

The initiative studies core questions such as why COVID-19 infections can have lingering effects, why new symptoms may develop, and what is the impact of SARS-CoV-2, the virus that causes COVID-19, on other diseases and conditions? Answering these fundamental questions will help to determine the underlying biologic basis of Long COVID. The answers will also help to tell us who is at risk for Long COVID and identify therapies to prevent or treat the condition.

The RECOVER initiative’s wide scope of research is also unprecedented. It is needed because Long COVID is so complex, and history indicates that similar post infectious conditions have defied definitive explanation or effective treatment. Indeed, those experiencing Long COVID report varying symptoms, making it highly unlikely that a single therapy will work for everyone, underscoring the need to pursue multiple therapeutic strategies.

To understand Long COVID fully, hundreds of RECOVER investigators are recruiting more than 17,000 adults (including pregnant people) and more than 18,000 children to take part in cohort studies. Hundreds of enrolling sites have been set up across the country. An autopsy research cohort will also provide further insight into how COVID-19 affects the body’s organs and tissues.

In addition, researchers will analyze electronic health records from millions of people to understand how Long COVID and its symptoms change over time. The RECOVER initiative is also utilizing consistent research protocols across all the study sites. The protocols have been carefully developed with input from patients and advocates, and they are designed to allow for consistent data collection, improve data sharing, and help to accelerate the pace of research.

From the very beginning, people suffering from Long COVID have been our partners in RECOVER. Patients and advocates have contributed important perspectives and provided valuable input into the master protocols and research plans.

Now, with RECOVER underway, individuals with Long COVID, their caregivers, and community members continue to serve a critical role in the Initiative. The National Community Engagement Group (NCEG) has been established to make certain that RECOVER meets the needs of all people affected by Long COVID. The RECOVER Patient and Community Engagement Strategy outlines all the approaches that RECOVER is using to engage with and gather input from individuals impacted by Long COVID.

The NIH recently made more than 40 awards to improve understanding of the underlying biology and pathology of Long COVID. There have already been several important findings published by RECOVER scientists.

For example, in a recent study published in the journal Lancet Digital Health, RECOVER investigators used machine learning to comb through electronic health records to look for signals that may predict whether someone has Long COVID [1]. As new findings, tools, and technologies continue to emerge that help advance our knowledge of the condition, the RECOVER Research Review (R3) Seminar Series will provide a forum for researchers and our partners with up-to-date information about Long COVID research.

It is important to note that post-viral conditions are not a new concept. Many, but not all, of the symptoms reported in Long COVID, including fatigue, post-exertional malaise, chronic musculoskeletal pain, sleep disorders, postural orthostatic tachycardia (POTS), and cognitive issues, overlap with myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS).

ME/CFS is a serious disease that can occur following infection and make people profoundly sick for decades. Like Long COVID, ME/CFS is a heterogenous condition that does not affect everybody in the same way, and the knowledge gained through research on Long COVID may also positively impact the understanding, treatment, and prevention of POTS, ME/CFS, and other chronic diseases.

Unlike other post-viral conditions, people who experience Long COVID were all infected by the same virus—albeit different variants—at a similar point in time. This creates a unique opportunity for RECOVER researchers to study post-viral conditions in real-time.

The opportunity enables scientists to study many people simultaneously while they are still infected to monitor their progress and recovery, and to try to understand why some individuals develop ongoing symptoms. A better understanding of the transition from acute to chronic disease may offer an opportunity to intervene, identify who is at risk of the transition, and develop therapies for people who experience symptoms long after the acute infection has resolved.

The RECOVER initiative will soon announce clinical trials, leveraging data from clinicians and patients in which symptom clusters were identified and can be targeted by various interventions. These trials will investigate therapies that are indicated for other non-COVID conditions and novel treatments for Long COVID.

Through extensive collaboration across the multiple NIH institutes and offices that contribute to the RECOVER effort, our hope is critical answers will emerge soon. These answers will help us to recognize the full range of outcomes and needs resulting from PASC and, most important, enable many people to make a full recovery from COVID-19. We are indebted to the over 10,000 subjects who have already enrolled in RECOVER. Their contributions and the hard work of the RECOVER investigators offer hope for the future to the millions still suffering from the pandemic.

Reference:

[1] Identifying who has long COVID in the USA: a machine learning approach using N3C data. Pfaff ER, Girvin AT, Bennett TD, Bhatia A, Brooks IM, Deer RR, Dekermanjian JP, Jolley SE, Kahn MG, Kostka K, McMurry JA, Moffitt R, Walden A, Chute CG, Haendel MA; N3C Consortium. Lancet Digit Health. 2022 Jul;4(7):e532-e541.

Links:

COVID-19 Research (NIH)

Long COVID (NIH)

RECOVER: Researching COVID to Enhance Recovery (NIH)

NIH builds large nationwide study population of tens of thousands to support research on long-term effects of COVID-19,” NIH News Release, September 15, 2021.

Director’s Messages (National Institute of Neurological Disorders and Stroke/NIH)

Note: Dr. Lawrence Tabak, who performs the duties of the NIH Director, has asked the heads of NIH’s Institutes and Centers (ICs) to contribute occasional guest posts to the blog to highlight some of the interesting science that they support and conduct. This is the 18th in the series of NIH IC guest posts that will run until a new permanent NIH director is in place.


Using AI to Find New Antibiotics Still a Work in Progress

Posted on by

Protein over a computer network

Each year, more than 2.8 million people in the United States develop bacterial infections that don’t respond to treatment and sometimes turn life-threatening [1]. Their infections are antibiotic-resistant, meaning the bacteria have changed in ways that allow them to withstand our current widely used arsenal of antibiotics. It’s a serious and growing health-care problem here and around the world. To fight back, doctors desperately need new antibiotics, including novel classes of drugs that bacteria haven’t seen and developed ways to resist.

Developing new antibiotics, however, involves much time, research, and expense. It’s also fraught with false leads. That’s why some researchers have turned to harnessing the predictive power of artificial intelligence (AI) in hopes of selecting the most promising leads faster and with greater precision.

It’s a potentially paradigm-shifting development in drug discovery, and a recent NIH-funded study, published in the journal Molecular Systems Biology, demonstrates AI’s potential to streamline the process of selecting future antibiotics [2]. The results are also a bit sobering. They highlight the current limitations of one promising AI approach, showing that further refinement will still be needed to maximize its predictive capabilities.

These findings come from the lab of James Collins, Massachusetts Institute of Technology (MIT), Cambridge, and his recently launched Antibiotics-AI Project. His audacious goal is to develop seven new classes of antibiotics to treat seven of the world’s deadliest bacterial pathogens in just seven years. What makes this project so bold is that only two new classes of antibiotics have reached the market in the last 50 years!

In the latest study, Collins and his team looked to an AI program called AlphaFold2 [3]. The name might ring a bell. AlphaFold’s AI-powered ability to predict protein structures was a finalist in Science Magazine’s 2020 Breakthrough of the Year. In fact, AlphaFold has been used already to predict the structures of more than 200 million proteins, or almost every known protein on the planet [4].

AlphaFold employs a deep learning approach that can predict most protein structures from their amino acid sequences about as well as more costly and time-consuming protein-mapping techniques.
In the deep learning models used to predict protein structure, computers are “trained” on existing data. As computers “learn” to understand complex relationships within the training material, they develop a model that can then be applied for making predictions of 3D protein structures from linear amino acid sequences without relying on new experiments in the lab.

Collins and his team hoped to combine AlphaFold with computer simulations commonly used in drug discovery as a way to predict interactions between essential bacterial proteins and antibacterial compounds. If it worked, researchers could then conduct virtual rapid screens of millions of new synthetic drug compounds targeting key bacterial proteins that existing antibiotics don’t. It would also enable the rapid development of antibiotics that work in novel ways, exactly what doctors need to treat antibiotic-resistant infections.

To test the strategy, Collins and his team focused first on the predicted structures of 296 essential proteins from the Escherichia coli bacterium as well as 218 antibacterial compounds. Their computer simulations then predicted how strongly any two molecules (essential protein and antibacterial) would bind together based on their shapes and physical properties.

It turned out that screening many antibacterial compounds against many potential targets in E. coli led to inaccurate predictions. For example, when comparing their computational predictions with actual interactions for 12 essential proteins measured in the lab, they found that their simulated model had about a 50:50 chance of being right. In other words, it couldn’t identify true interactions between drugs and proteins any better than random guessing.

They suspect one reason for their model’s poor performance is that the protein structures used to train the computer are fixed, not flexible and shifting physical configurations as happens in real life. To improve their success rate, they ran their predictions through additional machine-learning models that had been trained on data to help them “learn” how proteins and other molecules reconfigure themselves and interact. While this souped-up model got somewhat better results, the researchers report that they still aren’t good enough to identify promising new drugs and their protein targets.

What now? In future studies, the Collins lab will continue to incorporate and train the computers on even more biochemical and biophysical data to help with the predictive process. That’s why this study should be interpreted as an interim progress report on an area of science that will only get better with time.

But it’s also a sobering reminder that the quest to find new classes of antibiotics won’t be easy—even when aided by powerful AI approaches. We certainly aren’t there yet, but I’m confident that we will get there to give doctors new therapeutic weapons and turn back the rise in antibiotic-resistant infections.

References:

[1] 2019 Antibiotic resistance threats report. Centers for Disease Control and Prevention.

[2] Benchmarking AlphaFold-enabled molecular docking predictions for antibiotic discovery. Wong F, Krishnan A, Zheng EJ, Stark H, Manson AL, Earl AM, Jaakkola T, Collins JJ. Molecular Systems Biology. 2022 Sept 6. 18: e11081.

[3] Highly accurate protein structure prediction with AlphaFold. Jumper J, Evans R, Pritzel A, Kavukcuoglu K, Kohli P, Hassabis D., et al. Nature. 2021 Aug;596(7873):583-589.

[4] ‘The entire protein universe’: AI predicts shape of nearly every known protein. Callaway E. Nature. 2022 Aug;608(7921):15-16.

Links:

Antimicrobial (Drug) Resistance (National Institute of Allergy and Infectious Diseases/NIH)

Collins Lab (Massachusetts Institute of Technology, Cambridge)

The Antibiotics-AI Project, The Audacious Project (TED)

AlphaFold (Deep Mind, London, United Kingdom)

NIH Support: National Institute of Allergy and Infectious Diseases; National Institute of General Medical Sciences


Using AI to Advance Understanding of Long COVID Syndrome

Posted on by

The COVID-19 pandemic continues to present considerable public health challenges in the United States and around the globe. One of the most puzzling is why many people who get over an initial and often relatively mild COVID illness later develop new and potentially debilitating symptoms. These symptoms run the gamut including fatigue, shortness of breath, brain fog, anxiety, and gastrointestinal trouble.

People understandably want answers to help them manage this complex condition referred to as Long COVID syndrome. But because Long COVID is so variable from person to person, it’s extremely difficult to work backwards and determine what these people had in common that might have made them susceptible to Long COVID. The variability also makes it difficult to identify all those who have Long COVID, whether they realize it or not. But a recent study, published in the journal Lancet Digital Health, shows that a well-trained computer and its artificial intelligence can help.

Researchers found that computers, after scanning thousands of electronic health records (EHRs) from people with Long COVID, could reliably make the call. The results, though still preliminary and in need of further validation, point the way to developing a fast, easy-to-use computer algorithm to help determine whether a person with a positive COVID test is likely to battle Long COVID.

In this groundbreaking study, NIH-supported researchers led by Emily Pfaff, University of North Carolina, Chapel Hill, and Melissa Haendel, the University of Colorado Anschutz Medical Campus, Aurora, relied on machine learning. In machine learning, a computer sifts through vast amounts of data to look for patterns. One reason machine learning is so powerful is that it doesn’t require humans to tell the computer which features it should look for. As such, machine learning can pick up on subtle patterns that people would otherwise miss.

In this case, Pfaff, Haendel, and team decided to “train” their computer on EHRs from people who had reported a COVID-19 infection. (The records are de-identified to protect patient privacy.) The researchers found just what they needed in the National COVID Cohort Collaborative (N3C), a national, publicly available data resource sponsored by NIH’s National Center for Advancing Translational Sciences. It is part of NIH’s Researching COVID to Enhance Recovery (RECOVER) initiative, which aims to improve understanding of Long COVID.

The researchers defined a group of more than 1.5 million adults in N3C who either had been diagnosed with COVID-19 or had a record of a positive COVID-19 test at least 90 days prior. Next, they examined common features, including any doctor visits, diagnoses, or medications, from the group’s roughly 100,000 adults.

They fed that EHR data into a computer, along with health information from almost 600 patients who’d been seen at a Long COVID clinic. They developed three machine learning models: one to identify potential long COVID patients across the whole dataset and two others that focused separately on people who had or hadn’t been hospitalized.

All three models proved effective for identifying people with potential Long-COVID. Each of the models had an 85 percent or better discrimination threshold, indicating they are highly accurate. That’s important because, once researchers can identify those with Long COVID in a large database of people such as N3C, they can begin to ask and answer many critical questions about any differences in an individual’s risk factors or treatment that might explain why some get Long COVID and others don’t.

This new study is also an excellent example of N3C’s goal to assemble data from EHRs that enable researchers around the world to get rapid answers and seek effective interventions for COVID-19, including its long-term health effects. It’s also made important progress toward the urgent goal of the RECOVER initiative to identify people with or at risk for Long COVID who may be eligible to participate in clinical trials of promising new treatment approaches.

Long COVID remains a puzzling public health challenge. Another recent NIH study published in the journal Annals of Internal Medicine set out to identify people with symptoms of Long COVID, most of whom had recovered from mild-to-moderate COVID-19 [2]. More than half had signs of Long COVID. But, despite extensive testing, the NIH researchers were unable to pinpoint any underlying cause of the Long COVID symptoms in most cases.

So if you’d like to help researchers solve this puzzle, RECOVER is now enrolling adults and kids—including those who have and have not had COVID—at more than 80 study sites around the country.

References:

[1] Identifying who has long COVID in the USA: a machine learning approach using N3C data. Pfaff ER, Girvin AT, Bennett TD, Bhatia A, Brooks IM, Deer RR, Dekermanjian JP, Jolley SE, Kahn MG, Kostka K, McMurry JA, Moffitt R, Walden A, Chute CG, Haendel MA; N3C Consortium. Lancet Digit Health. 2022 May 16:S2589-7500(22)00048-6.

[2] A longitudinal study of COVID-19 sequelae and immunity: baseline findings. Sneller MC, Liang CJ, Marques AR, Chung JY, Shanbhag SM, Fontana JR, Raza H, Okeke O, Dewar RL, Higgins BP, Tolstenko K, Kwan RW, Gittens KR, Seamon CA, McCormack G, Shaw JS, Okpali GM, Law M, Trihemasava K, Kennedy BD, Shi V, Justement JS, Buckner CM, Blazkova J, Moir S, Chun TW, Lane HC. Ann Intern Med. 2022 May 24:M21-4905.

Links:

COVID-19 Research (NIH)

National COVID Cohort Collaborative (N3C) (National Center for Advancing Translational Sciences/NIH)

RECOVER Initiative

Emily Pfaff (University of North Carolina, Chapel Hill)

Melissa Haendel (University of Colorado, Aurora)

NIH Support: National Center for Advancing Translational Sciences; National Institute of General Medical Sciences; National Institute of Allergy and Infectious Diseases


Next Page