Skip to main content

deep learning

What A Year It Was for Science Advances!

Posted on by

Science Breakthroughs of the Year 2020

At the close of every year, editors and writers at the journal Science review the progress that’s been made in all fields of science—from anthropology to zoology—to select the biggest advance of the past 12 months. In most cases, this Breakthrough of the Year is as tough to predict as the Oscar for Best Picture. Not in 2020. In a year filled with a multitude of challenges posed by the emergence of the deadly coronavirus disease 2019 (COVID-2019), the breakthrough was the development of the first vaccines to protect against this pandemic that’s already claimed the lives of more than 360,000 Americans.

In keeping with its annual tradition, Science also selected nine runner-up breakthroughs. This impressive list includes at least three areas that involved efforts supported by NIH: therapeutic applications of gene editing, basic research understanding HIV, and scientists speaking up for diversity. Here’s a quick rundown of all the pioneering advances in biomedical research, both NIH and non-NIH funded:

Shots of Hope. A lot of things happened in 2020 that were unprecedented. At the top of the list was the rapid development of COVID-19 vaccines. Public and private researchers accomplished in 10 months what normally takes about 8 years to produce two vaccines for public use, with more on the way in 2021. In my more than 25 years at NIH, I’ve never encountered such a willingness among researchers to set aside their other concerns and gather around the same table to get the job done fast, safely, and efficiently for the world.

It’s also pretty amazing that the first two conditionally approved vaccines from Pfizer and Moderna were found to be more than 90 percent effective at protecting people from infection with SARS-CoV-2, the coronavirus that causes COVID-19. Both are innovative messenger RNA (mRNA) vaccines, a new approach to vaccination.

For this type of vaccine, the centerpiece is a small, non-infectious snippet of mRNA that encodes the instructions to make the spike protein that crowns the outer surface of SARS-CoV-2. When the mRNA is injected into a shoulder muscle, cells there will follow the encoded instructions and temporarily make copies of this signature viral protein. As the immune system detects these copies, it spurs the production of antibodies and helps the body remember how to fend off SARS-CoV-2 should the real thing be encountered.

It also can’t be understated that both mRNA vaccines—one developed by Pfizer and the other by Moderna in conjunction with NIH’s National Institute of Allergy and Infectious Diseases—were rigorously evaluated in clinical trials. Detailed data were posted online and discussed in all-day meetings of an FDA Advisory Committee, open to the public. In fact, given the high stakes, the level of review probably was more scientifically rigorous than ever.

First CRISPR Cures: One of the most promising areas of research now underway involves gene editing. These tools, still relatively new, hold the potential to fix gene misspellings—and potentially cure—a wide range of genetic diseases that were once to be out of reach. Much of the research focus has centered on CRISPR/Cas9. This highly precise gene-editing system relies on guide RNA molecules to direct a scissor-like Cas9 enzyme to just the right spot in the genome to cut out or correct a disease-causing misspelling.

In late 2020, a team of researchers in the United States and Europe succeeded for the first time in using CRISPR to treat 10 people with sickle cell disease and transfusion-dependent beta thalassemia. As published in the New England Journal of Medicine, several months after this non-heritable treatment, all patients no longer needed frequent blood transfusions and are living pain free [1].

The researchers tested a one-time treatment in which they removed bone marrow from each patient, modified the blood-forming hematopoietic stem cells outside the body using CRISPR, and then reinfused them into the body. To prepare for receiving the corrected cells, patients were given toxic bone marrow ablation therapy, in order to make room for the corrected cells. The result: the modified stem cells were reprogrammed to switch back to making ample amounts of a healthy form of hemoglobin that their bodies produced in the womb. While the treatment is still risky, complex, and prohibitively expensive, this work is an impressive start for more breakthroughs to come using gene editing technologies. NIH, including its Somatic Cell Genome Editing program, continues to push the technology to accelerate progress and make gene editing cures for many disorders simpler and less toxic.

Scientists Speak Up for Diversity: The year 2020 will be remembered not only for COVID-19, but also for the very public and inescapable evidence of the persistence of racial discrimination in the United States. Triggered by the killing of George Floyd and other similar events, Americans were forced to come to grips with the fact that our society does not provide equal opportunity and justice for all. And that applies to the scientific community as well.

Science thrives in safe, diverse, and inclusive research environments. It suffers when racism and bigotry find a home to stifle diversity—and community for all—in the sciences. For the nation’s leading science institutions, there is a place and a calling to encourage diversity in the scientific workplace and provide the resources to let it flourish to everyone’s benefit.

For those of us at NIH, last year’s peaceful protests and hashtags were noticed and taken to heart. That’s one of the many reasons why we will continue to strengthen our commitment to building a culturally diverse, inclusive workplace. For example, we have established the NIH Equity Committee. It allows for the systematic tracking and evaluation of diversity and inclusion metrics for the intramural research program for each NIH institute and center. There is also the recently founded Distinguished Scholars Program, which aims to increase the diversity of tenure track investigators at NIH. Recently, NIH also announced that it will provide support to institutions to recruit diverse groups or “cohorts” of early-stage research faculty and prepare them to thrive as NIH-funded researchers.

AI Disentangles Protein Folding: Proteins, which are the workhorses of the cell, are made up of long, interconnected strings of amino acids that fold into a wide variety of 3D shapes. Understanding the precise shape of a protein facilitates efforts to figure out its function, its potential role in a disease, and even how to target it with therapies. To gain such understanding, researchers often try to predict a protein’s precise 3D chemical structure using basic principles of physics—including quantum mechanics. But while nature does this in real time zillions of times a day, computational approaches have not been able to do this—until now.

Of the roughly 170,000 proteins mapped so far, most have had their structures deciphered using powerful imaging techniques such as x-ray crystallography and cryo–electron microscopy (cryo-EM). But researchers estimate that there are at least 200 million proteins in nature, and, as amazing as these imaging techniques are, they are laborious, and it can take many months or years to solve 3D structure of a single protein. So, a breakthrough certainly was needed!

In 2020, researchers with the company Deep Mind, London, developed an artificial intelligence (AI) program that rapidly predicts most protein structures as accurately as x-ray crystallography and cryo-EM can map them [2]. The AI program, called AlphaFold, predicts a protein’s structure by computationally modeling the amino acid interactions that govern its 3D shape.

Getting there wasn’t easy. While a complete de novo calculation of protein structure still seemed out of reach, investigators reasoned that they could kick start the modeling if known structures were provided as a training set to the AI program. Utilizing a computer network built around 128 machine learning processors, the AlphaFold system was created by first focusing on the 170,000 proteins with known structures in a reiterative process called deep learning. The process, which is inspired by the way neural networks in the human brain process information, enables computers to look for patterns in large collections of data. In this case, AlphaFold learned to predict the underlying physical structure of a protein within a matter of days. This breakthrough has the potential to accelerate the fields of structural biology and protein research, fueling progress throughout the sciences.

How Elite Controllers Keep HIV at Bay: The term “elite controller” might make some people think of video game whizzes. But here, it refers to the less than 1 percent of people living with human immunodeficiency virus (HIV) who’ve somehow stayed healthy for years without taking antiretroviral drugs. In 2020, a team of NIH-supported researchers figured out why this is so.

In a study of 64 elite controllers, published in the journal Nature, the team discovered a link between their good health and where the virus has inserted itself in their genomes [3]. When a cell transcribes a gene where HIV has settled, this so-called “provirus,” can produce more virus to infect other cells. But if it settles in a part of a chromosome that rarely gets transcribed, sometimes called a gene desert, the provirus is stuck with no way to replicate. Although this discovery won’t cure HIV/AIDS, it points to a new direction for developing better treatment strategies.

In closing, 2020 presented more than its share of personal and social challenges. Among those challenges was a flood of misinformation about COVID-19 that confused and divided many communities and even families. That’s why the editors and writers at Science singled out “a second pandemic of misinformation” as its Breakdown of the Year. This divisiveness should concern all of us greatly, as COVID-19 cases continue to soar around the country and our healthcare gets stretched to the breaking point. I hope and pray that we will all find a way to come together, both in science and in society, as we move forward in 2021.

References:

[1] CRISPR-Cas9 gene editing for sickle cell disease and β-thalassemia. Frangoul H et al. N Engl J Med. 2020 Dec 5.

[2] ‘The game has changed.’ AI triumphs at protein folding. Service RF. Science. 04 Dec 2020.

[3] Distinct viral reservoirs in individuals with spontaneous control of HIV-1. Jiang C et al. Nature. 2020 Sep;585(7824):261-267.

Links:

COVID-19 Research (NIH)

2020 Science Breakthrough of the Year (American Association for the Advancement of Science, Washington, D.C)


Artificial Intelligence Speeds Brain Tumor Diagnosis

Posted on by

Real time diagnostics in the operating room
Caption: Artificial intelligence speeds diagnosis of brain tumors. Top, doctor reviews digitized tumor specimen in operating room; left, the AI program predicts diagnosis; right, surgeons review results in near real-time.
Credit: Joe Hallisy, Michigan Medicine, Ann Arbor

Computers are now being trained to “see” the patterns of disease often hidden in our cells and tissues. Now comes word of yet another remarkable use of computer-generated artificial intelligence (AI): swiftly providing neurosurgeons with valuable, real-time information about what type of brain tumor is present, while the patient is still on the operating table.

This latest advance comes from an NIH-funded clinical trial of 278 patients undergoing brain surgery. The researchers found they could take a small tumor biopsy during surgery, feed it into a trained computer in the operating room, and receive a diagnosis that rivals the accuracy of an expert pathologist.

Traditionally, sending out a biopsy to an expert pathologist and getting back a diagnosis optimally takes about 40 minutes. But the computer can do it in the operating room on average in under 3 minutes. The time saved helps to inform surgeons how to proceed with their delicate surgery and make immediate and potentially life-saving treatment decisions to assist their patients.

As reported in Nature Medicine, researchers led by Daniel Orringer, NYU Langone Health, New York, and Todd Hollon, University of Michigan, Ann Arbor, took advantage of AI and another technological advance called stimulated Raman histology (SRH). The latter is an emerging clinical imaging technique that makes it possible to generate detailed images of a tissue sample without the usual processing steps.

The SRH technique starts off by bouncing laser light rapidly through a tissue sample. This light enables a nearby fiberoptic microscope to capture the cellular and structural details within the sample. Remarkably, it does so by picking up on subtle differences in the way lipids, proteins, and nucleic acids vibrate when exposed to the light.

Then, using a virtual coloring program, the microscope quickly pieces together and colors in the fine structural details, pixel by pixel. The result: a high-resolution, detailed image that you might expect from a pathology lab, minus the staining of cells, mounting of slides, and the other time-consuming processing procedures.

To interpret the SRH images, the researchers turned to computers and machine learning. To teach a computer, it must be fed large datasets of examples in order to learn how to perform a given task. In this case, they used a special class of machine learning called deep neural networks, or deep learning. It’s inspired by the way neural networks in the human brain process information.

In deep learning, computers look for patterns in large collections of data. As they begin to recognize complex relationships, some connections in the network are strengthened while others are weakened. The finished network is typically composed of multiple information-processing layers, which operate on the data to return a result, in this case a brain tumor diagnosis.

The team trained the computer to classify tissues samples into one of 13 categories commonly found in a brain tumor sample. Those categories included the most common brain tumors: malignant glioma, lymphoma, metastatic tumors, and meningioma. The training was based on more than 2.5 million labeled images representing samples from 415 patients.

Next, they put the machine to the test. The researchers split each of 278 brain tissue samples into two specimens. One was sent to a conventional pathology lab for prepping and diagnosis. The other was imaged with SRH, and then the trained machine made a diagnosis.

Overall, the machine’s performance was quite impressive, returning the right answer about 95 percent of the time. That’s compared to an accuracy of 94 percent for conventional pathology.

Interestingly, the machine made a correct diagnosis in all 17 cases that a pathologist got wrong. Likewise, the pathologist got the right answer in all 14 cases in which the machine slipped up.

The findings show that the combination of SRH and AI can be used to make real-time predictions of a patient’s brain tumor diagnosis to inform surgical decision-making. That may be especially important in places where expert neuropathologists are hard to find.

Ultimately, the researchers suggest that AI may yield even more useful information about a tumor’s underlying molecular alterations, adding ever greater precision to the diagnosis. Similar approaches are also likely to work in supplying timely information to surgeons operating on patients with other cancers too, including cancers of the skin and breast. The research team has made a brief video to give you a more detailed look at the new automated tissue-to-diagnosis pipeline.

Reference:

[1] Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks. Hollon TC, Pandian B, Adapa AR, Urias E, Save AV, Khalsa SSS, Eichberg DG, D’Amico RS, Farooq ZU, Lewis S, Petridis PD, Marie T, Shah AH, Garton HJL, Maher CO, Heth JA, McKean EL, Sullivan SE, Hervey-Jumper SL, Patil PG, Thompson BG, Sagher O, McKhann GM 2nd, Komotar RJ, Ivan ME, Snuderl M, Otten ML, Johnson TD, Sisti MB, Bruce JN, Muraszko KM, Trautman J, Freudiger CW, Canoll P, Lee H, Camelo-Piragua S, Orringer DA. Nat Med. 2020 Jan 6.

Links:

Video: Artificial Intelligence: Collecting Data to Maximize Potential (NIH)

New Imaging Technique Allows Quick, Automated Analysis of Brain Tumor Tissue During Surgery (National Institute of Biomedical Imaging and Bioengineering/NIH)

Daniel Orringer (NYU Langone, Perlmutter Cancer Center, New York City)

Todd Hollon (University of Michigan, Ann Arbor)

NIH Support: National Cancer Institute; National Institute of Biomedical Imaging and Bioengineering


Using Artificial Intelligence to Detect Cervical Cancer

Posted on by

Doctor reviewing cell phone
Credit: gettyimages/Dean Mitchell

My last post highlighted the use of artificial intelligence (AI) to create an algorithm capable of detecting 10 different kinds of irregular heart rhythms. But that’s just one of the many potential medical uses of AI. In this post, I’ll tell you how NIH researchers are pairing AI analysis with smartphone cameras to help more women avoid cervical cancer.

In work described in the Journal of the National Cancer Institute [1], researchers used a high-performance computer to analyze thousands of cervical photographs, obtained more than 20 years ago from volunteers in a cancer screening study. The computer learned to recognize specific patterns associated with pre-cancerous and cancerous changes of the cervix, and that information was used to develop an algorithm for reliably detecting such changes in the collection of images. In fact, the AI-generated algorithm outperformed human expert reviewers and all standard screening tests in detecting pre-cancerous changes.

Nearly all cervical cancers are caused by the human papillomavirus (HPV). Cervical cancer screening—first with Pap smears and now also using HPV testing—have greatly reduced deaths from cervical cancer. But this cancer still claims the lives of more than 4,000 U.S. women each year, with higher frequency among women who are black or older [2]. Around the world, more than a quarter-million women die of this preventable disease, mostly in poor and remote areas [3].

These troubling numbers have kept researchers on the lookout for low cost, but easy-to-use, tools that could be highly effective at detecting HPV infections most likely to advance to cervical cancer. Such tools would also need to work well in areas with limited resources for sample preparation and lab analysis. That’s what led to this collaboration involving researchers from NIH’s National Cancer Institute (NCI) and Global Good, Bellevue, WA, which is an Intellectual Ventures collaboration with Bill Gates to invent life-changing technologies for the developing world.

Global Good researchers contacted NCI experts hoping to apply AI to a large dataset of cervical images. The NCI experts suggested an 18-year cervical cancer screening study in Costa Rica. The NCI-supported project, completed in the 1990s, generated nearly 60,000 cervical images, later digitized by NIH’s National Library of Medicine and stored away safely.

The researchers agreed that all these images, obtained in a highly standardized way, would serve as perfect training material for a computer to develop a detection algorithm for cervical cancer. This type of AI, called machine learning, involves feeding tens of thousands of images into a computer equipped with one or more high-powered graphics processing units (GPUs), similar to something you’d find in an Xbox or PlayStation. The GPUs allow the computer to crunch large sets of visual data in the images and devise a set of rules, or algorithms, that allow it to learn to “see” physical features.

Here’s how they did it. First, the researchers got the computer to create a convolutional neural network. That’s a fancy way of saying that they trained it to read images, filter out the millions of non-essential bytes, and retain the few hundred bytes in the photo that make it uniquely identifiable. They fed 1.28 million color images covering hundreds of common objects into the computer to create layers of processing ability that, like the human visual system, can distinguish objects and their qualities.

Once the convolutional neural network was formed, the researchers took the next big step: training the system to see the physical properties of a healthy cervix, a cervix with worrisome cellular changes, or a cervix with pre-cancer. That’s where the thousands of cervical images from the Costa Rican screening trial literally entered the picture.

When all these layers of processing ability were formed, the researchers had created the “automated visual evaluation” algorithm. It went on to identify with remarkable accuracy the images associated with the Costa Rican study’s 241 known precancers and 38 known cancers. The algorithm’s few minor hiccups came mainly from suboptimal images with faded colors or slightly blurred focus.

These minor glitches have the researchers now working hard to optimize the process, including determining how health workers can capture good quality photos of the cervix with a smartphone during a routine pelvic exam and how to outfit smartphones with the necessary software to analyze cervical photos quickly in real-world settings. The goal is to enable health workers to use a smartphone or similar device to provide women with cervical screening and treatment during a single visit.

In fact, the researchers are already field testing their AI-inspired approach on smartphones in the United States and abroad. If all goes well, this low-cost, mobile approach could provide a valuable new tool to help reduce the burden of cervical cancer among underserved populations.

The day that cervical cancer no longer steals the lives of hundreds of thousands of women a year worldwide will be a joyful moment for cancer researchers, as well as a major victory for women’s health.

References:

[1] An observational study of Deep Learning and automated evaluation of cervical images for cancer screening. Hu L, Bell D, Antani S, Xue Z, Yu K, Horning MP, Gachuhi N, Wilson B, Jaiswal MS, Befano B, Long LR, Herrero R, Einstein MH, Burk RD, Demarco M, Gage JC, Rodriguez AC, Wentzensen N, Schiffman M. J Natl Cancer Inst. 2019 Jan 10. [Epub ahead of print]

[2] “Study: Death Rate from Cervical Cancer Higher Than Thought,” American Cancer Society, Jan. 25, 2017.

[3] “World Cancer Day,” World Health Organization, Feb. 2, 2017.

Links:

Cervical Cancer (National Cancer Institute/NIH)

Global Good (Intellectual Ventures, Bellevue, WA)

NIH Support: National Cancer Institute; National Library of Medicine


Using Artificial Intelligence to Catch Irregular Heartbeats

Posted on by

ECG Readout
Credit: gettyimages/enot-poloskun

Thanks to advances in wearable health technologies, it’s now possible for people to monitor their heart rhythms at home for days, weeks, or even months via wireless electrocardiogram (EKG) patches. In fact, my Apple Watch makes it possible to record a real-time EKG whenever I want. (I’m glad to say I am in normal sinus rhythm.)

For true medical benefit, however, the challenge lies in analyzing the vast amounts of data—often hundreds of hours worth per person—to distinguish reliably between harmless rhythm irregularities and potentially life-threatening problems. Now, NIH-funded researchers have found that artificial intelligence (AI) can help.

A powerful computer “studied” more than 90,000 EKG recordings, from which it “learned” to recognize patterns, form rules, and apply them accurately to future EKG readings. The computer became so “smart” that it could classify 10 different types of irregular heart rhythms, including atrial fibrillation (AFib). In fact, after just seven months of training, the computer-devised algorithm was as good—and in some cases even better than—cardiology experts at making the correct diagnostic call.

EKG tests measure electrical impulses in the heart, which signal the heart muscle to contract and pump blood to the rest of the body. The precise, wave-like features of the electrical impulses allow doctors to determine whether a person’s heart is beating normally.

For example, in people with AFib, the heart’s upper chambers (the atria) contract rapidly and unpredictably, causing the ventricles (the main heart muscle) to contract irregularly rather than in a steady rhythm. This is an important arrhythmia to detect, even if it may only be present occasionally over many days of monitoring. That’s not always easy to do with current methods.

Here’s where the team, led by computer scientists Awni Hannun and Andrew Ng, Stanford University, Palo Alto, CA, saw an AI opportunity. As published in Nature Medicine, the Stanford team started by assembling a large EKG dataset from more than 53,000 people [1]. The data included various forms of arrhythmia and normal heart rhythms from people who had worn the FDA-approved Zio patch for about two weeks.

The Zio patch is a 2-by-5-inch adhesive patch, worn much like a bandage, on the upper left side of the chest. It’s water resistant and can be kept on around the clock while a person sleeps, exercises, or takes a shower. The wireless patch continuously monitors heart rhythms, storing EKG data for later analysis.

The Stanford researchers looked to machine learning to process all the EKG data. In machine learning, computers rely on large datasets of examples in order to learn how to perform a given task. The accuracy improves as the machine “sees” more data.

But the team’s real interest was in utilizing a special class of machine learning called deep neural networks, or deep learning. Deep learning is inspired by how our own brain’s neural networks process information, learning to focus on some details but not others.

In deep learning, computers look for patterns in data. As they begin to “see” complex relationships, some connections in the network are strengthened while others are weakened. The network is typically composed of multiple information-processing layers, which operate on the data and compute increasingly complex and abstract representations.

Those data reach the final output layer, which acts as a classifier, assigning each bit of data to a particular category or, in the case of the EKG readings, a diagnosis. In this way, computers can learn to analyze and sort highly complex data using both more obvious and hidden features.

Ultimately, the computer in the new study could differentiate between EKG readings representing 10 different arrhythmias as well as a normal heart rhythm. It could also tell the difference between irregular heart rhythms and background “noise” caused by interference of one kind or another, such as a jostled or disconnected Zio patch.

For validation, the computer attempted to assign a diagnosis to the EKG readings of 328 additional patients. Independently, several expert cardiologists also read those EKGs and reached a consensus diagnosis for each patient. In almost all cases, the computer’s diagnosis agreed with the consensus of the cardiologists. The computer also made its calls much faster.

Next, the researchers compared the computer’s diagnoses to those of six individual cardiologists who weren’t part of the original consensus committee. And, the results show that the computer actually outperformed these experienced cardiologists!

The findings suggest that artificial intelligence can be used to improve the accuracy and efficiency of EKG readings. In fact, Hannun reports that iRhythm Technologies, maker of the Zio patch, has already incorporated the algorithm into the interpretation now being used to analyze data from real patients.

As impressive as this is, we are surely just at the beginning of AI applications to health and health care. In recognition of the opportunities ahead, NIH has recently launched a working group on AI to explore ways to make the best use of existing data, and harness the potential of artificial intelligence and machine learning to advance biomedical research and the practice of medicine.

Meanwhile, more and more impressive NIH-supported research featuring AI is being published. In my next blog, I’ll highlight a recent paper that uses AI to make a real difference for cervical cancer, particularly in low resource settings.

Reference:

[1] Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Hannun AY, Rajpurkar P, Haghpanahi M, Tison GH, Bourn C, Turakhia MP, Ng AY.
Nat Med. 2019 Jan;25(1):65-69.

Links:

Arrhythmia (National Heart, Lung, and Blood Institute/NIH)

Video: Artificial Intelligence: Collecting Data to Maximize Potential (NIH)

Andrew Ng (Palo Alto, CA)

NIH Support: National Heart, Lung, and Blood Institute


Teaching Computers to “See” the Invisible in Living Cells

Posted on by

Brain Cell Analysis

Caption: While analyzing brain cells, a computer program “thinks” about which cellular structure to identify.
Credit: Steven Finkbeiner, University of California, San Francisco and the Gladstone Institutes

For centuries, scientists have trained themselves to look through microscopes and carefully study their structural and molecular features. But those long hours bent over a microscope poring over microscopic images could be less necessary in the years ahead. The job of analyzing cellular features could one day belong to specially trained computers.

In a new study published in the journal Cell, researchers trained computers by feeding them paired sets of fluorescently labeled and unlabeled images of brain tissue millions of times in a row [1]. This allowed the computers to discern patterns in the images, form rules, and apply them to viewing future images. Using this so-called deep learning approach, the researchers demonstrated that the computers not only learned to recognize individual cells, they also developed an almost superhuman ability to identify the cell type and whether a cell was alive or dead. Even more remarkable, the trained computers made all those calls without any need for harsh chemical labels, including fluorescent dyes or stains, which researchers normally require to study cells. In other words, the computers learned to “see” the invisible!