As hospitals use AI chatbots and algorithms, doctors and nurses say they can’t be replaced

After repositioning the tube, his breathing stabilized — a “simple intervention,” Milekic says, that might not have happened without the aid of the computer program.

Milekic’s morning could be an advertisement for the potential of AI to transform health care. Mount Sinai is among a group of elite hospitals pouring hundreds of millions of dollars into AI software and education, turning their institutions into laboratories for this technology. They’re buoyed by a growing body of scientific literature, such as a recent study finding AI readings of mammograms detected 20 percent more cases of breast cancer than radiologists — along with the conviction that AI is the future of medicine.

Researchers are also working to translate generative AI, which backs tools that can create words, sounds and text, into a hospital setting. Mount Sinai has deployed a group of AI specialists to develop medical tools in-house, which doctors and nurses are testing in clinical care. Transcription software completes billing paperwork; chatbots help craft patient summaries.

But the advances are triggering tension among front-line workers, many of whom fear the technology comes at a strong cost to humans. They worry about the technology making wrong diagnoses, revealing sensitive patient data and becoming an excuse for insurance and hospital administrators to cut staff in the name of innovation and efficiency.

Most of all, they say software can’t do the work of a human doctor or nurse.

“If we believe that in our most vulnerable moments … we want somebody who pays attention to us,” Michelle Mahon, the assistant director of nursing practice at the National Nurses United union, said, “then we need to be very careful in this moment.”

Hospitals have dabbled with AI for decades. In the 1970s, Stanford University researchers created a rudimentary AI system that asked doctors questions about a patient’s symptoms and provided a diagnosis based on a database of known infections.

In the 1990s and early 2000s, AI algorithms began deciphering complex patterns in X-rays, CT scans and MRI images to spot abnormalities that the human eye might miss.

Several years later, robots fueled with AI vision began operating alongside surgeons. With the advent of electronic medical records, companies incorporated algorithms that scanned troves of patient data to spot trends and commonalities in patients who had certain ailments, and recommend tailored treatments.

As higher computing power has turbocharged AI, algorithms have moved from spotting trends to predicting whether a specific patient will suffer from an ailment. The rise of generative AI has created tools that more closely mimic patient care.

Vijay Pande, a general partner at venture capital firm Andreessen Horowitz, said health care is at a turning point. “There’s a lot of excitement about AI right now,” he said. “The technology has … gone from being cute and interesting to where actually [people] can see it being deployed.”

In March, the University of Kansas health system started using medical chatbots to automate clinical notes and medical conversations. The Mayo Clinic in Minnesota is using a Google chatbot trained on medical licensing exam questions, called Med-Palm 2, to generate responses to health care questions, summarize clinical documents and organize data, according to a July report in the Wall Street Journal.

Some of these products have already raised eyebrows among elected officials. Sen. Mark R. Warner (D-Va.) on Tuesday urged caution in the rollout of Med-Palm 2, citing repeated inaccuracies in a letter to Google.

“While artificial intelligence (AI) undoubtedly holds tremendous potential to improve patient care and health outcomes, I worry that premature deployment of unproven technology could lead to the erosion of trust in our medical professionals and institutions,” he said in a statement.

Thomas J. Fuchs, the dean for AI at Mount Sinai’s Icahn School of Medicine, said it is imperative that research hospitals, which are staffed with pioneering physicians and researchers, act as laboratories to test this technology.

Mount Sinai has taken the premise literally, raising over 100 million dollars through private philanthropy and building research centers and on-site computing facilities. This allows programmers to build AI tools in-house that can be refined based on physician input, used in their hospitals and also sent to places that don’t have the money to do similar research.

“You cannot transplant people,” Fuchs said. “But you can transplant knowledge and experience to some degree with these models that then can help physicians in the community.”

But Fuchs added that “there’s enormous amount of hype” about AI in medicine right now, and “more start-up companies than you can count who … like to evangelize to sometimes absurd degrees” about the revolutionary powers the technology can hold in medicine. He worries they may create products that make biased diagnoses or put patient data at risk. Strong federal regulation, along with physician oversight, is paramount, he said.

David L. Reich, the president of The Mount Sinai hospital and Mount Sinai Queens, said his hospital has been wanting to use AI more broadly for a few years, but the pandemic delayed its rollout.

Though generative chatbots are becoming popular, Reich’s team is focusing mostly on using algorithms. Critical care physicians are piloting predictive software to identify patients who are at risk of issues such as sepsis or falling — the kind of software used by Milekic. Radiologists use AI to more accurately spot breast cancer. Nutritionists use AI to flag patients who are likely to be malnourished.

Reich said the ultimate goal is not to replace health workers, but something more simple: getting the right doctor to the right patient at the right time.

But some medical professionals aren’t as comfortable with the new technology.

Mahon, of National Nurses United, said there is very little empirical evidence to demonstrate AI is actually improving patient care.

“We do experiments in this country, we use the clinical trial, but for some reason, these technologies, they’re being given a pass,” she said. “They’re being marketed as superior, as ever present, and other types of things that just simply don’t bear out in their utilization.”

Though AI can analyze troves of data and predict how sick a patient might be, Mahon has often found that these algorithms can get it wrong. Nurses see beyond a patient’s vital signs, she argues. They see how a patient looks, smell unnatural odors from their body and can use these biological data points as predictors that something might be wrong. “AI can’t do that,” she said.

Some physicians interviewed by Duke University in a May survey expressed reservations AI models might exacerbate existing issues with care, including bias. “I don’t think we even really have a great understanding of how to measure an algorithm’s performance, let alone its performance across different race and ethnic groups,” one respondent told researchers in the study of caregivers at hospitals including the Mayo Clinic, Kaiser Permanente and the University of California San Francisco.

At a time of severe nursing shortage, Mahon said hospital administrators’ excitement to incorporate the technology is less about patient outcomes and more about plugging holes and saving costs.

“The [health care] industry really is helping people buy into the all the hype,” she said, “so that they can cut back on their labor without any questions.”

Robbie Freeman, Mount Sinai’s vice president of digital experience, said the hardest parts of getting AI into hospitals are the doctors and nurses themselves. “You may have come to work for 20 years and done it one way,” he said, “and now we’re coming in and asking you to do it another way.”

“People may feel like it’s flavor of the month,” he added. “They may not fully be … bought into the idea of adopting some sort of new practice or tool.”

And AI is not always a surefire method for saving time. When Rebecca Brown, a 45-year-old heart patient from Corning, N.Y., was flagged as one of the sickest patients in Mount Sinai’s critical care ward on a May morning, Milekic went to her room to run an examination.

Milekic quickly saw nothing was out of the ordinary, letting Brown continue eating her peanut butter and jelly sandwich.

Asked whether she would want AI to care for her over a doctor, Brown’s answer was simple: “There is something that technology can never do, and that is be human,” she said. “[I] hope that the human touch doesn’t go away.”