Meta’s VR technology is helping to train surgeons and treat patients

Just days before assisting in his first major shoulder-replacement surgery last year, Dr. Jake Shine strapped on a virtual reality headset and got to work.

As a third-year orthopedics resident at Kettering Health Dayton in Ohio, Shine was standing in the medical center’s designated VR lab with his attending physician, who would oversee the procedure. 

Both doctors were wearing Meta Quest 2 headsets as they walked through a 3D simulation of the surgery. The procedure, called a reverse total shoulder arthroplasty, can last around two hours and requires surgeons to carefully navigate around neurovascular structures and the lungs.

After the mock procedure, Shine took his headset home to practice. He did so roughly twice a day before the surgery.

“You can really fine-tune and learn what to do, but also what definitely not to do, with zero risk to the patient,” Shine told CNBC in an interview. 

Ultimately, there were no complications in the procedure and the patient made a full recovery, he said. 

“Anecdotally, I think it went smoother and quicker than it would have,” Shine said, than if the attending physician “was having to walk me through every step in the case the same way that he did in the VR.” 

While consumer VR remains a niche product and a massive money-burning venture for Meta CEO Mark Zuckerberg, the technology is proving to be valuable in certain corners of health care. Kettering Health Dayton is one of dozens of health systems in the U.S. working with emerging technologies like VR as one tool for helping doctors to train on and treat patients.

The broad category of “extended reality” includes fully immersive VR headsets like the Quest 2, and augmented reality (AR) devices, where the user can see a digital overlay on top of real-world surroundings.

Whether the nascent technology can ever be cost-effective across the medical industry is very much an open question, but early tests are showing the potential utility of VR in helping to improve health outcomes.

Meta, then known as Facebook, entered the market with the purchase of Oculus in 2014. Three years later, the company introduced its first stand-alone headset. In 2021, Facebook rebranded as Meta, and Zuckerberg committed to spending billions, betting the metaverse would be “the next chapter for the internet.” Since the beginning of last year, Meta’s Reality Labs unit, which develops the company’s VR and AR, has lost over $21 billion. 

Apple is preparing to enter the VR market, going after the higher-end user with the $3,500 Vision Pro that’s expected to debut early next year. Meta is slated to release the Meta Quest 3 as soon as next month.

An Apple spokesperson didn’t provide a comment on potential uses in health care and directed CNBC to an announcement in June regarding Vision Pro’s software developer kit. In that announcement, Jan Herzhoff, Elsevier Health’s president, is quoted as saying that her company’s Complete HeartX mixed reality offering “will help prepare medical students for clinical practice by using hyper-realistic 3D models and animations that help them understand and visualize medical issues, such as ventricular fibrillation, and how to apply their knowledge with patients.”

Extended reality as treatment for patients

To date, one of the primary applications of VR in health care has been targeted at pain treatment.

“It’s very hard to keep track of pain when you’re in a fantastical cyberdelic world,” said Dr. Brennan Spiegel, director of health services research at Cedars-Sinai in Los Angeles.

Spiegel said that when someone is injured, there is both a physical and an emotional component to their pain. Those signals are sent to two different parts of the brain, and VR can serve to tamp down the signals in both regions.

“It’s training people how to modify their spotlight of attention so they can swing it away from the painful experiences,” Spiegel said. “Not just the physical, but the emotional experiences.” 

Spiegel said Cedars-Sinai is preparing to launch a virtual platform to help people with gastrointestinal issues like Crohn’s disease, celiac disease or acid reflux, as well as others for anxiety, addiction and perimenopausal health.    

The technology has also attracted the attention of the U.S. Department of Veterans Affairs, which is using extended reality at more than 160 facilities to help patients with pain management, behavioral therapy and both physical and cognitive rehabilitation. 

Caitlin Rawlins, the immersive program manager at the VA, said there are currently more than 40 separate use cases for the technology across the agency’s different sites. The VA first introduced extended reality in a limited capacity around 2015, and has found more opportunities to put it to use as the technology has improved. 

“I’ve seen it change a whole lot,” Rawlins told CNBC in an interview. “The first virtual reality headset that I used was this big clunky headset that had all these wires it had to be connected to a laptop to function.”

Rawlins said what drew her to extended reality was seeing the immediate response from patients. She recalled the first time she watched a patient use VR. He was a man in his 80s who had just undergone knee replacement surgery. The pain was so severe that opioids didn’t help, Rawlins said.

After mere minutes in VR, he told Rawlins he couldn’t feel the pain in his leg anymore. 

“Just using that for a simple 30-minute session can mean the difference between excruciating pain, unable to do the exercises and the ambulation that they need to, to actually get up and move and get ready to go home,” she said.

Rawlins described another patient as a “surly” wheelchair-bound Army veteran who was experiencing some cognitive decline. The VA had the patient try VR to see if it could lessen the need for antipsychotic medications. 

With the headset on, Rawlins had the patient navigate through a virtual nature scene, walking through the woods, climbing rocks and interacting with birds and deer. Rawlins said the patient was smiling and laughing and was transformed into a “completely different person.”

“To see a patient who has been wheelchair-bound for like 15 years getting to walk through the woods and interact with animals again, it was a pretty powerful moment,” Rawlins said. “Those are the sort of experiences that we keep seeing over and over and over again.”

Both Spiegel and Rawlins said their organizations are hardware agnostic, meaning they can use headsets made by Meta, Apple or any other company as long as they can support the right software.

Spiegel said there’s “potentially millions and millions of people who might be willing to actually buy a headset” but who see them as a gaming and entertainment devices and have no idea about the health applications.

Meta has loosely identified health care as a target market. The company has released case studies and promoted short videos depicting futuristic surgeons in training.

However, it doesn’t appear to be as much a priority as gaming and entertainment. For example, while Cedars-Sinai can technically make its software available in the Meta Quest Store, users would have to go to a section of the store called the App Lab to access it. Software in the App Lab is not marketed traditionally or as easily discoverable via search.

Meta didn’t provide a comment, directing CNBC to a post on Sept. 7, about uses of metaverse technology. The post says: “Training for surgery is just one of the many industries being transformed in ways that are positively impacting lives.”

‘Together in the virtual world’

The technology is also becoming a fixture in many medical schools and residency programs. 

At Kettering Health Dayton, VR recently became a mandatory component of the curriculum for first-year orthopedics residents. In July, the new doctors completed a monthlong “boot camp,” where they carried out clinical services in the mornings and practiced in VR in the afternoons. They now have to complete at least three modules a week in VR with a score of over 70%.

For more senior level residents like Shine, VR training is not yet mandatory, but Kettering Health Dayton is actively working to build it into each level of the program.

“The way I trained in the late 80s, I mean, basically you read the books,” said Dr. Brent Bamberger, the director of the orthopedic surgery residency program at Kettering Health Dayton. “We didn’t have the videos at that time. You may go to a lecture, you may get lucky and have a specimen lab or some type of lab to do it, but you were learning by watching.”

Dr. Reem Daboul, a first-year resident at the hospital, said headsets can’t replicate the physical feeling of a procedure. But she’s found them very useful in important ways. She can already use a headset to walk through the steps of an anterior hip replacement, which many orthopedic surgeons don’t learn until their third year of residency or later.

“Being able to have something help me and see what I’m supposed to be doing and be able to walk through the steps, it’s been super helpful for me,” Daboul said in an interview.

For its orthopedics program, Kettering Health Dayton uses software developed by PrecisionOS, a company that builds VR modules for training surgeons, medical residents and medical device representatives.  PrecisionOS co-founder and CEO, Dr. Danny Goel, said the company has nearly 80 customers across the globe. 

Orthopedics residents at the University of Rochester also use PrecisionOS. Dr. Richard Miller, a retired professor at the university, said the software is “sophisticated” and “very realistic,” especially as a way to learn the steps of a procedure. He finds it so compelling that he’s been actively helping the orthopedics department implement the technology even though he retired three years ago. 

Miller said the VR is a useful way for residents to hone their skills without having to immediately deal with operating room pressures. They can also practice at home.

“I can be at home in my study at night, and they can be in their dorm at night, and we can do a procedure together in the virtual world,” Miller said.

Despite VR’s advantages, Miller said the software has to be able to update frequently to stay current with standards of care, best practices and surgery techniques.

“Next year, they may change the procedure a little bit, now you have other tools and things are a little bit different. Who’s going to change that? Who’s going to bring it up to date?” Miller said.

Those are important questions for quality of care. They’re also important because hospitals generally have to work on tight budgets, and the costs aren’t always clear.

“I can’t get straight answers from anybody, really, as to exactly how much it costs and who does what,” Miller said. “It’s got to be a hurdle.”

PrecisionOS declined to share specific pricing information with CNBC. Goel said costs of using the company’s software vary based on the institution and the partnership.  

Kettering’s Bamberger said that in addition to the software challenges, the hardware is still rather “clunky.” Others in his field also see the limitations.

Dr. Rafael Grossmann, a surgeon at Portsmouth Regional Hospital in New Hampshire, has spent much of his career educating people about health-care applications for emerging technologies like extended reality. 

In 2013, Grossmann became the first person to use the infamous Google Glass during a surgery as a way to stream the procedure, with the patient’s consent, into a room of students. Google had built a lightweight AR device that displayed tiny bits of information on a transparent screen in the user’s field of view. It was first sold to developers and early adopters in 2013 for $1,500, and quickly captured the imagination of tech enthusiasts. 

But Glass never took off. The built-in camera led to fights over privacy, and the product became the butt of jokes on late-night television. 

Ten years later, Grossmann said he now sees a substantial market for the technology, particularly within health care. He said headsets have improved dramatically, even if they’re still bulky and not entirely functional for doctors.

“The interface is better than it was three years ago, but it’s certainly not ideal for really any sort of health-care setting,” Grossmann said.

A growing area of research

As with all technology in health care, extended reality is going to have to clear regulatory hurdles. 

The U.S. Food and Drug Administration has a small team of researchers that are responsible for carrying out “regulatory science” around the technology. 

Ryan Beams, a physicist at the FDA, conducts this research alongside a team, consulting with a range of experts about emerging AR, VR and mixed reality devices. As a result, Beams said the FDA is able to help establish general best practices for how to test promising devices and bring them to market safely. 

“We can say these are the tests we need done, these are the kinds of ways you should do the tests, and then we can help the companies get through those,” Beams told CNBC. “What you don’t want is a device that potentially could help someone getting delayed because there’s uncertainty about how to go about doing the testing.”

Spiegel of Cedars-Sinai also helped co-found a new medical society called the American Medical Extended Reality Association in late 2022. He said it was created as a way for physicians, clinicians and other health-care professionals to help guide the future of the field. 

The society currently has about 300 paying members, a number Spiegel hopes will reach into the thousands in the coming years. It’s also gearing up to launch its first official peer-reviewed journal called the Journal of Medical Extended Reality. 

“This is not fringe science anymore. This is now mainstream,” Spiegel said. “There’s still a lot of work to do. It’s not like this is a done deal, cake’s not baked, but we’ve seen massive advances on many levels that make this a real science now.”

WATCH: Apple’s VR ‘blows away anything that we’ve ever seen,’ says analyst