The Perfect Stitch

When Professor Yan Liu was growing up in Changchun, China, her father wanted her to become a doctor like him. When she chose computer science, “I was a tiny bit disappointed,” Xiwen Liu, a retired anesthesiologist, acknowledges.

But their worlds collided in 2011, when the elder Liu was diagnosed with prostate cancer. It was a surprise. At 67, he was relatively young, had a healthy lifestyle and had no symptoms. Like many of the 1.2 million men diagnosed with prostate cancer each year, he had surgery to remove tumors in his prostate, but he suffered from incontinence, bleeding and infections for years after the surgery.

Nearly 6,000 miles away, at the USC Viterbi School of Engineering, Yan Liu felt helpless.

Even with access to the best doctors and hospitals, he experienced pain, slow recovery and long-term side effects.

Yan Liu

“The complications involved with prostate cancer surgery brought my dad significant personal challenges,” says Liu, a professor of computer science, electrical and computer engineering and biomedical sciences. “Even with access to the best doctors and hospitals, he experienced pain, slow recovery and long-term side effects.”

Liu resolved to use her skills to help others like her father. Working on research at the intersection of artificial intelligence and health care for more than a decade, she teamed up in 2016 with Dr. Andrew Hung, a urologist at Keck Medicine of USC, to create AI tools that measure and help improve surgeons’ technical skills during the radical prostatectomy procedure — the removal of the entire prostate gland.

By harnessing deep learning algorithms, their system learns from past movements to identify specific areas where a surgeon can improve during a robotic surgery. Together, Hung and Liu have since published more than 17 peer-reviewed papers in this field and recently received a $3 million award from the National Institutes of Health to advance their research.

They are working on developing an AI-based system to deliver real-time feedback during a procedure and even alert surgeons if they are at risk of erring. Their aim? To shorten the learning curve for surgeons, maximize patient safety and reduce postoperative complications.

“AI comes into play to assess the skill of the surgeon — to see what parts of the operation they are good at, and what needs to be improved,” says Liu, director of the USC Melady Lab, which focuses on machine learning with real-world applications. “It also comes into play during the simulation and learning stage for training new surgeons.

“Ultimately, what we want to do is provide real-time assistance to surgeons as they operate,” she says.

The Learning Curve

Removing the prostate gland through surgery is an option for men whose cancer has not spread. Today, robotically assisted radical prostatectomy constitutes about 85% of the 90,000 such operations performed every year in the United States.

Robotically assisted radical prostatectomy constitutes about 85% of the 90,000 such operations performed every year in the United States.

To operate using a robotic system, the surgeon makes tiny incisions and manipulates miniaturized instruments from a nearby console, making detailed work less invasive. As with any surgery, however, risk is involved: The surgeon must avoid damaging adjacent tissues and organs. Despite advances in medical technology, up to 40% of patients experience incontinence after the procedure.

Decades of research suggests that some hospitals and surgeons have significantly better outcomes than others. Indeed, the procedure has a steep learning curve: Studies say surgeons must perform about 100 radical prostatectomies to start reaching optimal outcomes. Yet surgeons often do not get a good sense of how they are performing.

“As surgeons in the operating room, we don’t get much immediate feedback in terms of the things we do,” says Hung, a leader in innovative surgical simulation technology and the director of the Center for Robotic Simulation and Education at Keck School of Medicine of USC. “Short of a major complication, some of the outcomes after prostate surgery are not apparent until weeks, months or years later.”

While surgical performance assessment is a critical need, it’s not always available, Hung says. “It’s simply not scalable to have a perfect surgeon evaluator look at every surgeon’s operation and give feedback.”

How AI Can Help

During robotic surgery, every snip, clamp and stitch generates massive amounts video data and kinematic data tracing the surgeon’s movements. AI can analyze this data to give surgeons feedback on instrument moving speed, distances traveled and wrist angulations during the robotic surgery. Using data and expertise from Hung and his group, Liu and her team have developed algorithms that teach the computer to learn as it is fed thousands of these data points.

Liu’s team uses this data to train the classifier to “understand what it sees,” she said. Once training is complete, deep learning models can provide objective evaluations based on learned representations of ideal versus non-ideal technical skills. To test the accuracy of the system, the machine’s ratings are compared with the surgeons’ ratings of the same surgeries.

“Based on this, we can predict whether the patient will have complications after the surgery and the expertise of each participant,” Liu says.

Using the raw data, Liu and her team determined that machine learning algorithms could predict each participant’s level of expertise with 87% accuracy. Eventually, she hopes the system will alert surgeons to potential problems, such as risk of injury to vital organs that could result in long-term complications.

“When we move toward the grand challenge goal of AI-assisted surgery, then we need to look at specific segments of the operation,” Liu says. “Then we can use this information to predict, based on the current stage of the operation, if there will be any risk factors for the next step so we can provide a timely warning.”

‘A Needle in a Haystack’

The process has allowed the researchers to identify the parts of the operation that seem most closely tied to outcomes. “It’s like a needle in a haystack,” Hung says. “How can you find the one needle that drives the outcomes, and what do you focus on in a procedure that’s between two and four hours long? AI brings objectivity and also scalability.”

AI brings objectivity and also scalability.

Andrew Hung

In a recent study focused on predicting urinary control outcomes, researchers found that algorithms home in on a critical step in the radical prostatectomy: when the clinician must suture a gap between the bladder and the urethra after the prostate is removed.

Done properly, this step prevents internal leakage of urine during and after the procedure. If it is incorrectly done, the patient can suffer such complications as incontinence and damage to the bladder.

Using AI, researchers found almost all the metrics that predict continence recovery were related to suturing, possibly because, Hung says, “measuring surgeon performance is nicely captured by the suturing task, and it lends itself well to evaluation.”

But while machine learning can help find the needle in the haystack, it doesn’t offer explanations — that’s still in the hands of the human experts.

Robots in the OR

When Hung and Liu started working together, they were looking at metrics that summarized an entire operation. Now they can analyze performance at the level of individual stitches, narrowing the focus to different levels of suturing. This would allow the system to give surgeons specific, actionable feedback. Their recent research results showed a 20% to 30% improvement in assessing surgical skills when using AI-assisted assessment compared with human graders.

Research results showed a 20% to 30% improvement in assessing surgical skills when using AI-assisted assessment compared with human graders.

“When you’re measuring technical skills and you provide that kind of feedback, it’s actually meaningful to surgeons,” Hung says. “As opposed to telling a surgeon, ‘You’re just not moving your hands fast enough,’ you’re actually telling them, ‘The way you’re holding the needles in this specific suture is incorrect.’”

The research is at the investigational stage, meaning the systems are not yet used in any high-stakes evaluations. But how do surgeons feel about a future of AI-assisted robots in the operating room?

“When I present this idea to my peers, I definitely get folks in both camps: those who embrace the idea and those who feel threatened,” Hung says. “In some cases, surgeons can be resistant to the concept at first. No one really likes being evaluated, least of all by a machine. But once they see the opportunity for evolving their skills and helping their patients, they become curious and open up to the idea.”

Although their exploration in understanding the “perfect stitch” has focused on urology, it could also be applied to other procedures, including hysterectomies and hernia repairs.

“We want to find out how surgeons can evolve more quickly,” says Hung. “[Improving] how we teach and train surgeons to do surgery will not cure disease magically, but it can certainly enhance how we anticipate patient outcomes and take better care of our patients.”

The final step is to prove that such feedback can improve outcomes and use it to train new surgeons. “Our goal is to use AI to help surgeons by detecting potential issues and [offering] warnings and possible suggestions in terms of what type of action the surgeon should take,” Liu says.

For Liu, the team’s progress brings hope that they can improve the lives of people like her father and the approximately 644,000 people who undergo some robotic surgery each year in the United States alone.

“In terms of translating AI into practice, I think this project has one of the shortest runways,” Liu says. “A very realistic goal that could happen in the next five to 10 years is this AI-assisted robotic surgery, which could improve post-surgery outcomes and recovery.”

As a patient, father and former clinician, Xiwen Liu welcomes that news.

“On nights when a patient would pass away during a surgery, I felt awful, always asking the what-if questions,” he says. “Improving health delivery means [everything] to patients, but it could even improve doctors’ psychological health, as well.

“Somehow, after all these years, our paths of medicine and computer science have crossed. Maybe [my daughter] found the best way to help people and save lives after all.”