Imaging techniques for robot assisted surgery

In previous articles we’ve looked at the history of surgical robots and what benefits they bring. As far as the technology behind them goes, a wide range of imaging techniques are being leveraged to bring state-of-the-art robotics to the operating room, and we’ve taken a closer look at some.

2D and 3D endoscopy

2D endoscopes have been in use for over a hundred years and are still one of the most basic yet effective forms of medical imaging. Nowadays, optical fibers are used inside a flexible tube, some carrying light into the body, others reflecting this light back out to form the image users rely on to identify anomalies in internal organs. 3D endoscopy has recently reached an even higher level of quality, adding depth perception to more complex procedures. However, so-called 3D imaging in medical applications often still relies on stereoscopic vision – the collection of two 2D images via the right and left eye, which are overlaid to encourage the brain to gauge relative distances and create the illusion of a 3D image with depth. A single stereoscopic endoscope is used in Da Vinci, for example.

Virtual healthcare

The use of Virtual Reality (VR) and Augmented Reality (AR) has become widely adopted in medical imaging, allowing surgeons to overlay data and graphical information on patient images. This helps with better understanding the position of organs, arteries and anomalies and increased surgical success. Microsoft’s HoloLens has even made it into the operating room, for example, in a system developed by Accenture’s Advanced Technology Center (ATC) in use in Mexico. The system collates surgical schedules, patient records, X-rays, CT and MRI scans with 3D models built by the systems from the scans. While operating, the scans and models are super-imposed on the surgical field to help determine the best point of entry to minimize recovery time and optimize outcomes for the patient.

AI for medical imaging

Talk of AI is abundant in medical fields but what does it mean for medical imaging? Machine learning-enhanced image classification is perhaps the most common AI tool in use in surgical robots. Systems are trained to recognize artifacts such as abnormal cells for cancer diagnoses, the individual tools used during procedures, or stationary objects in the room for object avoidance. This goes hand-in-hand with object tracking techniques so a robot can monitor blood flow or instrument positioning.

Added to this is a requirement for temporal understanding, so that the robot can understand the current state of an operation. In an attempt to reach an end goal of automated surgery, researchers from RSIP Vision have identified that surgical procedures need to be broken into separate segments, or phases, in order to teach robots how to operate. One way of knowing which phase a procedure is in is by recognizing the tools being used at a particular time – for example, a scalpel suggests an incision at an early point in the operation, a suture needle will be seen at the end. As well as being trained on each tool, a robotic system must also understand the positioning of that tool, for which stereoscopic vision was used. Deep learning platforms and convolutional neural networks were needed to train the robot, and both real-life data and synthetic data was used to ensure a wider range of anatomies and circumstances were included. While recognizing that fully automated robotic surgery is a long way off, the team have made inroads into increasing the accuracy and adaptability of robotics in the operating theater.

Medical images create digital twins

Digital twins are becoming commonplace in industrial development, but they’re now appearing in healthcare plans too. Being able to create a digital model of a patient, including precise positioning of organs and arteries, means that surgeons can plan more accurately for complex procedures and even conduct practice runs before getting to work on the real thing.

Sim&Cure have created Sim&Size – a software platform which uses patient-specific 3D modeling to assist surgeons and robots during surgery. The tool delivers real-time 3D artery reconstruction enabling surgeons to create a precise and personalized operating plan for treating brain aneurisms. We’ve seen in previous articles that brain surgery is one area benefitting from a great deal of research into robotic-assisted viability, mostly because the field requires extremely steady hand movements and tiny human tremors can have grave consequences in surgical outcomes. Combining digital twins to plan procedures with surgical robots to carry them out could improve results enormously in this sector.

Imaging capture and processing for surgical applications

Many of our imaging products are certified for used in medical environments. We custom design embedded vision systems and supply tailored frame grabbers for use in operating theaters, while our Harrier autofocus-zoom cameras are recording and live-streaming operations across the globe. Speak to our team for customized support in your robotic or surgical application and sign up to our newsletter to stay up to date with our latest product news.