Advancing Surgical Training and Education Through Technology

AR-Enabled Simulator for Total Hip Arthroplasty 

This initiative aims to elevate open surgery training beyond the traditional human specimen-based gold standard. Our solution seamlessly combines sawbone- and human specimen-based training with simulation through augmented reality. This results in the benefit of realistic anatomical visualization while saving both resources and time. This integrated solution, embedded in a didactic training concept, comprises three essential components: the physical training setup, the AR-based guidance/instruction module, and the performance evaluation module. The training setup, as shown in Figure 1, includes the AR glasses, sawbone/human specimen anatomy, surgical instruments, and prosthetic implants. This unique combination allows the haptic sensation and computer-generated information can be used to visualize soft tissue and possible surgical complications. 

A second pivotal feature is the training guidance provided by the AR application, which is used alongside the integrated performance evaluation. The trainee is AR-guided through the relevant steps of the intervention, while artificial intelligence (AI) is automatically identifying the different steps of the surgery. The level of support varies based on the skill level, ranging up to full navigation-like guidance, where for instance the exact position of a cut is displayed on the bone. Following training, performance is assessed automatically by evaluating surgical accuracy based on implant position, alongside tool, hand, and eye tracking data. This holistic approach aims to redefine open surgery training by seamlessly blending the advantages of cadaver-based realism with the technological precision of augmented reality.

Innovative Visualization

This focus aims to enhance the surgical training curriculum by integrating advanced visualization tools and learning materials. Moving beyond traditional teaching aids such as 2D illustrations and video recordings of surgical procedures, the initiative aims to develop photorealistic 3D reconstructions of critical surgical steps and detailed visualizations of essential anatomical structures. This enables surgeons to participate in a surgery virtually and to experience crucial steps in 3D. To realize this vision, we are establishing a comprehensive pipeline for data capture, processing, and visualization. This effort is the result of a collaborative partnership among Balgrist University Hospital, ETH Zurich, ZHAW, and Microsoft. 

First, utilizing the state-of-the-art OR-X facility, we are setting up an experimental multi-camera system (Fig. 2) designed to efficiently document surgical operations. This phase involves identifying the optimal hardware and configuration settings to achieve superior visualization quality with minimal disruption to the surgical workflow. Importantly, this capture system is designed with the capability for deployment in actual operating rooms. Following the capture phase, the recorded data undergoes processing with cutting-edge deep learning techniques to produce photorealistic 3D reconstructions of the surgical scenes (Fig. 3). These reconstructions support interactive user engagement, offering novel opportunities for enhancing spatial understanding of surgical procedures. This research segment, led by ETH Zurich, focuses on advancing and customizing generalpurpose 3D reconstruction and tracking for operating room settings.

In the final phase, the effectiveness of the innovative educational approach will be assessed by means of a user study conducted with surgical residents at Balgrist University Hospital.