Publications

Abstract:

Graphical visualization systems are a common clinical tool for displaying digital images and three-dimensional volumetric data. These systems provide a broad spectrum of information to support physicians in their clinical routine. For example, the field of radiology enjoys unrestricted options for interaction with the data, since information is pre-recorded and available entirely in digital form. However, some fields, such as microsurgery, do not benefit from this yet. Microscopes, endoscopes, and laparoscopes show the surgical site as it is. To allow free data manipulation and information fusion, 3D digitization of surgical sites is required. We aimed to find the number of cameras needed to add this functionality to surgical microscopes. For this, we performed in silico simulations of the 3D reconstruction of representative models of microsurgical sites with different numbers of cameras in narrow-baseline setups. Our results show that eight independent camera views are preferable, while at least four are necessary for a digital surgical site. In most cases, eight cameras allow the reconstruction of over 99% of the visible part. With four cameras, still over 95% can be achieved. This answers one of the key questions for the development of a prototype microscope. In future, such a system can provide functionality which is unattainable today

Abstract:

The fiber orientation in the atria has a significant contribution to the electrophysiologic behavior of the heart and to the genesis of arrhythmia. Atrial fiber orientation has a direct effect on excitation propagation, activation patterns and the P-wave. We present a rule-based algorithm that works robustly on different volumetric meshes composed of either isotropic hexahedra or arbitrary tetrahedra as well as on 3-dimensional triangular surface meshes in patient-specific geometric models. This method fosters the understanding of general pro-arrhythmic mechanisms and enhances patient-specific modeling approaches.

Abstract:

Cranio-maxillofacial surgery often alters the aesthetics of the face which can be a heavy burden for patients to decide whether or not to undergo surgery. Today, physicians can predict the post-operative face using surgery planning tools to support the patient’s decision-making. While these planning tools allow a simulation of the post-operative face, the facial texture must usually be captured by another 3D texture scan and subsequently mapped on the simulated face. This approach often results in face predictions that do not appear realistic or lively looking and are therefore ill-suited to guide the patient’s decision-making. Instead, we propose a method using a generative adversarial network to modify a facial image according to a 3D soft-tissue estimation of the post-operative face. To circumvent the lack of available data pairs between pre- and post-operative measurements we propose a semi-supervised training strategy using cycle losses that only requires paired open-source data of images and 3D surfaces of the face’s shape. After training on “in-the-wild” images we show that our model can realistically manipulate local regions of a face in a 2D image based on a modified 3D shape. We then test our model on four clinical examples where we predict the post-operative face according to a 3D soft-tissue prediction of surgery outcome, which was simulated by a surgery planning tool. As a result, we aim to demonstrate the potential of our approach to predict realistic post-operative images of faces without the need of paired clinical data, physical models, or 3D texture scans.

Abstract:

Electrocardiographic imaging (ECGI) strongly relies on a priori assumptions and additional information to overcome ill-posedness. The major challenge of obtaining good reconstructions consists in finding ways to add information that effectively restricts the solution space without violating properties of the sought solution. In this work, we attempt to address this problem by constructing a spatio-temporal basis of body surface potentials (BSP) from simulations of many focal excitations. Measured BSPs are projected onto this basis and reconstructions are expressed as linear combinations of corresponding transmembrane voltage (TMV) basis vectors. The novel method was applied to simulations of 100 atrial ectopic foci with three different conduction velocities. Three signal-to-noise ratios (SNR) and bases of six different temporal lengths were considered. Reconstruction quality was evaluated using the spatial correlation coefficient of TMVs as well as estimated local activation times (LAT). The focus localization error was assessed by computing the geodesic distance between true and reconstructed foci. Compared with an optimally parameterized Tikhonov-Greensite method, the BSP basis reconstruction increased the mean TMV correlation by up to 22, 24, and 32% for an SNR of 40, 20, and 0 dB, respectively. Mean LAT correlation could be improved by up to 5, 7, and 19% for the three SNRs. For 0 dB, the average localization error could be halved from 15.8 to 7.9 mm. For the largest basis length, the localization error was always below 34 mm. In conclusion, the new method improved reconstructions of atrial ectopic activity especially for low SNRs. Localization of ectopic foci turned out to be more robust and more accurate. Preliminary experiments indicate that the basis generalizes to some extent from the training data and may even be applied for reconstruction of non-ectopic activity.

Abstract:

Current surgical microscope systems have excellent optical properties but still involve some limitations. A future fully digital surgical microscope may overcome some major limitations of typical optomechanical systems, like ergonomic restrictions or limited number of observers. Furthermore, it can leverage and provide the full potential of digital reality. To achieve this, the frontend, the reconstruction of the digital twin of the surgical scenery, as well as the backend, the 3-D visualization interface for the surgeon, need to work in real-time. To investigate the visualization chain, we developed a virtual reality environment allowing pretesting this new form of 3-D data presentation. In this study, we wanted to answer the following question: How must the visualization pipeline look like to achieve a real-time update of the 3-D digital reality scenery. With our current approach, we were able to obtain visualizations with a frame rate of 120 frames per second and a 3-D data update rate of approximately 90 datasets per second. In a further step, a first prototype of a real-time mixed-reality head mounted visualization system could be manufactured based on the knowledge gained during the virtual reality pretesting.

Abstract:

Contemporary surgical microscope systems have excellent optical properties but some desirable features re- main unavailable. The number of co-observers is currently re- stricted, by spatial and optical limitations, to only two. More- over, ergonomics poses are a problem: Current microscope systems impede free movement and sometimes demand that surgeons take uncomfortable postures over long periods of time. To rectify these issues, some companies developed surgi- cal microscope systems based on a streaming approach. These systems remove some of the limitations. Multi-observer po- sitions, for example, are not independent from each other, for example. In order to overcome the aforementioned limitations, we are currently developing an approach for the next genera- tion of surgical microscope: Namely the fully digital surgi- cal microscope, where the current observation system is re- placed with a camera array, allowing real-time 3D reconstruc- tion of surgical scenes and, consequently, the rendering of al- most unlimited views for multiple observers. These digital mi- croscopes could make the perspective through the microscope unnecessary allowing the surgeon to move freely and work in more comfortable postures. The requirements on the camera array in such a system have to be determined. For this purpose, we propose of estimation the minimal number of cameras and their positions needed for the 3D reconstruction of microsurgi- cal scenes. The method of estimation is based on the require- ments for the 3D reconstruction. Within the MATLAB simu- lation environment, we have developed a 3D model of a mi- crosurgical scene, used for the determination of the number of required cameras. In a next step a small, compact and cost- ef cient s ystem w ith f ew o pto-mechanical c omponents could be manufactured.

Abstract:

Nowadays, the surgical microscope is the goldstandard for microsurgical procedures. Additional functionalities such as surgical navigation, data injection or imageoverlay are providing additional valuable information to the surgeon. For substituting the conventional optical system by a fully-digital multi-camera setup the three dimensional (3D) reconstruction of the scenery in the field of view is required. However, for in camera-based systems, an exact alignment of the cameras is a challenging task. Therefore, a final adjustment through a digital image rotation becomes necessary. Even though the digital rotation is a commonly used procedure, it leads to unavoidable errors because of the discretized grid of the image. Previous research reported in literature has demonstrated that the method of digitally rotating the images combined with the Fourier interpolation delivers the results of best quality. Nevertheless, the performance evaluation of this algorithm was carried out rotating an image in multiple threestep rotations to a total of 90 or 180 degrees and comparing it to the original image rotated in one step. This is a valid approach because a rotation of 90 or 180 degrees does not produce rotation artifacts. In this research project, we verify the performance of the three-step rotation algorithm using recorded images for which the test pattern was rotated as ground truth. A series of photographs with a rotation angle of 3 to 45 degrees was created. The advantage of this setup is that the result of the digital rotation can be directly compared to the recorded image. In addition, with the knowledge obtained about the interpolation error, we can improve pixel matching in the further triangulation used for 3D reconstruction. By doing so, the estimation of the interpolation error helps to reduce the triangulation error.

Abstract:

Future fully digital surgical visualization systems enable a wide range of new options. Caused by optomechanical limitations a main disadvantage of todays surgical microscopes is their incapability of providing arbitrary perspectives to more than two observers. In a fully digital microscopic system, multiple arbitrary views can be generated from a 3D reconstruction. Modern surgical microscopes allow replacing the eyepieces by cameras in order to record stereoscopic videos. A reconstruction from these videos can only contain the amount of detail the recording camera system gathers from the scene. Therefore, covered surfaces can result in a faulty reconstruction for deviating stereoscopic perspectives. By adding cameras recording the object from different angles, additional information of the scene is acquired, allowing to improve the reconstruction. Our approach is to use a fixed four-camera setup as a front-end system to capture enhanced 3D topography of a pseudo-surgical scene. This experimental setup would provide images for the reconstruction algorithms and generation of multiple observing stereo perspectives. The concept of the designed setup is based on the common main objective (CMO) principle of current surgical microscopes. These systems are well established and optically mature. Furthermore, the CMO principle allows a more compact design and a lowered effort in calibration than cameras with separate optics. Behind the CMO four pupils separate the four channels which are recorded by one camera each. The designed system captures an area of approximately 28mm × 28mm with four cameras. Thus, allowing to process images of 6 different stereo perspectives. In order to verify the setup, it is modelled in silico. It can be used in further studies to test algorithms for 3D reconstruction from up to four perspectives and provide information about the impact of additionally recorded perspectives on the enhancement of a reconstruction.