Publications

Abstract:

Supervised training of neural networks requires large, diverse and well annotated data sets. In the medical field, this is often difficult to achieve due to constraints in time, expert knowledge and prevalence of an event. Artificial data augmentation can help to prevent overfitting and improve the detection of rare events as well as overall performance. However, most augmentation techniques use purely spatial transformations, which are not sufficient for video data with temporal correlations. In this paper, we present a novel methodology for workflow augmentation and demonstrate its benefit for event recognition in cataract surgery. The proposed approach increases the frequency of event alternation by creating artificial videos. The original video is split into event segments and a workflow graph is extracted from the original annotations. Finally, the segments are assembled into new videos based on the workflow graph. Compared to the original videos, the frequency of event alternation in the augmented cataract surgery videos increased by 26%. Further, a 3% higher classification accuracy and a 7.8% higher precision was achieved compared to a state-of-the-art approach. Our approach is particularly helpful to increase the occurrence of rare but important events and can be applied to a large variety of use cases.

Abstract:

Graphical visualization systems are a common clinical tool for displaying digital images and three-dimensional volumetric data. These systems provide a broad spectrum of information to support physicians in their clinical routine. For example, the field of radiology enjoys unrestricted options for interaction with the data, since information is pre-recorded and available entirely in digital form. However, some fields, such as microsurgery, do not benefit from this yet. Microscopes, endoscopes, and laparoscopes show the surgical site as it is. To allow free data manipulation and information fusion, 3D digitization of surgical sites is required. We aimed to find the number of cameras needed to add this functionality to surgical microscopes. For this, we performed in silico simulations of the 3D reconstruction of representative models of microsurgical sites with different numbers of cameras in narrow-baseline setups. Our results show that eight independent camera views are preferable, while at least four are necessary for a digital surgical site. In most cases, eight cameras allow the reconstruction of over 99% of the visible part. With four cameras, still over 95% can be achieved. This answers one of the key questions for the development of a prototype microscope. In future, such a system can provide functionality which is unattainable today

Abstract:

The fiber orientation in the atria has a significant contribution to the electrophysiologic behavior of the heart and to the genesis of arrhythmia. Atrial fiber orientation has a direct effect on excitation propagation, activation patterns and the P-wave. We present a rule-based algorithm that works robustly on different volumetric meshes composed of either isotropic hexahedra or arbitrary tetrahedra as well as on 3-dimensional triangular surface meshes in patient-specific geometric models. This method fosters the understanding of general pro-arrhythmic mechanisms and enhances patient-specific modeling approaches.

Abstract:

PURPOSE: Atrial fibrillation is one of the most frequent cardiac arrhythmias in the industrialized world and ablation therapy is the method of choice for many patients. However, ablation scars alter the electrophysiological activation and the mechanical behavior of the affected atria. Different ablation strategies with the aim to terminate atrial fibrillation and prevent its recurrence exist but their impact on the performance of the heart is often neglected. METHODS: In this work, we present a simulation study analyzing five commonly used ablation scar patterns and their combinations in the left atrium regarding their impact on the pumping function of the heart using an electromechanical whole-heart model. We analyzed how the altered atrial activation and increased stiffness due to the ablation scars affect atrial as well as ventricular contraction and relaxation. RESULTS: We found that systolic and diastolic function of the left atrium is impaired by ablation scars and that the reduction of atrial stroke volume of up to 11.43% depends linearly on the amount of inactivated tissue. Consequently, the end-diastolic volume of the left ventricle, and thus stroke volume, was reduced by up to 1.4 and 1.8%, respectively. During ventricular systole, left atrial pressure was increased by up to 20% due to changes in the atrial activation sequence and the stiffening of scar tissue. CONCLUSION: This study provides biomechanical evidence that atrial ablation has acute effects not only on atrial contraction but also on ventricular performance. Therefore, the position and extent of ablation scars is not only important for the termination of arrhythmias but is also determining long-term pumping efficiency. If confirmed in larger cohorts, these results have the potential to help tailoring ablation strategies towards minimal global cardiovascular impairment.

Abstract:

Objective: Diagnosis of craniosynostosis using photogrammetric 3D surface scans is a promising radiation-free alternative to traditional computed tomography. We propose a 3D surface scan to 2D distance map conversion enabling the usage of the first convolutional neural networks (CNNs)-based classification of craniosynostosis. Benefits of using 2D images include preserving patient anonymity, enabling data augmentation during training, and a strong under-sampling of the 3D surface with good classification performance.Methods: The proposed distance maps sample 2D images from 3D surface scans using a coordinate transformation, ray casting, and distance extraction. We introduce a CNNbased classification pipeline and compare our classifier to alternative approaches on a dataset of 496 patients. We investigate into low-resolution sampling, data augmentation, and attribution mapping.Results: Resnet18 outperformed alternative classifiers on our dataset with an F1-score of 0.964 and an accuracy of 98.4 %. Data augmentation on 2D distance maps increased performance for all classifiers. Under-sampling allowed 256-fold computation reduction during ray casting while retaining an F1-score of 0.92. Attribution maps showed high amplitudes on the frontal head.Conclusion: We demonstrated a versatile mapping approach to extract a 2D distance map from the 3D head geometry increasing classification performance, enabling data augmentation during training on 2D distance maps, and the usage of CNNs. We found that low-resolution images were sufficient for a good classification performance.Significance: Photogrammetric surface scans are a suitable craniosynostosis diagnosis tool for clinical practice. Domain transfer to computed tomography seems likely and can further contribute to reducing ionizing radiation exposure for infants.

Abstract:

Craniosynostosis is a congenital disease character-ized by the premature closure of one or multiple sutures of theinfant’s skull. For diagnosis, 3D photogrammetric scans are aradiation-free alternative to computed tomography. However,data is only sparsely available and the role of data augmentation for the classification of craniosynostosis has not yet beenanalyzed.In this work, we use a 2D distance map representation ofthe infants’ heads with a convolutional-neural-network-basedclassifier and employ a generative adversarial network (GAN)for data augmentation. We simulate two data scarcity scenar-ios with 15 % and 10 % training data and test the influence ofdifferent degrees of added synthetic data and balancing under-represented classes. We used total accuracy and F1-score as ametric to evaluate the final classifiers.For 15 % training data, the GAN-augmented dataset showedan increased F1-score up to 0.1 and classification accuracy upto 3 %. For 10 % training data, both metrics decreased. We present a deep convolutional GAN capable of creatingsynthetic data for the classification of craniosynostosis. Us-ing a moderate amount of synthetic data using a GAN showedslightly better performance, but had little effect overall. Thesimulated scarcity scenario of 10 % training data may havelimited the model’s ability to learn the underlying data distribution.

Abstract:

Background: Craniosynostosis is a condition caused by the premature fusion of skull sutures, leading to irregular growth patterns of the head. Three-dimensional photogrammetry is a radiation-free alternative to the diagnosis using computed tomography. While statistical shape models have been proposed to quantify head shape, no shape-model-based classification approach has been presented yet. Methods: We present a classification pipeline that enables an automated diagnosis of three types of craniosynostosis. The pipeline is based on a statistical shape model built from photogrammetric surface scans. We made the model and pathology-specific submodels publicly available, making it the first publicly available craniosynostosis-related head model, as well as the first focusing on infants younger than 1.5 years. To the best of our knowledge, we performed the largest classification study for craniosynostosis to date. Results: Our classification approach yields an accuracy of 97.8 %, comparable to other state-of-the-art methods using both computed tomography scans and stereophotogrammetry. Regarding the statistical shape model, we demonstrate that our model performs similar to other statistical shape models of the human head. Conclusion: We present a state-of-the-art shape-model-based classification approach for a radiation-free diagnosis of craniosynostosis. Our publicly available shape model enables the assessment of craniosynostosis on realistic and synthetic data.

Abstract:

Cranio-maxillofacial surgery often alters the aesthetics of the face which can be a heavy burden for patients to decide whether or not to undergo surgery. Today, physicians can predict the post-operative face using surgery planning tools to support the patient’s decision-making. While these planning tools allow a simulation of the post-operative face, the facial texture must usually be captured by another 3D texture scan and subsequently mapped on the simulated face. This approach often results in face predictions that do not appear realistic or lively looking and are therefore ill-suited to guide the patient’s decision-making. Instead, we propose a method using a generative adversarial network to modify a facial image according to a 3D soft-tissue estimation of the post-operative face. To circumvent the lack of available data pairs between pre- and post-operative measurements we propose a semi-supervised training strategy using cycle losses that only requires paired open-source data of images and 3D surfaces of the face’s shape. After training on “in-the-wild” images we show that our model can realistically manipulate local regions of a face in a 2D image based on a modified 3D shape. We then test our model on four clinical examples where we predict the post-operative face according to a 3D soft-tissue prediction of surgery outcome, which was simulated by a surgery planning tool. As a result, we aim to demonstrate the potential of our approach to predict realistic post-operative images of faces without the need of paired clinical data, physical models, or 3D texture scans.

Abstract:

This contribution is part of a project concerning the creation of an artificial dataset comprising 3D head scans of craniosynostosis patients for a deep-learning-based classification. To conform to real data, both head and neck are required in the 3D scans. However, during patient recording, the neck is often covered by medical staff. Simply pasting an arbitrary neck leaves large gaps in the 3D mesh. We therefore use a publicly available statistical shape model (SSM) for neck reconstruction. However, most SSMs of the head are constructed using healthy subjects, so the full head reconstruction loses the craniosynostosis-specific head shape. We propose a method to recover the neck while keeping the pathological head shape intact. We propose a Laplace- Beltrami-based refinement step to deform the posterior mean shape of the full head model towards the pathological head. The artificial neck is created using the publicly available Liverpool-Y ork-Model. W e apply our method to construct artificial necks for head scans of 50 scaphocephaly patients. Our method reduces mean vertex correspondence error by approximately 1.3 mm compared to the ordinary posterior mean shape, preserves the pathological head shape, and creates a continuous transition between neck and head. The presented method showed good results for reconstructing a plausible neck to craniosynostosis patients. Easily generalized it might also be applicable to other pathological shapes.

Abstract:

Electrocardiographic imaging (ECGI) strongly relies on a priori assumptions and additional information to overcome ill-posedness. The major challenge of obtaining good reconstructions consists in finding ways to add information that effectively restricts the solution space without violating properties of the sought solution. In this work, we attempt to address this problem by constructing a spatio-temporal basis of body surface potentials (BSP) from simulations of many focal excitations. Measured BSPs are projected onto this basis and reconstructions are expressed as linear combinations of corresponding transmembrane voltage (TMV) basis vectors. The novel method was applied to simulations of 100 atrial ectopic foci with three different conduction velocities. Three signal-to-noise ratios (SNR) and bases of six different temporal lengths were considered. Reconstruction quality was evaluated using the spatial correlation coefficient of TMVs as well as estimated local activation times (LAT). The focus localization error was assessed by computing the geodesic distance between true and reconstructed foci. Compared with an optimally parameterized Tikhonov-Greensite method, the BSP basis reconstruction increased the mean TMV correlation by up to 22, 24, and 32% for an SNR of 40, 20, and 0 dB, respectively. Mean LAT correlation could be improved by up to 5, 7, and 19% for the three SNRs. For 0 dB, the average localization error could be halved from 15.8 to 7.9 mm. For the largest basis length, the localization error was always below 34 mm. In conclusion, the new method improved reconstructions of atrial ectopic activity especially for low SNRs. Localization of ectopic foci turned out to be more robust and more accurate. Preliminary experiments indicate that the basis generalizes to some extent from the training data and may even be applied for reconstruction of non-ectopic activity.

Abstract:

Current surgical microscope systems have excellent optical properties but still involve some limitations. A future fully digital surgical microscope may overcome some major limitations of typical optomechanical systems, like ergonomic restrictions or limited number of observers. Furthermore, it can leverage and provide the full potential of digital reality. To achieve this, the frontend, the reconstruction of the digital twin of the surgical scenery, as well as the backend, the 3-D visualization interface for the surgeon, need to work in real-time. To investigate the visualization chain, we developed a virtual reality environment allowing pretesting this new form of 3-D data presentation. In this study, we wanted to answer the following question: How must the visualization pipeline look like to achieve a real-time update of the 3-D digital reality scenery. With our current approach, we were able to obtain visualizations with a frame rate of 120 frames per second and a 3-D data update rate of approximately 90 datasets per second. In a further step, a first prototype of a real-time mixed-reality head mounted visualization system could be manufactured based on the knowledge gained during the virtual reality pretesting.

Abstract:

Contemporary surgical microscope systems have excellent optical properties but some desirable features re- main unavailable. The number of co-observers is currently re- stricted, by spatial and optical limitations, to only two. More- over, ergonomics poses are a problem: Current microscope systems impede free movement and sometimes demand that surgeons take uncomfortable postures over long periods of time. To rectify these issues, some companies developed surgi- cal microscope systems based on a streaming approach. These systems remove some of the limitations. Multi-observer po- sitions, for example, are not independent from each other, for example. In order to overcome the aforementioned limitations, we are currently developing an approach for the next genera- tion of surgical microscope: Namely the fully digital surgi- cal microscope, where the current observation system is re- placed with a camera array, allowing real-time 3D reconstruc- tion of surgical scenes and, consequently, the rendering of al- most unlimited views for multiple observers. These digital mi- croscopes could make the perspective through the microscope unnecessary allowing the surgeon to move freely and work in more comfortable postures. The requirements on the camera array in such a system have to be determined. For this purpose, we propose of estimation the minimal number of cameras and their positions needed for the 3D reconstruction of microsurgi- cal scenes. The method of estimation is based on the require- ments for the 3D reconstruction. Within the MATLAB simu- lation environment, we have developed a 3D model of a mi- crosurgical scene, used for the determination of the number of required cameras. In a next step a small, compact and cost- ef cient s ystem w ith f ew o pto-mechanical c omponents could be manufactured.

Abstract:

Nowadays, the surgical microscope is the goldstandard for microsurgical procedures. Additional functionalities such as surgical navigation, data injection or imageoverlay are providing additional valuable information to the surgeon. For substituting the conventional optical system by a fully-digital multi-camera setup the three dimensional (3D) reconstruction of the scenery in the field of view is required. However, for in camera-based systems, an exact alignment of the cameras is a challenging task. Therefore, a final adjustment through a digital image rotation becomes necessary. Even though the digital rotation is a commonly used procedure, it leads to unavoidable errors because of the discretized grid of the image. Previous research reported in literature has demonstrated that the method of digitally rotating the images combined with the Fourier interpolation delivers the results of best quality. Nevertheless, the performance evaluation of this algorithm was carried out rotating an image in multiple threestep rotations to a total of 90 or 180 degrees and comparing it to the original image rotated in one step. This is a valid approach because a rotation of 90 or 180 degrees does not produce rotation artifacts. In this research project, we verify the performance of the three-step rotation algorithm using recorded images for which the test pattern was rotated as ground truth. A series of photographs with a rotation angle of 3 to 45 degrees was created. The advantage of this setup is that the result of the digital rotation can be directly compared to the recorded image. In addition, with the knowledge obtained about the interpolation error, we can improve pixel matching in the further triangulation used for 3D reconstruction. By doing so, the estimation of the interpolation error helps to reduce the triangulation error.

Abstract:

Future fully digital surgical visualization systems enable a wide range of new options. Caused by optomechanical limitations a main disadvantage of todays surgical microscopes is their incapability of providing arbitrary perspectives to more than two observers. In a fully digital microscopic system, multiple arbitrary views can be generated from a 3D reconstruction. Modern surgical microscopes allow replacing the eyepieces by cameras in order to record stereoscopic videos. A reconstruction from these videos can only contain the amount of detail the recording camera system gathers from the scene. Therefore, covered surfaces can result in a faulty reconstruction for deviating stereoscopic perspectives. By adding cameras recording the object from different angles, additional information of the scene is acquired, allowing to improve the reconstruction. Our approach is to use a fixed four-camera setup as a front-end system to capture enhanced 3D topography of a pseudo-surgical scene. This experimental setup would provide images for the reconstruction algorithms and generation of multiple observing stereo perspectives. The concept of the designed setup is based on the common main objective (CMO) principle of current surgical microscopes. These systems are well established and optically mature. Furthermore, the CMO principle allows a more compact design and a lowered effort in calibration than cameras with separate optics. Behind the CMO four pupils separate the four channels which are recorded by one camera each. The designed system captures an area of approximately 28mm × 28mm with four cameras. Thus, allowing to process images of 6 different stereo perspectives. In order to verify the setup, it is modelled in silico. It can be used in further studies to test algorithms for 3D reconstruction from up to four perspectives and provide information about the impact of additionally recorded perspectives on the enhancement of a reconstruction.