Software Campus alumnus Dr. Moritz Fuchs was and participant Yannik Frisch is pursuing their doctorates at TU Darmstadt in the field of computer vision. They are researching how to create synthetic data sets for highly realistic simulations that can be used for cataract surgery trainings.
Dr. Moritz Fuchs led the project UC4S2R: AI-based transfer between simulation and real eye surgery supported by uncertainty analysis. This project aimed to generate highly realistic simulations that can be used to train surgeons. The iteration of uncertainty factors into the models enabled a form of autonomous training. Through iteration between image generation and training on uncertain or rare cases, the model could be continuously improved.
Yannik Frisch is leading the GRAPH-SIM project: Combining generative models and knowledge graph representations for realistic physics-based simulation of cataract surgery. He combines generative models, which ensure high-quality images, with physics-based logic in the form of graphs, which enable a more realistic simulation of interactions. This allows the establishment of a controllable and realistic simulation for assistance systems and surgeons in training.
For their Software Campus projects, both were and are collaborating with ZEISS as an industry partner. What makes these projects special is that Yannik had already worked with Moritz on another project as a master's student and later on, as a research assistant for Moritz's SWC project. Now, he is pursuing and leading this as his own Software Campus project as a PhD student, but with a different focus.
We spoke to them about their collaboration and changing perspectives. The interview was conducted by Tiziana Maglia and Susanne Kegler.
Moritz: What was your reason behind researching within the field of computer vision, and more specifically, on your project “UC4S2R: Uncertainty Estimation for Sim2Real Eye Surgery”?
Moritz: Overall, the motivation behind why I chose the Software Campus and this framework was that it allowed me to build a bridge between research and putting it into practice. Which, in my opinion, does not occur often during academic research.
During my dissertation, I did raise the question: How can the reliability of AI be increased in different models? At the same time, I have already gained a lot of experience with image-generative models during my master’s program. Back then, it wasn’t as good as it is right now—back then the generative models were only applied to “simpler” problems, such as generating novel smileys. That’s when the initial thought of doing something within the frame of uncertainty estimation crossed my mind to close the gap between simulation-to-real in surgery. Simulation-to-real means that there is a given simulation-environment and an AI-model that transforms simulations into realistic outputs—so that it’s actually difficult to distinguish the simulation from reality.
The aim was to keep this gap as small as possible so that surgeons have the possibility to train with a simulation without the need for direct contact with patients or plastic phantoms. Those phantoms are good alternatives, but still not close enough to reality.
Within the field of cataractsurgery—a type of eye surgery—, extensive public data sets were provided, which additionally included phases and tool annotation, as well as segmentation. Therefore, there were many downstream tasks in which generative models were evaluated and published, but they also offered adequate image data of good quality. All this is not easy to have access to; that is why I primarily chose this topic, and ZEISS did simultaneously research on this as well. Additionally, we were lucky to have Dr. Anirban Mukhopadhyay as a doctoral father since he did already research on every—to us relevant—topic. Thus, he was able to consult us, give us some valuable feedback and ideas, and connect things to one another. Apart from that, the exchange with Zeiss was really good and honest, also in regard to early adjustments within the project.
How was the search for you to find employees for your Software Campus project?
Moritz: That was not easy at all, since it was during Corona times. The job advertisement was not really successful, which is why I had to actively approach people and ask them if they’re interested in doing a PhD. Afterward, I had to see whether the candidates would be a suitable match or not and if they were bringing applicable experiences to the table. In that exact year, Yannik already worked with us as a research assistant. This is why I already knew that he had gained experience with generative models, which was something no other could offer. That is why it was possible to include him early on, even before we submitted our final version of the project description.
Yannik: Did you aim to pursue a doctor’s degree for a long time, or did you come across this idea during the collaboration with Moritz?
Yannik: I knew early on in my studies that I wanted to do research afterward in the field of AI. I did study cognitive science, and Moritz's offer in the field of medical imaging allowed me to combine my two interests in human science—medicine in this case—and technology, or more specifically, AI. So, I was very happy to be invited to work on this project. Medical imaging is one of the most ambitious goals I can imagine for myself because I'm researching technology that might help people one day. I also hope that my PhD will give me even more opportunities to pursue the path to a scientific leadership position. For instance, the training courses provided by the Software Campus are a great support for me.
What exactly is your research all about, and how does it build on Moritz's project?
Yannik: Moritz's project is the foundation for the research in my dissertation. Certainly, there are limitations, since we did create a simulation for surgical data [1, 2, 3]. However, if you compare this with other simulation techniques, there is still relatively little control over what really is generated, even if it looks very realistic. My research addresses a lot of these questions: How can we reinstate this controlling aspect? How can we simulate what we really want? Certainly, there are many other interesting fields of research that are being addressed in my Software Campus project, for example: How can we simulate consistent and complete videos of these kinds of surgical procedures [4, 5]? How can we tackle other medical domains? How can we benefit from what we have already discovered and perhaps improve it further?
How has the collaboration with your partner ZEISS been for the two of you so far?
Moritz: From my perspective, it has always been more than positive. Generally speaking, it was constructive but also open to new ideas. Early on, you’ll receive feedback from ZEISS, which really is a pleasant thing. Ideas will be challenged and stand to the test.
Before handing in our final project description, the content had to stand the test one more time. That’s when we realized a few changes must be made. An example is: We were sure that generative models were already pretty good in generating realistic images, which can’t be distinguished from real pictures. Therefore, our idea was to use the models’ assets and make them more measurable and controllable by using uncertainty estimation.
But we did not have an effective control-mechanism to prove that the generated outcome is really what the uncertainty estimation said in advance. So, we did reschedule our project a little bit to prove whether we’re able to transfer motions from one video to another effectively or not [1]—by that I mean adapting the control of motion within videos between different modalities. For another paper, we did generate targeted tools and different surgery phases [3]. Afterward, we did use the uncertainty estimation [2] to automate all this, which is published on WAVC now. The controllability of this entire process was further enhanced by Yannik’s project [4, 5].
Yannik: I was also very positively surprised by Ghazal Ghazaei's commitment and contact with Zeiss. With bi-weekly meetings, we had a very active exchange. Of course, what helps a lot is that Ghazal works in a related field and can bring her experience and new ideas in very well.
Since you worked with medical specialists, did they work in different medical areas and hospitals?
Yannik: Most of them were ophthalmologists from Mainz who are experts in cataract surgery. For a publication, we also consulted other doctors and therefore were able to notice the difference between how well the eye surgery experts can separate the simulation from real data and how well doctors working in a different field could.
Have you noticed significant differences in your evaluation?
Yannik: Yes, to a minimal extent. They were not mathematically significant, though, but there were slight trends.
Moritz: Generally speaking, it is extremely difficult to detect what is AI-generated and what is not. Therefore, even for doctors, it is not an easy task.
How was the transition of these projects for both of you? Were there any challenges you had to overcome during the organization? Could you share some of your learnings?
Moritz: I always tried to support Yannik as effectively as possible with mentoring, sharing ideas or giving feedback regarding his ideas, etc. To put it in a nutshell, it all started with the facilitation for the project application, but also to coordinate the collaboration with ZEISS.
Yannik: What helped me a lot was the constant exchange and knowing that I had a contact person in case of follow-up questions. We also shared an office, which was really helpful.
I found it difficult to separate my own research from the previous Software Campus project—which I am now supervising—and to give it its own direction. But I think, together, we managed it quite well. And, as Moritz has already mentioned, finding and selecting new candidates was really problematic, also for me. For example, there were two to three candidates who could have been a very good match, but they were understandably disappointed when they received their rejections. I realized that clear and transparent communication helped ease the situation because we had a candidate with a medical background who was most suitable for the project.
Do you have general suggestions for future Software Campus participants?
Moritz: Start as soon as possible! Once you successfully managed the first phases of application—even before having a collaborating partner from the industry—you can already start publishing your job advertisement. Also check whether candidates are suitable for your project, based on their experiences, and approach them. Additionally, think about which expectations you have for the person who would work for that specific position. Granted, for this project it is special: eye surgery is not everyone’s cup of tea. If someone can’t stand blood, then it’s not really helpful.
Yannik, soon your project is coming to an end. Can you share some insights from your change of perspective from your initial experiences as a research assistant and the subsequent transition to becoming the project lead?
Yannik: It was quite a change going from a master's student to a doctoral candidate. But overall, I had a very positive experience, especially thanks to the ongoing mentoring provided by Moritz and my doctoral supervisor, Dr Anirban Mukhopadhyay. This continuous feedback was very helpful, and we were ultimately able to achieve good results very early on, and as a research fellow, I was able to gain a lot of insight into AI research and development. What I really liked about my dissertation was the collaboration with the clinical staff. And, of course, the extensive networking at conferences, where I was able to learn a lot and build my own profile. In addition, the collaboration with industry through the Software Campus project was also very special.
Regarding the transition, I have noticed that the entire process requires a change of perspective. At the beginning, I was the person who always had someone giving me continuous feedback. Now, I have to be that person myself, but I think the mentoring has prepared me well, and the Software Campus training is also a useful additional support. You'll have to ask Ssharvien Kumar Sivakumar next year whether it worked out or not, but in general, I would say that everything went very smoothly for us.
Do you already have a future scientist in the pipeline who is going to follow up on and dive deeper into this research? Or rather, do you know how this research’s and your future will look like in general?
Yannik: The topic is now being further pursued by Ssharvien. There are still many open research questions that can be worked on beyond the Software Campus project. Of course, I would like to continue working on this if applicable in industry or further as a postdoc. We'll see if that works out.
Moritz: Even though my field of research did develop towards pathology, in my opinion, cataract surgery is extremely underrepresented in academic research. There is almost no scientist working on that. Overall, surgery is not as represented as other diagnostic tasks within the medical imaging community. Besides, generative models are rarely used in surgery. I’d be really happy if more scientists started working in this field. Of course, there are several risks that make it easy to say: we don’t want anything to start hallucinating. But this is the exact reason why our research is working towards the question: How can we make it more controllable? How do we minimize the risks while obtaining realistic accuracy? Because we want surgeons to be able to train or be trained with a video game that allows a realistic procedure—as if they would do it in real life—without noticing a difference. One step is, for instance, the capsulorhexis for which a circular piece of “skin” is subtracted from the lens. During the simulated training with a phantom, they’re working with a thin plastic film which must be subtracted from a petri plate. Even though this procedure is motorically very close to reality, the received visual feedback is not even close to a real surgery.
Thank you very much for your time and this interesting insight into your research!
Source language of this interview: German
_____________________________________
Published papers:
[1] Frisch, Yannik, Moritz Fuchs, and Anirban Mukhopadhyay. "Temporally consistent sequence-to-sequence translation of cataract surgeries." International Journal of Computer Assisted Radiology and Surgery 18.7 (2023): 1217-1224.
[2] Y. Frisch, C. Bornberg, M. Fuchs and A. Mukhopadhyay, "GAUDA: Generative Adaptive Uncertainty-Guided Diffusion-Based Augmentation for Surgical Segmentation," in 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Tucson, AZ, USA, 2025, pp. 3762-3771, doi: 10.1109/WACV61041.2025.00370.
url: https://doi.ieeecomputersociety.org/10.1109/WACV61041.2025.00370
[3] Frisch, Yannik, et al. "Synthesising rare cataract surgery samples with guided diffusion models." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2023.
[4] Frisch, Yannik, et al. "SurGrID: Controllable Surgical Simulation via Scene Graph to Image Diffusion." arXiv preprint arXiv:2502.07945 (2025).
[5] Sivakumar, Ssharvien Kumar, et al. "SASVi--Segment Any Surgical Video." arXiv preprint arXiv:2502.09653 (2025).
