In this interview we’ll introduce you to our Software Campus alumnus Dr. Moritz Fuchs and participant Yannik Frisch. They were/are pursuing their doctorates at TU Darmstadt in the field of computer vision and are researching on how to create synthetic data sets for highly realistic simulations that can be used for cataract surgery training.
Dr. Moritz Fuchs led the project UC4S2R: AI-based transfer between simulation and real eye surgery supported by uncertainty analysis.
Yannik Frisch is leading the GRAPH-SIM project: Combining generative models and knowledge graph representations for realistic physics-based simulation of cataract surgery.
For their Software Campus projects, both were and are collaborating with ZEISS as an industry partner. What makes these projects special is that Yannik had already worked with Moritz on another project as a master’s student and later on, as a research assistant for Moritz’s SWC project. Now, he is pursuing and leading this as his own Software Campus project as a PhD student, but with a different focus.
Moritz, how was the search for you to find employees for your Software Campus project?
Moritz: That was not easy at all, since it was during Corona times. The job advertisement was not really successful, which is why I had to actively approach people and ask them if they’re interested in doing a PhD. Afterward, I had to see whether the candidates would be a suitable match or not and if they were bringing applicable experiences to the table. In that exact year, Yannik already worked with us as a research assistant. This is why I already knew that he had gained experience with generative models, which was something no other could offer. That is why it was possible to include him early on, even before we submitted our final version of the project description.
Yannik, what exactly is your research all about, and how does it build on Moritz’s project?
Yannik: Moritz’s project is the foundation for the research in my dissertation. Certainly, there are limitations, since we did create a simulation for surgical data [1, 2, 3]. However, if you compare this with other simulation techniques, there is still relatively little control over what really is generated, even if it looks very realistic. My research addresses a lot of these questions: How can we reinstate this controlling aspect? How can we simulate what we really want? Certainly, there are many other interesting fields of research that are being addressed in my Software Campus project, for example: How can we simulate consistent and complete videos of these kinds of surgical procedures [4, 5]? How can we tackle other medical domains? How can we benefit from what we have already discovered and perhaps improve it further?
How has the collaboration with your partner ZEISS been for the two of you so far?
Moritz: From my perspective, it has always been more than positive. Generally speaking, it was constructive but also open to new ideas. Early on, you’ll receive feedback from ZEISS, which really is a pleasant thing. Ideas will be challenged and stand to the test.
Before handing in our final project description, the content had to stand the test one more time. That’s when we realized a few changes must be made. An example is: We were sure that generative models were already pretty good in generating realistic images, which can’t be distinguished from real pictures. Therefore, our idea was to use the models’ assets and make them more measurable and controllable by using uncertainty estimation.
But we did not have an effective control-mechanism to prove that the generated outcome is really what the uncertainty estimation said in advance. So, we did reschedule our project a little bit to prove whether we’re able to transfer motions from one video to another effectively or not—by that I mean adapting the control of motion within videos between different modalities. For another paper, we did generate targeted tools and different surgery phases. Afterward, we did use the uncertainty estimation to automate all this, which is published on WAVC now. The controllability of this entire process was further enhanced by Yannik’s project.
Yannik: I was also very positively surprised by Ghazal Ghazaei’s commitment and contact with Zeiss. With bi-weekly meetings, we had a very active exchange. Of course, what helps a lot is that Ghazal works in a related field and can bring her experience and new ideas in very well.
How was the transition of these projects for both of you? Were there any challenges you had to overcome during the organization? Could you share some of your learnings?
Moritz: I always tried to support Yannik as effectively as possible with mentoring, sharing ideas or giving feedback regarding his ideas, etc. To put it in a nutshell, it all started with the facilitation for the project application, but also to coordinate the collaboration with ZEISS.
Yannik: What helped me a lot was the constant exchange and knowing that I had a contact person in case of follow-up questions. We also shared an office, which was really helpful.
I found it difficult to separate my own research from the previous Software Campus project—which I am now supervising—and to give it its own direction. But I think, together, we managed it quite well. And, as Moritz has already mentioned, finding and selecting new candidates was really problematic, also for me. For example, there were two to three candidates who could have been a very good match, but they were understandably disappointed when they received their rejections. I realized that clear and transparent communication helped ease the situation because we had a candidate with a medical background who was most suitable for the project.
Yannik, soon your project is coming to an end. Can you share some insights from your change of perspective from your initial experiences as a research assistant and the subsequent transition to becoming the project lead?
Yannik: It was quite a change going from a master’s student to a doctoral candidate. But overall, I had a very positive experience, especially thanks to the ongoing mentoring provided by Moritz and my doctoral supervisor, Dr Anirban Mukhopadhyay. This continuous feedback was very helpful, and we were ultimately able to achieve good results very early on, and as a research fellow, I was able to gain a lot of insight into AI research and development. What I really liked about my dissertation was the collaboration with the clinical staff. And, of course, the extensive networking at conferences, where I was able to learn a lot and build my own profile. In addition, the collaboration with industry through the Software Campus project was also very special.
Regarding the transition, I have noticed that the entire process requires a change of perspective. At the beginning, I was the person who always had someone giving me continuous feedback. Now, I have to be that person myself, but I think the mentoring has prepared me well, and the Software Campus training is also a useful additional support. You’ll have to ask Ssharvien Kumar Sivakumar next year whether it worked out or not, but in general, I would say that everything went very smoothly for us.
Do you have general suggestions for future Software Campus participants?
Moritz: Start as soon as possible! Once you successfully managed the first phases of application—even before having a collaborating partner from the industry—you can already start publishing your job advertisement. Also check whether candidates are suitable for your project, based on their experiences, and approach them. Additionally, think about which expectations you have for the person who would work for that specific position. Granted, for this project it is special: eye surgery is not everyone’s cup of tea. If someone can’t stand blood, then it’s not really helpful.
Click here, to read the full interview.
Source language of this interview: German
