Name of the participant: Thomas Pöllabauer
Description of the IT research project: Rendering in computer graphics refers to the mapping of a three-dimensional scene onto a two-dimensional plane. The process mimics the imaging process in real cameras, because there too we map the three-dimensional world onto a two-dimensional sensor. In computer vision, based on these two-dimensional images, we try to learn something about the three-dimensional world captured in them. The two processes are therefore two inverse problems.
The renaissance of neural networks has led to a wealth of very powerful architectures for computer vision, for example for image segmentation, object detection or object pose determination.
New architectures attempt to use differentiable renderings not only to detect objects, but also to reconstruct relevant properties such as material properties and geometry. In this way, a 3D object can be reconstructed from a single 2D image.
Federated learning refers to the joint solving of a learning problem. In particular, an attempt is made to learn cooperatively without sharing the database necessary for learning.
DIFFED-Pose (Differentiable Federated Pose) combines both approaches to reconstruct the 6D pose of objects in a distributed setting. For this purpose, a model pre-trained on renderings is re-trained on site. Differentiable rendering is used to automatically annotate the recorded images. This approach enables an improvement in the performance of the model compared to a model trained solely on synthetic images, without having to share potentially confidential image material.
Software Campus partners: Fraunhofer Institut für Graphische Datenverarbeitung (IGD), Software AG
Implementation period: 1.2.2021-31.1.2022
