FMDeploy: Foundation Models on Resource-Constrained Devices

Name of participant: Benedikt Dietrich

Project’s name: FMDeploy: Foundation Models on Resource-Constrained Devices

Project description:

So-called foundation models are regarded as the most advanced AI models. Their use has led to considerable successes in recent years, for example in natural language processing using large language models (LLMs) and in computer vision. This success is largely due to the high number of parameters and capacity of these models, which exceed those of earlier architectures and result in increased resource consumption. For this reason, they are primarily used with high-performance graphics processing units. To ensure efficient operation in scenarios where resources are limited, such as in technical and industrial applications like robotics and autonomous driving, a holistic approach to optimisation is required that considers the application, the model and the executing hardware. Physical constraints here relate to computing power, memory, electrical power, energy consumption and temperature. Furthermore, embedded systems often feature a heterogeneous architecture with different processor cores and accelerators, a shared memory hierarchy, and competition between concurrently running computations for the limited resources.

The FMDeploy research project is dedicated to efficiently executing FMs on resource-constrained embedded systems. To this end, platform-specific approaches are pursued, encompassing processor modelling and the characterisation of model blocks at design time, as well as dynamic resource management at runtime through scheduling, frequency scaling and memory management, in order to make optimal use of the available hardware. Context-specific optimisations are also to be taken into account. Conventional methods always execute the same computational steps regardless of the input sequence. This approach can be inefficient as it always uses the maximum available resources, even when the input does not require the model’s full capacity. Context-dependent complexity arises, for example, in image processing with static or dynamic backgrounds, in changing weather conditions, or at different times of day and under varying lighting conditions. Furthermore, input data often contains redundant information or details that are irrelevant to the target application. This means that certain calculations can be skipped or that specific information does not need to be processed at all.

One aim of this micro-project is therefore to identify these potential savings and to develop suitable, context-specific optimisation methods. This may also involve using dynamic model architectures such as early-exit or mixture-of-experts networks, in which a different computation path is activated depending on the input. This allows resources to be saved for simple inputs without compromising application performance for complex inputs.

As dynamic architectures result in a context-dependent computational load, the study will also examine how context-specific optimisations can be combined with platform-specific approaches and dynamic resource management.

Software Campus Partner: KIT & Bosch

Implementation period: 01.01.2026 – 31.12.2027