Mentor: Dominik Hildebrand
Email: Dominik.Hildebrandspam prevention@uni-tuebingen.de
Large Language Models (LLMs) such as “ChatGPT” can quickly turn huge amounts of text into clear and helpful responses such as when you need to draft an email, translate a paragraph, or provide a quick summary. Thus, they are becoming a larger and larger part of our everyday lives by making everyday tasks faster and easier.
However, LLMs - as their name suggests - are indeed large with parameter counts ranging from 1 Billion (B) over 56B all the way up to 671B. As such, running inference with these models is expensive. For instance, the 671B model (called “DeepSeek R1”) requires (without optimization, lower bound) ~700 GB of (GPU) memory which needs 8 H100 just for loading it (market price as of March 2026: ~40,000€ / unit). Thus, these models are usually deployed using cloud-based solutions where your query is sent to and processed by a server-cluster.
This means a number of issues for the user such as potentially high latency, no way to query it offline and privacy concerns of both your and other's data. For instance, using ChatGPT to summarize your chat messages means you are giving away not just your data but also that of the other participants.
To address this, model compression is an active area of research that aims to lower resource requirements of models by “shrinking” them. Ideally, this allows running those models locally and even in resource-constrained settings (on “edge devices” like a smartphone). However, the effectiveness of such methods should be verified empirically by doing actual deployment on edge devices. A previous thesis has established a framework for deploying LLMs on the Nvidia Orin AGX Development Kit. This thesis dives deeper into both hard- and software to establish what is going on “behind the scenes”.
Specifically, the student should
- Set up a profiling software (e.g. Nvidia Nsights) on the edge device
- Use it to profile the framework established in the previous thesis
- Based on the results, identify and - ideally - fix potential bottlenecks
Necessary Background:
- You can work independently
- You can follow basic instructions such as those found under “Contact Details”
Recommended Background:
- Has used a package manager like Anaconda before
Ideal Background:
- Some experience using transformers, llama.cpp or MLC-LLM
- Knows what CUDA is
- Basic understanding of the transformer architecture (i.e. attention mechanism, auto-regressive decoding, kv-cache, …)
Contact Details:
- Please contact me only via e-mail
- Attach your Transcript of Records (feel free to hide your grades, I only want to see what lectures you have heard)
- I try to get back to you within a week. If I don't, please contact me again (ideally just resend your original mail). If you don't, I'll assume you are no longer interested.
