Supervisor: Frank Schneider

At the moment, training a contemporary deep neural network is a time-consuming and resource-intensive process, largely due to the many crucial decisions practitioners must make
regarding the underlying training algorithms (e.g. should I use SGD, ADAM, or SHAMPOO? What learning rate (schedule)? Weight decay? etc.). ALGOPERF, a benchmark (competition) measuring training speed, provides a platform to compare various algorithmic choices and thus guide practitioners meaningfully. This project aims to prepare a submission for the ALGOPERF benchmark by either implementing and tweaking existing methods or by innovatively combining them.

Possible approaches are:

• Developing a self-tuning submission based on the winning external tuning submission, SHAMPOO.
• Improving the winning self-tuning submission, SCHEDULE-FREE ADAMW,
  e.g. by sequentially scheduling a second hyperparameter config for training the IMAGENET RESNET workload.
•  Benchmarking new algorithms that did not yet have a submission, e.g. ADEMAMI

Prerequisites:

• (basic) experience in PyTorch or JAX
• (basic) knowledge of deep learning

Supervisor: Bálint Mucsányi

The field of uncertainty disentanglement (UD) has been steadily gaining traction in recent years. UD aims to construct multiple uncertainty estimators that are each tailored to one and only one source of uncertainty (e.g., epistemic and aleatoric uncertainty). Recently, it has been found that the most widely used uncertainty decomposition formulas fail to disentangle uncertainties. One possible reason is that the different uncertainties are all captured in the output space, which leads to high correlations. This project explores breaking these correlations by calculating the estimates at different points of the computation graph. The primary objective is to equip weight-space-based UQ methods – such as the Laplace approximation – with disentanglement capabilities by pushing the uncertainty forward to an intermediate representation for epistemic uncertainty and measuring aleatoric uncertainty in the output
space. The new approaches can be evaluated on an existing UD benchmark and compared to latent density methods.

Prerequisites:

•  experience with PyTorch

Supervisor: Carla Sagebiel

The Fréchet inception distance (FID) is one of the most important tools to assess the quality of Generative models in Computer Vision today. However, this metric is not without flaws [1] and also highly biased by the dataset the Inception Model was trained on. The goal of this project is to train a custom Inception network based on some specialized dataset and then evaluate the resulting FID, for example by measuring its ability to differentiate between original data and different test datasets (i.e. artificially distorted data or out of distribution examples), and by creating heat maps using GradCAM like in the mentioned paper. Special care should be used to keep the training of the model and the evaluating of
the FID separate.

Prerequisites:

• experience in Python, basic experience with PyTorch
• enough knowledge in ML to read the scientific literature mentioned above

[1] http://arxiv.org/abs/2203.06026
 

Supervisor: Tobias Weber

Synthetic datasets are increasingly recognized as valuable for training deep architectures. The emergence of potentially infinite data spaces introduces novel opportunities, particularly in dynamically sampling new data based on the current model’s performance. In this thesis, we will explore the use of uncertainty-aware data sampling strategies in synthetic data generation and selection by focusing on the replacement of the traditional data loader in machine learning pipelines with a data simulator. In doing so, we aim to connect this approach to the recent discussion on whether, and in which cases, synthetic data leads to model collapse. While the project ideally starts with a literature review and iteration over small-scale experiments, conducting a research project first might be beneficial, though not strictly necessary.

Prerequisites:

• Excitement for reading research papers.
• Experience with setting up Deep Learning experiments in Jax or PyTorch.
   

Supervisor: Nathaël da Costa

Standard neural architectures for classification typically comprise of a neural network mapping to a Euclidean space (the logits), followed by a softmax activation function. Such architectures are then trained using the cross entropy loss. Bayesian deep learning then attempts to build a Gaussian distribution over the parameters over the neural network. For each input to the network, this results in a Gaussian distribution over the logit space, which must be pushed forward through the softmax and integrated to obtain predictive probabilities. This last step cannot be done tractably in closed form, and is thus approximated through Monte-Carlo sampling, which is computationally costly and noisy. This is an exploratory project to test different choices of last-layer activations (to replace the softmax) and losses (to replace the cross-entropy loss) that allow for closed-form approximate distributions over the predictive space, bypassing Monte-Carlo sampling.

Prerequisites:

• experience with PyTorch or Jax
• experience with deep learning