Recent developments in deep generative models, particularly in large-scale ones, have revolutionized multiple areas. We are now able to synthesize high-resolution images, generate text from images and vice versa. More excitingly, we can control the synthesis in many desirable ways, which enables use cases such as synthetic data augmentation, out-of-distribution generalization, and systematic error analysis. In this seminar, you will delve into the latest advancements in deep generative models. With interactive discussions and expert guidance, you will gain the knowledge and skills necessary to understand the core techniques behind the deep generative models, explore their various use cases, and grasp key limitations and challenges.

Overview:

• Seminar number: ML4501d;
• Credits: 3 CP;
• Language: English;
• Recommended for Master students in the final semester;
• The seminar has maximum 15 seats (registration is open on Mar. 30, 2023, 12:00 via ILIAS);
• The seminar will kick off with an initial all-hands meeting to formally introduce students to the seminar's content, format, tutors, and exam type;
• Next up, each student will be required to select a research paper to study during the semester from a list of available options; and give a final presentation on the selected paper;
• Each paper has a designated tutor who will be available to schedule appointments with the student and discuss the content of the paper;
• In addition to studying the selected papers, the seminar will provide some hackathon projects where students can collaborate with their tutors to experiment with some off-the-shelf models. For instance, students can explore novel use cases and limitations of large-scale generative models. Findings from these projects can be presented in the final presentation, although participating in this hackathon is entirely optional, and considered as a bonus task in the final grading.

Qualification Goals:

• Read, understand, and explore scientific literature;
• Identify and summarize one research work from a given list of papers;
• Give a presentation (20 min) about your topic;
• Moderate a scientific discussion (10 min) after the presentation of one of your fellow students.

Assessment in the seminar will be based on presentation quality, scientific discussions (with tutors during the semester and with the whole group after the final presentation), and active participation in the entire event: we have 70% attendance policy for this seminar.

Prerequisites:

• Basic Computer Science skills: Variables, functions, loops, classes, algorithms;
• Basic Python and PyTorch coding skills;
• Basic Math skills: Linear algebra, and probability;
• Experience with deep learning and computer vision.

Schedule:

• First all-hands meeting (May 3, 2023): Slides
• Meetings with tutors are by request (at least one meeting, optionally extra two meetings)
• Final presentations (split into three sessions, mandatory to attend at least 2 out of 3 sessions):
  • July 12 13:00-16:00
  • July 13 10:00-13:00
  • July 14 10:00-13:00

Paper Selection: Please enter your name next to your selected paper in the document.

Paper List (for an overview):

• Diffusion models:
  • Consistency Models
  • Elucidating the Design Space of Diffusion-Based Generative Models
• Content generation with control:
  • Universal Guidance for Diffusion Models
  • DreamFusion: Text-to-3D using 2D Diffusion 
  • Composer: Creative and Controllable Image Synthesis with Composable Conditions
  • Adding Conditional Control to Text-to-Image Diffusion Models
  • GigaGAN: Scaling up GANs for Text-to-Image Synthesis
  • Zero-shot Image-to-Image Translation
  • GLIGEN: Open-set Grounded Text-to-Image Generation
  • Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models
• Applications of generative models:
  • DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort
  • Fake It Till You Make It: Learning(s) from a Synthetic ImageNet Clone 
  • Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization
  • A Generalist Framework for Panoptic Segmentation of Images and Videos
  • Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
  • Unleashing Text-to-Image Diffusion Models for Visual Perception
  • Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning
  • GAN-Supervised Dense Visual Alignment
• Limitations:
  • Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models

Recommended Readings:

• Base readings:
  • Taming Transformers for High-Resolution Image Synthesis
  • Denoising diffusion probabilistic models
  • High-Resolution Image Synthesis with Latent Diffusion Models
  • A Style-Based Generator Architecture for Generative Adversarial Networks
• Tutorials for good scientific presentations:
  • Talks That Don't Suck - A Guide to Improve Your Talks (Cyrill Stachniss)
  • https://twitter.com/jbhuang0604/status/1397058827405742085?lang=en