Lecture: Deep Learning

Within the last decade, deep neural networks have emerged as an indispensable tool in many areas of artificial intelligence including computer vision, computer graphics, natural language processing, speech recognition and robotics. This course will introduce the practical and theoretical principles of deep neural networks. Amongst other topics, we will cover computation graphs, activation functions, loss functions, training, regularization and data augmentation as well as various basic and state-of-the-art deep neural network architectures including convolutional networks and graph neural networks. The course will also address deep generative models such as auto-encoders, variational auto-encoders and generative adversarial networks. In addition, applications from various fields will be presented throughout the course. The tutorials will deepen the understanding of deep neural networks by implementing and applying them in Python and PyTorch.

Qualification Goals

Students gain an understanding of the theoretical and practical concepts of deep neural networks including, optimization, inference, architectures and applications. After this course, students should be able to develop and train deep neural networks, reproduce research results and conduct original research in this area.

Overview

  • Course number: ML-4103 (also credited for INFO-4182 and INF-4182)
  • Credits: 6 ECTS
  • Recommended for: Master, 1st semester
  • Total Workload: 180h
  • This lecture is taught as flipped classroom: Lectures will be provided via YouTube and must be watched before the respective interactive live sessions.
  • Each week, we host an interactive live session where questions regarding the lecture and exercises are posed and discussed together. It is essential for students to attend the live sessions.
  • We also offer additional weekly helpdesks where students may ask questions to obtain individual feedback and support for solving the exercises.
  • In addition, we provide regular quizzes via our quiz server with questions on the lectures and exercises for self-assessment and self-motivation.
  • Finally, we are providing continuously and timely support via our chat.
  • See 'Important Links' in the sidebar to access the videos, slides, exercises, chat, zoom room and quiz.

Prerequisites

Registration

  • To participate, you must register via ILIAS and our Quiz Server (see sidebar)
  • Registration via ILIAS will open on 30.09. at 12:00
  • Information about exam registration can be found here

Exercises

The exercises play an essential role in understanding the content of the course. There will be 6 assignments in total. The assignments contain pen and paper questions as well as programming problems. In the first half of the course, the students will use the Educational Deep Learning Framework (EDF), a small Python only deep learning framework. This will allow them to understand every aspect of deep learning (computation graphs, backpropagation, optimization, regularization) in detail on small problems. In the second half of the course, the students will use PyTorch, a state-of-the-art deep learning framework which features GPU support and auto-differentiation, to address more challenging problems. If you have questions regarding the exercises or the lecture, please ask them during the live sessions, at the zoom helpdesk or in our chat.

Further Readings

Schedule

Date

Lecture Slides and Videos

Live Sessions (EDF | PyTorch)

TA Support

Recap: Math for Deep Learning

16.10.

L01 - Introduction | Slides

1.1 Organization | Video

1.2 History of Deep Learning | Video

1.3 Machine Learning Basics | Video

L01 - Lecture Organization
E01 - Exercise Introduction | Problems

Christian Reiser

23.10.

L02 - Computation Graphs | Slides

2.1 Logistic Regression | Video

2.2 Computation Graphs | Video

2.3 Backpropagation | Video

2.4 Educational Framework | Video

L02 - Lecture Q&A

E01 - Exercise Q&A

Christian Reiser

30.10.

L03 - Deep Neural Networks | Slides

3.1 Backpropagation with Tensors | Video

3.2 The XOR Problem | Video

3.3 Multi-Layer Perceptrons | Video

3.4 Universal Approximation | Video

L03 - Lecture Q&A

E01 - Exercise Q&A
E02 - Exercise Introduction | Problems

Christian Reiser

Takeru Miyato

06.11.

L04 - Deep Neural Networks II | Slides

4.1 Output and Loss Functions | Video

4.2 Activation Functions | Video

4.3 Preprocessing and Initialization | Video

L04 - Lecture Q&A

E02 - Exercise Q&A

Takeru Miyato

13.11.

No Lecture

20.11.

L05 - Regularization | Slides

5.1 Parameter Penalties | Video

5.2 Early Stopping | Video

5.3 Ensemble Methods | Video

5.4 Dropout | Video

5.5 Data Augmentation | Video

L05 - Lecture Q&A

E02 - Exercise Q&A
E03 - Exercise Introduction | Problems

Takeru Miyato

Bozidar Antic

27.11.

L06 - Optimization | Slides
6.1 Optimization Challenges | Video

6.2 Optimization Algorithms | Video

6.3 Optimization Strategies | Video

6.4 Debugging Strategies | Video

L06 - Lecture Q&A

E03 - Exercise Q&A

Bozidar Antic

04.12.

L07 - Convolutional Neural Networks | Slides

7.1 Convolution | Video

7.2 Downsampling | Video

7.3 Upsampling | Video

7.4 Architectures | Video

7.5 Visualization | Video

L07 - Lecture Q&A

E03 - Exercise Q&A
E04 - Exercise Introduction | Problems

Bozidar Antic

Anpei Chen

11.12.

L08 - Sequence Models | Slides
8.1 Recurrent Networks | Video

8.2 Recurrent Network Applications | Video

8.3 Gated Recurrent Networks | Video

8.4 Autoregressive Models | Video

L08 - Lecture Q&A

E04 - Exercise Q&A

Anpei Chen

18.12.

L09 - Natural Language Processing | Slides

9.1 Language Models | Video

9.2 Traditional Language Models | Video

9.3 Neural Language Models | Video

9.4 Neural Machine Translation | Video

L09 - Lecture Q&A

E04 - Exercise Q&A
E05 - Exercise Introduction | Problems

Anpei Chen

Daniel Dauner

08.01.

L10 - Graph Neural Networks | Slides

10.1 Machine Learning on Graphs | Video

10.2 Graph Convolution Filters | Video

10.3 Graph Convolution Networks | Video

L10 - Lecture Q&A

E05 - Exercise Q&A

Daniel Dauner

15.01.

No Lecture

22.01.

L11 - Autoencoders | Slides

11.1 Latent Variable Models | Video | Video

11.2 Principal Component Analysis | Video

11.3 Autoencoders | Video

11.4 Variational Autoencoders | Video

L11 - Lecture Q&A

E05 - Exercise Q&A
E06 - Exercise Introduction | Problems

Daniel Dauner

29.01.

L12 - Generative Adversarial Networks | Slides

12.1 Generative Adversarial Networks | Video

12.2 GAN Developments | Video

12.3 Research at AVG | Video

L12 - Lecture Q&A
E06 - Exercise Q&A

Daniel Dauner

Research @ AVG

1. Learning Robust Policies for Self-Driving
2. Generating Images and 3D Shapes

3. Constraining 3D Fields for NVS and Reconstruction