Lecture: Deep Learning

Within the last decade, deep neural networks have emerged as an indispensable tool in many areas of artificial intelligence including computer vision, computer graphics, natural language processing, speech recognition and robotics. This course will introduce the practical and theoretical principles of deep neural networks. Amongst other topics, we will cover computation graphs, activation functions, loss functions, training, regularization and data augmentation as well as various basic and state-of-the-art deep neural network architectures including convolutional networks and graph neural networks. The course will also address deep generative models such as auto-encoders, variational auto-encoders and generative adversarial networks. In addition, applications from various fields will be presented throughout the course. The tutorials will deepen the understanding of deep neural networks by implementing and applying them in Python and PyTorch.

Qualification Goals

Students gain an understanding of the theoretical and practical concepts of deep neural networks including, optimization, inference, architectures and applications. After this course, students should be able to develop and train deep neural networks, reproduce research results and conduct original research in this area.

Overview

  • Course number: ML-4103 (also credited for INFO-4182 and INF-4182)
  • Credits: 6 ECTS
  • Recommended for: Master, 1st semester
  • Total Workload: 180h
  • This lecture is taught as flipped classroom. Lectures will be held asynchronously via YouTube (see sidebar for link). We will provide all lectures before the respective interactive live sessions for self-study. Please watch the relevant videos before participating in the interactive live sessions.
  • Each week, we host an interactive live session where questions regarding the lecture and exercises are discussed together (see sidebar for details).
  • We also offer a weekly zoom helpdesk where students may ask questions or share their screen to obtain individual feedback and support for solving the exercises (see sidebar for details).
  • Exercises will not be graded. Instead, we will discuss the solution together.
  • Students may obtain bonus points for the exam by answering questions about the lectures and exercises in weekly quizzes. The questions also serve as a measure for self-assessment and self-motivation. All quizzes are provided via our Lecture Quiz Server (see sidebar for details).

Prerequisites

Registration

  • To participate in this lecture, you must enroll via ILIAS (see sidebar for link)
  • Registration via ILIAS will open on 30.09. at 12:00
  • Information about exam registration can be found here

Exercises

The exercises play an essential role in understanding the content of the course. There will be 6 assignments in total. The assignments contain pen and paper questions as well as programming problems. In the first half of the course, the students will use the Educational Deep Learning Framework (EDF), a small Python only deep learning framework. This will allow them to understand every aspect of deep learning (computation graphs, backpropagation, optimization, regularization) in detail on small problems. In the second half of the course, the students will use PyTorch, a state-of-the-art deep learning framework which features GPU support and auto-differentiation, to address more challenging problems. If you have questions regarding the exercises or the lecture, please ask them during the live sessions, at the zoom helpdesk or in our ILIAS forum.

Further Readings

Schedule

Date

Lecture Slides and Videos

Live Sessions (EDF | PyTorch)

TA Support

Recap: Math for Deep Learning

19.10.

L01 - Introduction | Slides

1.1 Organization | Video

1.2 History of Deep Learning | Video

1.3 Machine Learning Basics | Video

L01 - Lecture Organization
E01 - Exercise Introduction | Problems

Bozidar Antic

26.10.

L02 - Computation Graphs | Slides

2.1 Logistic Regression | Video

2.2 Computation Graphs | Video

2.3 Backpropagation | Video

2.4 Educational Framework | Video

L02 - Lecture Q&A

E01 - Exercise Q&A

Bozidar Antic

02.11.

L03 - Deep Neural Networks | Slides

3.1 Backpropagation with Tensors | Video

3.2 The XOR Problem | Video

3.3 Multi-Layer Perceptrons | Video

3.4 Universal Approximation | Video

L03 - Lecture Q&A

E01 - Exercise Q&A
E02 - Exercise Introduction | Problems

Haoyu He

09.11.

L04 - Deep Neural Networks II | Slides

4.1 Output and Loss Functions | Video

4.2 Activation Functions | Video

4.3 Preprocessing and Initialization | Video

L04 - Lecture Q&A

E02 - Exercise Q&A

Haoyu He

16.11.

L05 - Regularization | Slides

5.1 Parameter Penalties | Video

5.2 Early Stopping | Video

5.3 Ensemble Methods | Video

5.4 Dropout | Video

5.5 Data Augmentation | Video

L05 - Lecture Q&A

E02 - Exercise Q&A
E03 - Exercise Introduction | Problems

Bozidar Antic

23.11.

L06 - Optimization | Slides
6.1 Optimization Challenges | Video

6.2 Optimization Algorithms | Video

6.3 Optimization Strategies | Video

6.4 Debugging Strategies | Video

L06 - Lecture Q&A

E03 - Exercise Q&A

Bozidar Antic

30.11.

L07 - Convolutional Neural Networks | Slides

7.1 Convolution | Video

7.2 Downsampling | Video

7.3 Upsampling | Video

7.4 Architectures | Video

7.5 Visualization | Video

L07 - Lecture Q&A

E03 - Exercise Q&A
E04 - Exercise Introduction | Problems

Bozidar Antic

07.12.

L08 - Sequence Models | Slides
8.1 Recurrent Networks | Video

8.2 Recurrent Network Applications | Video

8.3 Gated Recurrent Networks | Video

8.4 Autoregressive Models | Video

L08 - Lecture Q&A

E04 - Exercise Q&A

Bozidar Antic

14.12.

L09 - Natural Language Processing | Slides

9.1 Language Models | Video

9.2 Traditional Language Models | Video

9.3 Neural Language Models | Video

9.4 Neural Machine Translation | Video

L09 - Lecture Q&A

E04 - Exercise Q&A
E05 - Exercise Introduction | Problems
Live Session via Zoom Only

Haoyu He

21.12.

L10 - Graph Neural Networks | Slides

10.1 Machine Learning on Graphs | Video

10.2 Graph Convolution Filters | Video

10.3 Graph Convolution Networks | Video

L10 - Lecture Q&A

E05 - Exercise Q&A
Live Session via Zoom Only

Haoyu He

28.12.

No Lecture

04.01.

No Lecture

11.01.

L11 - Autoencoders | Slides

11.1 Latent Variable Models | Video | Video

11.2 Principal Component Analysis | Video

11.3 Autoencoders | Video

11.4 Variational Autoencoders | Video

L11 - Lecture Q&A

E05 - Exercise Q&A
E06 - Exercise Introduction | Problems

Haoyu He

18.01.

No Lecture

25.01.

L12 - Generative Adversarial Networks | Slides

12.1 Generative Adversarial Networks | Video

12.2 GAN Developments | Video

12.3 Research at AVG | Video

L12 - Lecture Q&A
E06 - Exercise Q&A

Haoyu He

Research @ AVG

1. Learning Robust Policies for Self-Driving
2. Generating Images and 3D Shapes

3. Constraining 3D Fields for NVS and Reconstruction