Lecture: Deep Learning

Within the last decade, deep neural networks have emerged as an indispensable tool in many areas of artificial intelligence including computer vision, computer graphics, natural language processing, speech recognition and robotics. This course will introduce the practical and theoretical principles of deep neural networks. Amongst other topics, we will cover computation graphs, activation functions, loss functions, training, regularization and data augmentation as well as various basic and state-of-the-art deep neural network architectures including convolutional networks and graph neural networks. The course will also address deep generative models such as auto-encoders, variational auto-encoders and generative adversarial networks. In addition, applications from various fields will be presented throughout the course. The tutorials will deepen the understanding of deep neural networks by implementing and applying them in Python and PyTorch.

Qualification Goals

Students gain an understanding of the theoretical and practical concepts of deep neural networks including, optimization, inference, architectures and applications. After this course, students should be able to develop and train deep neural networks, reproduce research results and conduct original research in this area.

Overview

  • Course number: ML-4103 (also credited for INFO-4182 and INF-4182)
  • Credits: 6 ECTS (2h lecture + 2h exercise)
  • Total Workload: 180h
  • Lectures and exercises will be held asynchronously through YouTube (see sidebar for link). We will provide all lectures and exercise introductions several days before the respective interactive live sessions for self-study. You should watch these videos before participating in the interactive live sessions.
  • Each Wednesday, we will host an interactive plenary lecture Q&A session starting at 12:00 via Zoom (see sidebar for link) where questions regarding the lecture and exercises are answered. Every second Wednesday (when a new exercise starts) this session is followed by a plenary exericse Q&A session where remaining questions for the current exercise can be answered. Make sure that you have the latest Zoom client installed.
  • Every other week, we will hold individual live exercise Q&A sessions throughout the entire Wednesday in smaller groups to obtain personalized feedback and help on the currently running exercise or get answers to general questions. The assignments to these individual sessions is done through ILIAS.
  • Exercises will not be graded. We will provide solutions before the final plenary Q&A session.
  • Students shall watch both the lecture and exercise videos before the Q&A session and take note of questions that they like to have answered during the Q&A sessions.

Prerequisites

Registration

  • To participate in this lecture or to enroll for our exam, you must enroll via ILIAS (see sidebar for link) until October 28, 2020. Note that the registration deadline has passed and we are not accepting new students to the course. If you have missed the deadline but this course is mandatory for you, email us.
  • Information about exam registration can be found here.

Exam

  • To qualify for the final exam, students must have registered to the lecture on ILIAS by 28.10.2020
  • To obtain a 0.3 bonus in the final exam, students must have submitted lecture notes for one lecture
  • Please bring a pen, a ruler, your student ID and a FFP2/medical mask to the exam. Paper will be provided by us. No scripts, notes, books, slides, calculators, mobile phones, laptops or other additional tools are allowed, only what is listed above. You may bring a drinking bottle. Food is not allowed.

All topics discussed in the lecture, Q&A sessions and exercises (pen and paper and coding exercises) are relevent for the final exam. All students must participate in the main exam that will take place during the official examination period. Only students that failed the main exam and students that were ill during the main exam (doctoral certificate required) are allowed to enroll for a make-up exam.

Exercises

The exercises play an essential role in understanding the content of the course. There will be 6 assignments in total (see content table below). The assignments contain pen and paper questions as well as programming problems. In the first half of the course, the students will develop and use the Educational Deep Learning Framework (EDF), a small Python only deep learning framework. This will allow them to understand every aspect of deep learning (computation graphs, backpropagation, optimization, regularization) in detail on small problems (MNIST). In the second half of the course, the students will use PyTorch, a state-of-the-art deep learning framework which features GPU support and auto-differentiation, to address more challenging problems. If you have questions regarding the exercises or the lecture, please ask them during the interactive zoom sessions or in our ILIAS forum.

Lecture Notes

The students will collectively write Latex lecture notes to complement the slides, summarizing the content discussed in the lecture videos. In the beginning of the course, every registered student will be assigned one lecture. The lecture notes must be submitted via ILIAS at the latest 7 days after the respective official lecture date (see content table below). Lecture notes must be written individually (not in groups). We will continuously merge and consolidate the lecture notes into a single document. You can edit the lecture notes in Overleaf or a local Latex editor. To get started, copy the Deep Learning Lecture Notes Latex Template

Further Readings

Schedule

Date

Lectures

Exercises (EDF | PyTorch)

TA Support

04.11.

L01 - Introduction | Slides

1.1 Introduction | Video

1.2 History of Deep Learning | Video

1.3 Machine Learning Basics | Video

E01 - Plenary Q&A | Problems & Solutions

Introduction to EDF

Christian Reiser

11.11.

L02 - Computation Graphs | Slides

2.1 Logistic Regression | Video

2.2 Computation Graphs | Video

2.3 Backpropagation | Video

2.4 Educational Framework | Video

E01 - Plenary Q&A

Christian Reiser

18.11.

No Lecture

E01 - Plenary Q&A

Christian Reiser

25.11.

L03 - Deep Neural Networks | Slides

3.1 Backpropagation with Tensors | Video

3.2 The XOR Problem | Video

3.3 Multi-Layer Perceptrons | Video

3.4 Universal Approximation | Video

E01 - Plenary Q&A
E02 - Plenary Q&A | Problems & Solutions
Image Classification

Songyou Peng

02.12.

L04 - Deep Neural Networks II | Slides

4.1 Output and Loss Functions | Video

4.2 Activation Functions | Video

4.3 Preprocessing and Initialization | Video

E02 - Individual Q&A

Songyou Peng

09.12.

L05 - Regularization | Slides

5.1 Parameter Penalties | Video

5.2 Early Stopping | Video

5.3 Ensemble Methods | Video

5.4 Dropout | Video

5.5 Data Augmentation | Video

E02 - Plenary Q&A
E03 - Plenary Q&A | Problems & Solutions

Regularization and Optimization

Joo Ho Lee

16.12.

L06 - Optimization | Slides
6.1 Optimization Challenges | Video

6.2 Optimization Algorithms | Video

6.3 Optimization Strategies | Video

6.4 Debugging Strategies | Video

E03 - Individual Q&A

Joo Ho Lee

13.01.

L07 - Convolutional Neural Networks | Slides

7.1 Convolution | Video

7.2 Downsampling | Video

7.3 Upsampling | Video

7.4 Architectures | Video

7.5 Visualization | Video

E03 - Plenary Q&A
E04 - Plenary Q&A | Problems & Solutions

CNNs and Sequence Models

Axel Sauer

20.01.

L08 - Sequence Models | Slides
8.1 Recurrent Networks | Video

8.2 Recurrent Network Applications | Video

8.3 Gated Recurrent Networks | Video

8.4 Autoregressive Models | Video

E04 - Individual Q&A

Axel Sauer

27.01.

L09 - Natural Language Processing | Slides

9.1 Language Models | Video

9.2 Traditional Language Models | Video

9.3 Neural Language Models | Video

9.4 Neural Machine Translation | Video

E04 - Plenary Q&A
E05 - Plenary Q&A | Problems & Solutions

NLP and GNNs

Aditya Prakash

03.02.

L10 - Graph Neural Networks | Slides

10.1 Machine Learning on Graphs | Video

10.2 Graph Convolution Filters | Video

10.3 Graph Convolution Networks | Video

E05 - Individual Q&A

Aditya Prakash

10.02.

L11 - Autoencoders | Slides

11.1 Latent Variable Models | Video | Video

11.2 Principal Component Analysis | Video

11.3 Autoencoders | Video

11.4 Variational Autoencoders | Video

E05 - Plenary Q&A
E06 - Plenary Q&A | Problems & Solutions

Latent Variable Models

Christian Reiser

17.02.

No Lecture

E06 - Individual Q&A

Christian Reiser

24.02.

L12 - Generative Adversarial Networks | Slides

12.1 Generative Adversarial Networks | Video

12.2 GAN Developments | Video

12.3 Research at AVG | Video

E06 - Plenary Q&A

Christian Reiser