Lecture: Deep Learning

Within the last decade, deep neural networks have emerged as an indispensable tool in many areas of artificial intelligence including computer vision, computer graphics, natural language processing, speech recognition and robotics. This course will introduce the practical and theoretical principles of deep neural networks. Amongst other topics, we will cover computation graphs, activation functions, loss functions, training, regularization and data augmentation as well as various basic and state-of-the-art deep neural network architectures including convolutional networks and graph neural networks. The course will also address deep generative models such as auto-encoders, variational auto-encoders and generative adversarial networks. In addition, applications from various fields will be presented throughout the course. The tutorials will deepen the understanding of deep neural networks by implementing and applying them in Python and PyTorch.

Qualification Goals

Students gain an understanding of the theoretical and practical concepts of deep neural networks including, optimization, inference, architectures and applications. After this course, students should be able to develop and train deep neural networks, reproduce research results and conduct original research in this area.

Overview

  • Credits: 6 ECTS (2h lecture + 2h exercise)
  • Course number: ML-4103 (also credited for INFO-4182 and INF-4182)
  • Lectures and exercises will be held asynchronously through YouTube (see sidebar for link). We will provide all lectures and exercise introductions several days before the respective interactive live sessions for self-study. You should watch these videos before participating in the interactive live sessions.
  • Each Wednesday, we will host an interactive plenary lecture Q&A session starting at 12:00 via Zoom (see sidebar for link) where questions regarding the lecture and exercises are answered. Every second Wednesday (when a new exercise starts) this session is followed by a plenary exericse Q&A session where remaining questions for the current exercise can be answered. Make sure that you have the latest Zoom client installed.
  • Every other week, we will hold individual live exercise Q&A sessions throughout the entire Wednesday in smaller groups to obtain personalized feedback and help on the currently running exercise or get answers to general questions. The assignments to these individual sessions is done through ILIAS.
  • Exercises will not be graded. We will provide solutions before the final plenary Q&A session.
  • Students shall watch both the lecture and exercise videos before the Q&A session and take note of questions that they like to have answered during the Q&A sessions.

Prerequisites

Registration

  • To participate in this lecture or to enroll for our exam, you must enroll via ILIAS (see sidebar for link) until October 28, 2020. Note that the registration deadline has passed and we are not accepting new students to the course. If you have missed the deadline but this course is mandatory for you, email us.
  • Information about exam registration and make-up exams can be found here.

Exercises

The exercises play an essential role in understanding the content of the course. There will be 6 assignments in total (see content table below). The assignments contain pen and paper questions as well as programming problems. In the first half of the course, the students will develop and use the Educational Deep Learning Framework (EDF), a small Python only deep learning framework. This will allow them to understand every aspect of deep learning (computation graphs, backpropagation, optimization, regularization) in detail on small problems (MNIST). In the second half of the course, the students will use PyTorch, a state-of-the-art deep learning framework which features GPU support and auto-differentiation, to address more challenging problems. If you have questions regarding the exercises or the lecture, please ask them during the interactive zoom sessions or in our ILIAS forum.

Lecture Notes

The students will collectively write Latex lecture notes to complement the slides, summarizing the content discussed in the lecture videos. In the beginning of the course, every registered student will be assigned one lecture. The lecture notes must be submitted via ILIAS at the latest 7 days after the respective official lecture date (see content table below). Lecture notes must be written individually (not in groups). We will continuously merge and consolidate the lecture notes into a single document. You can edit the lecture notes in Overleaf or a local Latex editor. To get started, copy the Deep Learning Lecture Notes Latex Template.

Exam

  • To qualify for the final exam, students must have registered to the lecture on ILIAS by 28.10.2020
  • To obtain a 0.3 bonus in the final exam, students must have submitted lecture notes for one lecture

All topics discussed in the lecture, Q&A sessions and exercises (pen and paper and coding exercises) are relevent for the final exam. All students must participate in the main exam that will take place during the official examination period. Only students that failed the main exam and students that were ill during the main exam (doctoral certificate required) are allowed to enroll for a make-up exam.

Content

Date

Nr

Lectures

Exercises (EDF | PyTorch)

TA Support

04.11.

L01

Introduction | Slides

1.1 Introduction | Video

1.2 History of Deep Learning | Video

1.3 Machine Learning Basics | Video

E01 - Plenary Q&A | Problems | Solutions

Introduction to EDF and

Computation Graphs

Christian Reiser

11.11.

L02

Computation Graphs | Slides

2.1 Logistic Regression | Video

2.2 Computation Graphs | Video

2.3 Backpropagation | Video

2.4 Educational Framework | Video

E01 - Plenary Q&A

Christian Reiser

18.11.

No Lecture

E01 - Plenary Q&A

Christian Reiser

25.11.

L03

Deep Neural Networks | Slides

3.1 Backpropagation with Tensors | Video

3.2 The XOR Problem | Video

3.3 Multi-Layer Perceptrons | Video

3.4 Universal Approximation | Video

E01 - Plenary Q&A
E02 - Plenary Q&A | Problems | Solutions
Image Classification

Songyou Peng

02.12.

L04

Deep Neural Networks II | Slides

4.1 Output and Loss Functions | Video

4.2 Activation Functions | Video

4.3 Preprocessing and Initialization | Video

E02 - Individual Q&A

Songyou Peng

09.12.

L05

Regularization | Slides

5.1 Parameter Penalties | Video

5.2 Early Stopping | Video

5.3 Ensemble Methods | Video

5.4 Dropout | Video

5.5 Data Augmentation | Video

E02 - Plenary Q&A
E03 - Plenary Q&A | Problems | Solutions

Regularization and Optimization

Joo Ho Lee

16.12.

L06

Optimization | Slides
6.1 Optimization Challenges | Video

6.2 Optimization Algorithms | Video

6.3 Optimization Strategies | Video

6.4 Debugging Strategies | Video

E03 - Individual Q&A

Joo Ho Lee

13.01.

L07

Convolutional Neural Networks | Slides

7.1 Convolution | Video

7.2 Downsampling | Video

7.3 Upsampling | Video

7.4 Architectures | Video

7.5 Visualization | Video

E03 - Plenary Q&A
E04 - Plenary Q&A | Problems | Solutions

Introduction to PyTorch and

Convolutional Networks

Axel Sauer

20.01.

L08

Sequence Models | Slides
8.1 Recurrent Networks | Video

8.2 Recurrent Network Applications | Video

8.3 Gated Recurrent Networks | Video

8.4 Autoregressive Models | Video

E04 - Individual Q&A

Axel Sauer

27.01.

L09

Natural Language Processing | Slides

9.1 Language Models | Video

9.2 Traditional Language Models | Video

9.3 Neural Language Models | Video

9.4 Neural Machine Translation | Video

E04 - Plenary Q&A
E05 - Plenary Q&A | Problems | Solutions

Natural Language Processing

Aditya Prakash

03.02.

L10

Graph Neural Networks | Slides

10.1 Machine Learning on Graphs | Video

10.2 Graph Convolution Filters | Video

10.3 Graph Convolution Networks | Video

E05 - Individual Q&A

Aditya Prakash

10.02.

L11

Generative Models | Slides

11.1 Introduction | Video | Video

11.2 Variational Autoencoders | Video

11.3 Generative Adversarial Networks | Video

11.4 Evaluating Generative Models | Video

E05 - Plenary Q&A
E06 - Plenary Q&A | Problems | Solutions

Generative Models

Christian Reiser

17.02.

No Lecture

E06 - Individual Q&A

Christian Reiser

24.02.

L12

Self-Supervised Learning | Slides

12.1 Contrastive Learning for NLP and CV | Video

12.2 Pretext Tasks | Video

12.3 Self-supervision for Low-level Vision | Video

E06 - Plenary Q&A

Christian Reiser