Lecture: Deep Learning

Within the last decade, deep neural networks have emerged as an indispensable tool in many areas of artificial intelligence including computer vision, computer graphics, natural language processing, speech recognition and robotics. This course will introduce the practical and theoretical principles of deep neural networks. Amongst other topics, we will cover computation graphs, activation functions, loss functions, training, regularization and data augmentation as well as various basic and state-of-the-art deep neural network architectures including convolutional networks and graph neural networks. The course will also address deep generative models such as auto-encoders, variational auto-encoders and generative adversarial networks. In addition, applications from various fields will be presented throughout the course. The tutorials will deepen the understanding of deep neural networks by implementing and applying them in Python and PyTorch.

Students gain an understanding of the theoretical and practical concepts of deep neural networks including, optimization, inference, architectures and applications. After this course, students should be able to develop and train deep neural networks, reproduce research results and conduct original research in this area.

• Course number: ML-4103 (also credited for INFO-4182 and INF-4182)
• Credits: 6 ECTS (2h lecture + 2h exercise)
• Total Workload: 180h
• Lectures and exercises will be held asynchronously through YouTube (see sidebar for link). We will provide all lectures and exercise introductions several days before the respective interactive live sessions for self-study. You should watch these videos before participating in the interactive live sessions.
• Each Wednesday, we will host an interactive plenary lecture Q&A session starting at 12:00 via Zoom (see sidebar for link) where questions regarding the lecture and exercises are answered. Every second Wednesday (when a new exercise starts) this session is followed by a plenary exericse Q&A session where remaining questions for the current exercise can be answered. Make sure that you have the latest Zoom client installed.
• Every other week, we will hold individual live exercise Q&A sessions throughout the entire Wednesday in smaller groups to obtain personalized feedback and help on the currently running exercise or get answers to general questions. The assignments to these individual sessions is done through ILIAS.
• Exercises will not be graded. We will provide solutions before the final plenary Q&A session.
• Students shall watch both the lecture and exercise videos before the Q&A session and take note of questions that they like to have answered during the Q&A sessions.

• Basic Computer Science skills: Variables, functions, loops, classes, algorithms
• Basic Python and PyTorch coding skills
• Basic Math skills: Linear algebra, probability and information theory (eg., Math for ML lecture
  https://www.tml.cs.uni-tuebingen.de/teaching/2020_maths_for_ml/index.php)
  As a refresher we recommend reading Chapters 1-4 of: http://www.deeplearningbook.org

• To participate in this lecture or to enroll for our exam, you must enroll via ILIAS (see sidebar for link) until October 28, 2020. Note that the registration deadline has passed and we are not accepting new students to the course. If you have missed the deadline but this course is mandatory for you, email us.
• Information about exam registration can be found here.

• To qualify for the final exam, students must have registered to the lecture on ILIAS by 28.10.2020
• To obtain a 0.3 bonus in the final exam, students must have submitted lecture notes for one lecture
• Please bring a pen, a ruler, your student ID and a FFP2/medical mask to the exam. Paper will be provided by us. No scripts, notes, books, slides, calculators, mobile phones, laptops or other additional tools are allowed, only what is listed above. You may bring a drinking bottle. Food is not allowed.

All topics discussed in the lecture, Q&A sessions and exercises (pen and paper and coding exercises) are relevent for the final exam. All students must participate in the main exam that will take place during the official examination period. Only students that failed the main exam and students that were ill during the main exam (doctoral certificate required) are allowed to enroll for a make-up exam.

The exercises play an essential role in understanding the content of the course. There will be 6 assignments in total (see content table below). The assignments contain pen and paper questions as well as programming problems. In the first half of the course, the students will develop and use the Educational Deep Learning Framework (EDF), a small Python only deep learning framework. This will allow them to understand every aspect of deep learning (computation graphs, backpropagation, optimization, regularization) in detail on small problems (MNIST). In the second half of the course, the students will use PyTorch, a state-of-the-art deep learning framework which features GPU support and auto-differentiation, to address more challenging problems. If you have questions regarding the exercises or the lecture, please ask them during the interactive zoom sessions or in our ILIAS forum.

The students will collectively write Latex lecture notes to complement the slides, summarizing the content discussed in the lecture videos. In the beginning of the course, every registered student will be assigned one lecture. The lecture notes must be submitted via ILIAS at the latest 7 days after the respective official lecture date (see content table below). Lecture notes must be written individually (not in groups). We will continuously merge and consolidate the lecture notes into a single document. You can edit the lecture notes in Overleaf or a local Latex editor. To get started, copy the Deep Learning Lecture Notes Latex Template

• Current version of the Deep Learning Lecture Notes
• Goodfellow, Bengio and Courville: Deep Learning
• Zhang, Lipton, Li, Smola: Dive into Deep Learning
• Bishop: Pattern Recognition and Machine Learning
• Deisenroth, Faisal and Ong: Mathematics for Machine Learning
• Articles and papers mentioned in the lecture slides (footer)