Lecture: Deep Learning

Within the last decade, deep neural networks have emerged as an indispensable tool in many areas of artificial intelligence including computer vision, computer graphics, natural language processing, speech recognition and robotics. This course will introduce the practical and theoretical principles of deep neural networks. Amongst other topics, we will cover computation graphs, activation functions, loss functions, training, regularization and data augmentation as well as various basic and state-of-the-art deep neural network architectures including convolutional networks and graph neural networks. The course will also address deep generative models such as auto-encoders, variational auto-encoders and generative adversarial networks. In addition, applications from various fields will be presented throughout the course. The tutorials will deepen the understanding of deep neural networks by implementing and applying them in Python and PyTorch.

Students gain an understanding of the theoretical and practical concepts of deep neural networks including, optimization, inference, architectures and applications. After this course, students should be able to develop and train deep neural networks, reproduce research results and conduct original research in this area.

• Course number: ML-4103 (also credited for INFO-4182 and INF-4182)
• Credits: 6 ECTS
• Recommended for: Master, 1st semester
• Total Workload: 180h
• This lecture is taught as flipped classroom: Lectures will be provided via YouTube and must be watched before the respective interactive live sessions.
• Each week, we host an interactive live session where questions regarding the lecture and exercises are posed and discussed together. It is essential for students to attend the live sessions.
• We also offer additional weekly helpdesks where students may ask questions to obtain individual feedback and support for solving the exercises.
• In addition, we provide regular quizzes via our quiz server with questions on the lectures and exercises for self-assessment and self-motivation.
• Finally, we are providing continuously and timely support via our forum.
• See 'Important Links' in the sidebar to access the videos, slides, exercises, forum, zoom room and quiz.

• Basic Computer Science skills: Variables, functions, loops, classes, algorithms
• Basic Python and PyTorch coding skills
• Basic Math skills: Linear algebra, probability and information theory (eg., Math for ML lecture
  https://www.tml.cs.uni-tuebingen.de/teaching/2020_maths_for_ml/index.php)
  As a refresher we recommend reading Chapters 1-4 of: http://www.deeplearningbook.org or watching our newly micro tutorials Mathematics for Deep Learning

• To participate in this lecture, you must enroll via ILIAS (see sidebar for link)
• Registration via ILIAS will open on 30.09. at 12:00
• Information about exam registration can be found here

The exercises play an essential role in understanding the content of the course. There will be 6 assignments in total. The assignments contain pen and paper questions as well as programming problems. In the first half of the course, the students will use the Educational Deep Learning Framework (EDF), a small Python only deep learning framework. This will allow them to understand every aspect of deep learning (computation graphs, backpropagation, optimization, regularization) in detail on small problems. In the second half of the course, the students will use PyTorch, a state-of-the-art deep learning framework which features GPU support and auto-differentiation, to address more challenging problems. If you have questions regarding the exercises or the lecture, please ask them during the live sessions, at the zoom helpdesk or in our ILIAS forum.

• Student's Deep Learning Lecture Notes
• Goodfellow, Bengio and Courville: Deep Learning
• Zhang, Lipton, Li, Smola: Dive into Deep Learning
• Bishop: Pattern Recognition and Machine Learning
• Deisenroth, Faisal and Ong: Mathematics for Machine Learning
• Articles and papers mentioned in the lecture slides
• Micro Tutorials Mathematics for Deep Learning