CS 6890: Deep Learning

Spring 2018

This course will introduce the

Logistic and Softmax Regression, Feed-Forward Neural Networks, Backpropagation, Vectorization, PCA and Whitening, Deep Networks, Convolution and Pooling, Recurrent Neural Networks, Long Short-Term Memory, Gated Recurrent Units, Neural Attention Models, Sequence-to-Sequence Models, Memory Networks, Distributional Representations, Generative Adversarial Networks, Deep Reinforcement Learning.

Previous exposure to basic concepts in machine learning, such as: supervised vs. unsupervised learning, classification vs. regression, linear regression, logistic and softmax regression, cost functions, overfitting and regularization, gradient-based optimization. Substantial experience with programming and familiarity with basic concepts in linear algebra and statistics.

- Syllabus & Introduction
- Hand notes Jan 16 one, Jan 16 two.

- Logistic Regression, Softmax Regression, and Gradient Descent
- Hand notes Jan 23.
- An overview of gradient descent optimization algorithms, Sebastian Ruder, CoRR 2016

- Linear algebra and optimization in Python
- Machine Learning with PyTorch
- PyTorch examples
- Linear regression with gradient descent in PyTorch
- PyTorch video lecture and slides by Soumith Chintala.

- Feed-Forward Neural Networks and Backpropagation
- Hand notes Feb 6 one, Feb 6 two, Feb 6 three.

- Unsupervised Feature Learning with Autoencoders
- PCA, PCA whitening, and ZCA whitening
- Variational Autoencoders
- Hand notes Feb 15 one, Feb 15 two, Feb 20, and Feb 22
- Auto-Encoding Variational Bayes, Kingma and Welling, ICLR2014
- Tutorial on Variational Autoencoders, Carl Doersch, CMU 2016
- VEA implementation in PyTorch, Agustinus Kristiadi's Blog, 2017
- VAE lecture from U of Illinois
- Attribute2Image: Conditional Image Generation from Visual Attributes, Yan et al., ECCV 2016

- Convolutional Neural Networks
- Andrej Karpathy's notes on CS231n: Convolutional Neural Networks for Visual Recognition.
- UFLDL Tutorial at Stanford.

- Assignment and code.
- Assignment and code.
- Assignment, code and data.
- Assignment, code and data.

- James H. Martin's Introduction to probabilities
- Jason Eisner's equestrian Introduction to probabilities
- Inderjit Dhillon's Linear Algebra Background
- MIT instructor's Introduction to Matrices and Linear Algebra Review
- Strang's Video Lectures on Linear Algebra
- Convex Optimization, Stephen Boyd and Lieven Vandenberghe, Cambridge University Press 2004
- Mike Brookes' Matrix Reference Manual
- Petersen et al.'s The Matrix Cookbook