CS 6890: Deep Learning
Spring 2017

Time and Location: Tue, Thu, Fri 9:00 – 10:20am, ARC 101
Instructor: Razvan Bunescu
Office: Stocker 341
Office Hours: Tue, Thu 10:30 – 11:00am, or by email appointment
Email: bunescu @ ohio edu

  • Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville. MIT Press, 2016.

  • Recommended introductory material:
  • Machine Learning @ Stanford video lectures.
  • Machine Learning @ Ohio course website.

  • Course description:
    This course will introduce a number of approaches for learning representations of data that have been shown to substantially improve the performance of machine learning algorithms on a wide array of tasks in computer vision, speech recognition, and natural language processing. Basic features, such as edge detectors in computer vision, will be induced automatically from unlabeled data using sparse or denoising autoencoders. These features can be assembled into increasingly more complex representations by greedy layering in hierarchical architectures. Learning high-level representations that are expressed in terms of other, simpler representation is the hallmark of deep learning. The course will introduce the multi-layer perceptron, a common deep learning architecture, and its gradient-based training through the backpropagation algorithm. The later part of the course will explore specialized neural network architectures such as convolutional neural networks (for images), recurrent neural networks (for sequences), and memory-augmented neural networks. The lectures will cover theoretical aspects of deep learning models, whereas homework assignments will give students the opportunity to build and experiment with shallow and deep learning models, for which skeleton code will be provided.

    Proposed topics:
    Logistic and Softmax Regression, Feed-Forward Neural Networks, Backpropagation, Sparse Autoencoders, Denoising Autoencoders, Linear Decoders, Vectorization, PCA and Whitening, Self-Taught Learning, Deep Networks, Convolution and Pooling, Recurrent Neural Networks, Long Short-Term Memory, Gated Recurrent Units, Neural Attention Models, Sequence-to-Sequence Models, Memory Networks.

    Previous exposure to basic concepts and models in machine learning, such as: supervised vs. unsupervised learning, classification vs. regression, linear regression, logistic and softmax regression, cost functions, overfitting and regularization, gradient-based optimization. Experience with programming and familiarity with basic concepts in linear algebra and statistics.

    Lecture notes:
    1. Syllabus & Introduction
    2. Linear Regression, Perceptrons, Logistic and Softmax Regression
    3. Linear algebra and optimization in Python
    4. Machine Learning with Computation Graphs in TensorFlow
    5. Feed-Forward Neural Networks and Backpropagation
    6. Unsupervised Feature Learning
    7. Self-Taught Learning and Deep Learning
    8. Gradient-based learning
    9. Convolutional Neural Networks
    10. Recurrent Neural Networks, LSTMS, GRUs

    Homework Assignments1,2:
    1. Assignment, code and data.
    2. Assignment, code and data.
    3. Assignment and code.
    4. Assignment, code and data.
    5. Assignment, code and data.
    6. Assignment, code, word2vec Google News embeddings, and the Stanford Natural Language Inference (SNLI) dataset.
    1 The code for assignments 1, 2, 3, and 4 is based on the UFLDL tutorial exercises. Assignment 5 is based on Michael Nielsen's book on Neural Networks and Deep Learning
    2 The code for assignments 1, 2, and 3 was written by Razvan Bunescu. The code for assignments 4, 5, and 6 was written by Hui Shen.

    Final Project:
    Paper Presentations
    1. Deep Dreams:
    2. DeepTox:
    3. Restricted Boltzman Machines (RBMs):
    4. Horses, Adversarial Examples and Adversarial Training:
    5. Neural Machine Translation:
    6. Memory Augmented Networks:
    7. Meta-learning and One-shot learning:
    8. Dense Associative Memory:
    9. Deep Reinforcement Learning:

    Other topics
    Other papers:
    Online reading materials: