// Databricks notebook source exported at Tue, 28 Jun 2016 09:28:40 UTC

Scalable Data Science

prepared by Raazesh Sainudiin and Sivanand Sivaram

supported by and

The html source url of this databricks notebook and its recorded Uji Image of Uji, Dogen's Time-Being:

sds/uji/week9/16_Deep_learning/030_Deep_learning

Deep learning: A Crash Introduction

This notebook provides an introduction to Deep Learning. It is meant to help you descend more fully into these learning resources and references:

  • Udacity’s course on Deep Learning https://www.udacity.com/course/deep-learning–ud730 by Arpan Chakraborty and Vincent Vanhoucke
  • Neural networks and deep learning http://neuralnetworksanddeeplearning.com/ by Michael Nielsen
  • Deep learning book http://www.deeplearningbook.org/ by Ian Goodfellow, Yoshua Bengio and Aaron Courville

  • Deep learning - buzzword for Artifical Neural Networks
  • What is it?
    • Supervised learning model - Classifier
    • Unsupervised model - Anomaly detection
  • Needs lots of data
  • Online learning model - backpropogation
  • Optimization - Stochastic gradient descent
  • Regularization - L1, L2, Dropout ** **
  • Supervised
    • Fully connected network
    • Convolutional neural network - Eg: For classifying images
    • Recurrent neural networks - Eg: For use on text, speech
  • Unsupervised
    • Autoencoder


A quick recap of logistic regression / linear models

(watch now 46 seconds):

Udacity: Deep Learning by Vincent Vanhoucke - Training a logistic classifier


– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke


Regression

Regression
y = mx + c

Another way to look at a linear model

Another way to look at a linear model

– Image Credit: Michael Nielsen



Recap - Gradient descent

(1:54 seconds):

Udacity: Deep Learning by Vincent Vanhoucke - Gradient descent


– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke



Recap - Stochastic Gradient descent

(2:25 seconds):

Udacity: Deep Learning by Vincent Vanhoucke - Stochastic Gradient descent


– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke

HOGWILD! Parallel SGD without locks http://i.stanford.edu/hazy/papers/hogwild-nips.pdf



Why deep learning? - Linear model

(24 seconds):

Udacity: Deep Learning by Vincent Vanhoucke - Linear model


– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke

ReLU - Rectified linear unit or Rectifier - max(0, x)

ReLU

– Image Credit: Wikipedia



Neural Network

Watch now (45 seconds)

Udacity: Deep Learning by Vincent Vanhoucke - Neural network *** – Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke

Is decision tree a linear model?
http://datascience.stackexchange.com/questions/6787/is-decision-tree-algorithm-a-linear-or-nonlinear-algorithm


Neural Network ** Neural network ** – Image credit: Wikipedia

Multiple hidden layers

Many hidden layers *** – Image credit: Michael Nielsen



What does it mean to go deep? What do each of the hidden layers learn?

Watch now (1:13 seconds)

Udacity: Deep Learning by Vincent Vanhoucke - Neural network *** – Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke

Chain rule

(f o g)’ = (f’ o g) . g’



Chain rule in neural networks

Watch later (55 seconds)

Udacity: Deep Learning by Vincent Vanhoucke - Neural network *** – Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke

Backpropogation


Watch later (9:55 seconds)

Backpropogation ** **

Watch now (1: 54 seconds)
Backpropogation ***

####How do you set the learning rate? - Step size in SGD?

Convolutional Neural Networks


** Watch (3:55)
Udacity: Deep Learning by Vincent Vanhoucke - Convolutional Neural network
**

Recurrent neural network

Recurrent neural network
http://colah.github.io/posts/2015-08-Understanding-LSTMs/

http://karpathy.github.io/2015/05/21/rnn-effectiveness/
** Watch (3:55)
Udacity: Deep Learning by Vincent Vanhoucke - Recurrent Neural network
**

LSTM - Long short term memory

LSTM


GRU - Gated recurrent unit

Gated Recurrent unit http://arxiv.org/pdf/1406.1078v3.pdf

Autoencoder

Autoencoder ** Watch (3:51)
Autoencoder
**

Scalable Data Science

prepared by Raazesh Sainudiin and Sivanand Sivaram

supported by and