// Databricks notebook source exported at Tue, 28 Jun 2016 09:28:40 UTC
Scalable Data Science
prepared by Raazesh Sainudiin and Sivanand Sivaram
supported by 
and
The html source url of this databricks notebook and its recorded Uji 
:
Deep learning: A Crash Introduction
This notebook provides an introduction to Deep Learning. It is meant to help you descend more fully into these learning resources and references:
- Udacity’s course on Deep Learning https://www.udacity.com/course/deep-learning–ud730 by Arpan Chakraborty and Vincent Vanhoucke
- Neural networks and deep learning http://neuralnetworksanddeeplearning.com/ by Michael Nielsen
-
Deep learning book http://www.deeplearningbook.org/ by Ian Goodfellow, Yoshua Bengio and Aaron Courville
- Deep learning - buzzword for Artifical Neural Networks
- What is it?
- Supervised learning model - Classifier
- Unsupervised model - Anomaly detection
- Needs lots of data
- Online learning model - backpropogation
- Optimization - Stochastic gradient descent
- Regularization - L1, L2, Dropout ** **
- Supervised
- Fully connected network
- Convolutional neural network - Eg: For classifying images
- Recurrent neural networks - Eg: For use on text, speech
- Unsupervised
- Autoencoder
A quick recap of logistic regression / linear models
(watch now 46 seconds):
– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Regression
y = mx + c
Another way to look at a linear model
– Image Credit: Michael Nielsen
Recap - Gradient descent
(1:54 seconds):
– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Recap - Stochastic Gradient descent
(2:25 seconds):
– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
HOGWILD! Parallel SGD without locks http://i.stanford.edu/hazy/papers/hogwild-nips.pdf
Why deep learning? - Linear model
(24 seconds):
– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
ReLU - Rectified linear unit or Rectifier - max(0, x)
– Image Credit: Wikipedia
Neural Network
Watch now (45 seconds)
***
– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Is decision tree a linear model?
http://datascience.stackexchange.com/questions/6787/is-decision-tree-algorithm-a-linear-or-nonlinear-algorithm
Neural Network
**
**
– Image credit: Wikipedia
Multiple hidden layers
***
– Image credit: Michael Nielsen
What does it mean to go deep? What do each of the hidden layers learn?
Watch now (1:13 seconds)
***
– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Chain rule
(f o g)’ = (f’ o g) . g’
Chain rule in neural networks
Watch later (55 seconds)
***
– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Backpropogation
Watch later (9:55 seconds)
**
**
Watch now (1: 54 seconds)
***
####How do you set the learning rate? - Step size in SGD?
Convolutional Neural Networks
**
Watch (3:55)
**
- Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
- Convolutional Neural networks blog - http://colah.github.io/posts/2014-07-Conv-Nets-Modular/
Recurrent neural network
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
**
Watch (3:55)
**
LSTM - Long short term memory
GRU - Gated recurrent unit
http://arxiv.org/pdf/1406.1078v3.pdf
Autoencoder
**
Watch (3:51)
**
Scalable Data Science
prepared by Raazesh Sainudiin and Sivanand Sivaram
supported by
and