// Databricks notebook source exported at Tue, 28 Jun 2016 09:28:40 UTC
Scalable Data Science
prepared by Raazesh Sainudiin and Sivanand Sivaram
The html source url of this databricks notebook and its recorded Uji :
Deep learning: A Crash Introduction
This notebook provides an introduction to Deep Learning. It is meant to help you descend more fully into these learning resources and references:
 Udacity’s course on Deep Learning https://www.udacity.com/course/deeplearning–ud730 by Arpan Chakraborty and Vincent Vanhoucke
 Neural networks and deep learning http://neuralnetworksanddeeplearning.com/ by Michael Nielsen

Deep learning book http://www.deeplearningbook.org/ by Ian Goodfellow, Yoshua Bengio and Aaron Courville
 Deep learning  buzzword for Artifical Neural Networks
 What is it?
 Supervised learning model  Classifier
 Unsupervised model  Anomaly detection
 Needs lots of data
 Online learning model  backpropogation
 Optimization  Stochastic gradient descent
 Regularization  L1, L2, Dropout ** **
 Supervised
 Fully connected network
 Convolutional neural network  Eg: For classifying images
 Recurrent neural networks  Eg: For use on text, speech
 Unsupervised
 Autoencoder
A quick recap of logistic regression / linear models
(watch now 46 seconds):
– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Regression
y = mx + c
Another way to look at a linear model
– Image Credit: Michael Nielsen
Recap  Gradient descent
(1:54 seconds):
– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Recap  Stochastic Gradient descent
(2:25 seconds):
– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
HOGWILD! Parallel SGD without locks http://i.stanford.edu/hazy/papers/hogwildnips.pdf
Why deep learning?  Linear model
(24 seconds):
– Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
ReLU  Rectified linear unit or Rectifier  max(0, x)
– Image Credit: Wikipedia
Neural Network
Watch now (45 seconds)
*** – Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Is decision tree a linear model?
http://datascience.stackexchange.com/questions/6787/isdecisiontreealgorithmalinearornonlinearalgorithm
Neural Network ** ** – Image credit: Wikipedia
Multiple hidden layers
*** – Image credit: Michael Nielsen
What does it mean to go deep? What do each of the hidden layers learn?
Watch now (1:13 seconds)
*** – Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Chain rule
(f o g)’ = (f’ o g) . g’
Chain rule in neural networks
Watch later (55 seconds)
*** – Video Credit: Udacity’s deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Backpropogation
Watch later (9:55 seconds)
####How do you set the learning rate?  Step size in SGD?
Convolutional Neural Networks
 Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton  https://papers.nips.cc/paper/4824imagenetclassificationwithdeepconvolutionalneuralnetworks.pdf
 Convolutional Neural networks blog  http://colah.github.io/posts/201407ConvNetsModular/
Recurrent neural network
http://colah.github.io/posts/201508UnderstandingLSTMs/
http://karpathy.github.io/2015/05/21/rnneffectiveness/
**
Watch (3:55)
**
LSTM  Long short term memory
GRU  Gated recurrent unit
http://arxiv.org/pdf/1406.1078v3.pdf