%md

ScaDaMaLe Course [site](https://lamastex.github.io/scalable-data-science/sds/3/x/) and [book](https://lamastex.github.io/ScaDaMaLe/index.html)

This is a 2019-2021 augmentation and update of [Adam Breindel](https://www.linkedin.com/in/adbreind)'s initial notebooks.

_Thanks to [Christian von Koch](https://www.linkedin.com/in/christianvonkoch/) and [William Anzén](https://www.linkedin.com/in/william-anz%C3%A9n-b52003199/) for their contributions towards making these materials Spark 3.0.1 and Python 3+ compliant._

%md 

# Convolutional Neural Networks

## aka CNN, ConvNet

%md 

As a baseline, let's start a lab running with what we already know.

We'll take our deep feed-forward multilayer perceptron network, with ReLU activations and reasonable initializations, and apply it to learning the MNIST digits.

The main part of the code looks like the following (full code you can run is in the next cell):

```

# imports, setup, load data sets

model = Sequential()

model.add(Dense(20, input_dim=784, kernel_initializer='normal', activation='relu'))

model.add(Dense(15, kernel_initializer='normal', activation='relu'))

model.add(Dense(10, kernel_initializer='normal', activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['categorical_accuracy'])

categorical_labels = to_categorical(y_train, num_classes=10)

history = model.fit(X_train, categorical_labels, epochs=100, batch_size=100)

# print metrics, plot errors

```

Note the changes, which are largely about building a classifier instead of a regression model:

* Output layer has one neuron per category, with softmax activation

* __Loss function is cross-entropy loss__

* Accuracy metric is categorical accuracy

%md

Let's hold pointers into wikipedia for these new concepts.

%scala

//This allows easy embedding of publicly available information into any other notebook

//Example usage:

// displayHTML(frameIt("https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation#Topics_in_LDA",250))

def frameIt( u:String, h:Int ) : String = {

      """<iframe 

 src=""""+ u+""""

 width="95%" height="""" + h + """"

 sandbox>

  <p>

    <a href="http://spark.apache.org/docs/latest/index.html">

      Fallback link for browsers that, unlikely, don't support frames

    </a>

  </p>

</iframe>"""

   }

displayHTML(frameIt("https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_error_function_and_logistic_regression",500))

%scala

displayHTML(frameIt("https://en.wikipedia.org/wiki/Softmax_function",380))

%md

The following is from: [https://www.quora.com/How-does-Keras-calculate-accuracy](https://www.quora.com/How-does-Keras-calculate-accuracy).

**Categorical accuracy:**

```%python

def categorical_accuracy(y_true, y_pred):

 return K.cast(K.equal(K.argmax(y_true, axis=-1),

 K.argmax(y_pred, axis=-1)),

 K.floatx())

 ```

> `K.argmax(y_true)` takes the highest value to be the prediction and matches against the comparative set.

%md 

after about a minute we have:

```

...

Epoch 40/40

1s - loss: 0.0610 - categorical_accuracy: 0.9809 - val_loss: 0.1918 - val_categorical_accuracy: 0.9583

...

loss: 0.216120

categorical_accuracy: 0.955000

Start: 2017-12-06 07:35:33.948102

End: 2017-12-06 07:36:27.046130

Elapse: 0:00:53.098028

```

%md 

What are the big takeaways from this experiment?

1. We get pretty impressive "apparent error" accuracy right from the start! A small network gets us to training accuracy 97% by epoch 20

2. The model *appears* to continue to learn if we let it run, although it does slow down and oscillate a bit.

3. Our test accuracy is about 95% after 5 epochs and never gets better ... it gets worse!

4. Therefore, we are overfitting very quickly... most of the "training" turns out to be a waste.

5. For what it's worth, we get 95% accuracy without much work.

This is not terrible compared to other, non-neural-network approaches to the problem. After all, we could probably tweak this a bit and do even better.

But we talked about using deep learning to solve "95%" problems or "98%" problems ... where one error in 20, or 50 simply won't work. If we can get to "multiple nines" of accuracy, then we can do things like automate mail sorting and translation, create cars that react properly (all the time) to street signs, and control systems for robots or drones that function autonomously.

Try two more experiments (try them separately):

1. Add a third, hidden layer.

2. Increase the size of the hidden layers.

Adding another layer slows things down a little (why?) but doesn't seem to make a difference in accuracy.

Adding a lot more neurons into the first topology slows things down significantly -- 10x as many neurons, and only a marginal increase in accuracy. Notice also (in the plot) that the learning clearly degrades after epoch 50 or so.

... We need a new approach!

---

... let's think about this:

### What is layer 2 learning from layer 1? Combinations of pixels

#### Combinations of pixels contain information but...

There are a lot of them (combinations) and they are "fragile" 

In fact, in our last experiment, we basically built a model that memorizes a bunch of "magic" pixel combinations.

What might be a better way to build features?

* When humans perform this task, we look not at arbitrary pixel combinations, but certain geometric patterns -- lines, curves, loops.

* These features are made up of combinations of pixels, but they are far from arbitrary

* We identify these features regardless of translation, rotation, etc.

Is there a way to get the network to do the same thing?

I.e., in layer one, identify pixels. Then in layer 2+, identify abstractions over pixels that are translation-invariant 2-D shapes?

We could look at where a "filter" that represents one of these features (e.g., and edge) matches the image.

How would this work?

### Convolution

Convolution in the general mathematical sense is define as follows:

<img src="https://i.imgur.com/lurC2Cx.png" width=300>

The convolution we deal with in deep learning is a simplified case. We want to compare two signals. Here are two visualizations, courtesy of Wikipedia, that help communicate how convolution emphasizes features:

<img src="http://i.imgur.com/EDCaMl2.png" width=500>

---

#### Here's an animation (where we change \\({\tau}\\)) 

<img src="http://i.imgur.com/0BFcnaw.gif">

__In one sense, the convolution captures and quantifies the pattern matching over space__

If we perform this in two dimensions, we can achieve effects like highlighting edges:

<img src="http://i.imgur.com/DKEXIII.png">

The matrix here, also called a convolution kernel, is one of the functions we are convolving. Other convolution kernels can blur, "sharpen," etc.

### So we'll drop in a number of convolution kernels, and the network will learn where to use them? Nope. Better than that.

## We'll program in the *idea* of discrete convolution, and the network will learn what kernels extract meaningful features!

The values in a (fixed-size) convolution kernel matrix will be variables in our deep learning model. Although inuitively it seems like it would be hard to learn useful params, in fact, since those variables are used repeatedly across the image data, it "focuses" the error on a smallish number of parameters with a lot of influence -- so it should be vastly *less* expensive to train than just a huge fully connected layer like we discussed above.

This idea was developed in the late 1980s, and by 1989, Yann LeCun (at AT&T/Bell Labs) had built a practical high-accuracy system (used in the 1990s for processing handwritten checks and mail).

__How do we hook this into our neural networks?__

* First, we can preserve the geometric properties of our data by "shaping" the vectors as 2D instead of 1D.

* Then we'll create a layer whose value is not just activation applied to weighted sum of inputs, but instead it's the result of a dot-product (element-wise multiply and sum) between the kernel and a patch of the input vector (image).

    * This value will be our "pre-activation" and optionally feed into an activation function (or "detector")

<img src="http://i.imgur.com/ECyi9lL.png">

If we perform this operation at lots of positions over the image, we'll get lots of outputs, as many as one for every input pixel. 

<img src="http://i.imgur.com/WhOrJ0Y.jpg">

* So we'll add another layer that "picks" the highest convolution pattern match from nearby pixels, which

    * makes our pattern match a little bit translation invariant (a fuzzy location match)

    * reduces the number of outputs significantly

* This layer is commonly called a pooling layer, and if we pick the "maximum match" then it's a "max pooling" layer.

<img src="http://i.imgur.com/9iPpfpb.png">

__The end result is that the kernel or filter together with max pooling creates a value in a subsequent layer which represents the appearance of a pattern in a local area in a prior layer.__

__Again, the network will be given a number of "slots" for these filters and will learn (by minimizing error) what filter values produce meaningful features. This is the key insight into how modern image-recognition networks are able to generalize -- i.e., learn to tell 6s from 7s or cats from dogs.__

<img src="http://i.imgur.com/F8eH3vj.png">

## Ok, let's build our first ConvNet:

First, we want to explicity shape our data into a 2-D configuration. We'll end up with a 4-D tensor where the first dimension is the training examples, then each example is 28x28 pixels, and we'll explicitly say it's 1-layer deep. (Why? with color images, we typically process over 3 or 4 channels in this last dimension)

A step by step animation follows:

* http://cs231n.github.io/assets/conv-demo/index.html

%md 

Now the model: