057_DLbyABr_04b-Hands-On-MNIST-CNN

ScaDaMaLe Course site and book

This is a 2019-2021 augmentation and update of Adam Breindel's initial notebooks.

Thanks to Christian von Koch and William Anzén for their contributions towards making these materials Spark 3.0.1 and Python 3+ compliant.

Try out the first ConvNet -- the one we looked at earlier.

This code is the same, but we'll run to 20 epochs so we can get a better feel for fitting/validation/overfitting trend.

from keras.utils import to_categorical
import sklearn.datasets

train_libsvm = "/dbfs/databricks-datasets/mnist-digits/data-001/mnist-digits-train.txt"
test_libsvm = "/dbfs/databricks-datasets/mnist-digits/data-001/mnist-digits-test.txt"

X_train, y_train = sklearn.datasets.load_svmlight_file(train_libsvm, n_features=784)
X_train = X_train.toarray()

X_test, y_test = sklearn.datasets.load_svmlight_file(test_libsvm, n_features=784)
X_test = X_test.toarray()

X_train = X_train.reshape( (X_train.shape[0], 28, 28, 1) )
X_train = X_train.astype('float32')
X_train /= 255
y_train = to_categorical(y_train, num_classes=10)

X_test = X_test.reshape( (X_test.shape[0], 28, 28, 1) )
X_test = X_test.astype('float32')
X_test /= 255
y_test = to_categorical(y_test, num_classes=10)

Using TensorFlow backend.

from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D

model = Sequential()

model.add(Conv2D(8, # number of kernels 
				(4, 4), # kernel size
                padding='valid', # no padding; output will be smaller than input
                input_shape=(28, 28, 1)))

model.add(Activation('relu'))

model.add(MaxPooling2D(pool_size=(2,2)))

model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu')) # alternative syntax for applying activation

model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

history = model.fit(X_train, y_train, batch_size=128, epochs=20, verbose=2, validation_split=0.1)

scores = model.evaluate(X_test, y_test, verbose=1)

print
for i in range(len(model.metrics_names)):
	print("%s: %f" % (model.metrics_names[i], scores[i]))

WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Train on 54000 samples, validate on 6000 samples
Epoch 1/20
 - 43s - loss: 0.2952 - acc: 0.9172 - val_loss: 0.1003 - val_acc: 0.9703
Epoch 2/20
 - 43s - loss: 0.0869 - acc: 0.9739 - val_loss: 0.0626 - val_acc: 0.9837
Epoch 3/20
 - 42s - loss: 0.0613 - acc: 0.9814 - val_loss: 0.0589 - val_acc: 0.9833
Epoch 4/20
 - 41s - loss: 0.0474 - acc: 0.9859 - val_loss: 0.0540 - val_acc: 0.9857
Epoch 5/20
 - 44s - loss: 0.0394 - acc: 0.9878 - val_loss: 0.0480 - val_acc: 0.9862
Epoch 6/20
 - 48s - loss: 0.0315 - acc: 0.9906 - val_loss: 0.0483 - val_acc: 0.9870
Epoch 7/20
 - 48s - loss: 0.0276 - acc: 0.9913 - val_loss: 0.0535 - val_acc: 0.9873
Epoch 8/20
 - 55s - loss: 0.0223 - acc: 0.9932 - val_loss: 0.0455 - val_acc: 0.9868
Epoch 9/20
 - 58s - loss: 0.0179 - acc: 0.9947 - val_loss: 0.0521 - val_acc: 0.9873
Epoch 10/20
 - 57s - loss: 0.0172 - acc: 0.9946 - val_loss: 0.0462 - val_acc: 0.9887
Epoch 11/20
 - 56s - loss: 0.0150 - acc: 0.9953 - val_loss: 0.0469 - val_acc: 0.9887
Epoch 12/20
 - 57s - loss: 0.0124 - acc: 0.9962 - val_loss: 0.0479 - val_acc: 0.9887
Epoch 13/20
 - 57s - loss: 0.0099 - acc: 0.9969 - val_loss: 0.0528 - val_acc: 0.9873
Epoch 14/20
 - 57s - loss: 0.0091 - acc: 0.9973 - val_loss: 0.0607 - val_acc: 0.9827
Epoch 15/20
 - 53s - loss: 0.0072 - acc: 0.9981 - val_loss: 0.0548 - val_acc: 0.9887
Epoch 16/20
 - 51s - loss: 0.0071 - acc: 0.9979 - val_loss: 0.0525 - val_acc: 0.9892
Epoch 17/20
 - 52s - loss: 0.0055 - acc: 0.9982 - val_loss: 0.0512 - val_acc: 0.9877
Epoch 18/20
 - 52s - loss: 0.0073 - acc: 0.9977 - val_loss: 0.0559 - val_acc: 0.9885
Epoch 19/20
 - 52s - loss: 0.0037 - acc: 0.9990 - val_loss: 0.0522 - val_acc: 0.9893
Epoch 20/20
 - 51s - loss: 0.0073 - acc: 0.9976 - val_loss: 0.0938 - val_acc: 0.9792

   32/10000 [..............................] - ETA: 5s
  128/10000 [..............................] - ETA: 5s
  224/10000 [..............................] - ETA: 5s
  352/10000 [>.............................] - ETA: 5s
  448/10000 [>.............................] - ETA: 5s
  576/10000 [>.............................] - ETA: 4s
  704/10000 [=>............................] - ETA: 4s
  832/10000 [=>............................] - ETA: 4s
  960/10000 [=>............................] - ETA: 4s
 1056/10000 [==>...........................] - ETA: 4s
 1152/10000 [==>...........................] - ETA: 4s
 1248/10000 [==>...........................] - ETA: 4s
 1312/10000 [==>...........................] - ETA: 4s
 1440/10000 [===>..........................] - ETA: 4s
 1568/10000 [===>..........................] - ETA: 4s
 1632/10000 [===>..........................] - ETA: 4s
 1760/10000 [====>.........................] - ETA: 4s
 1888/10000 [====>.........................] - ETA: 4s
 2016/10000 [=====>........................] - ETA: 4s
 2144/10000 [=====>........................] - ETA: 4s
 2240/10000 [=====>........................] - ETA: 4s
 2368/10000 [======>.......................] - ETA: 4s
 2464/10000 [======>.......................] - ETA: 4s
 2560/10000 [======>.......................] - ETA: 4s
 2624/10000 [======>.......................] - ETA: 4s
 2752/10000 [=======>......................] - ETA: 4s
 2848/10000 [=======>......................] - ETA: 4s
 2944/10000 [=======>......................] - ETA: 4s
 3072/10000 [========>.....................] - ETA: 4s
 3168/10000 [========>.....................] - ETA: 3s
 3264/10000 [========>.....................] - ETA: 3s
 3328/10000 [========>.....................] - ETA: 3s
 3456/10000 [=========>....................] - ETA: 3s
 3584/10000 [=========>....................] - ETA: 3s
 3744/10000 [==========>...................] - ETA: 3s
 3840/10000 [==========>...................] - ETA: 3s
 3968/10000 [==========>...................] - ETA: 3s
 4064/10000 [===========>..................] - ETA: 3s
 4160/10000 [===========>..................] - ETA: 3s
 4256/10000 [===========>..................] - ETA: 3s
 4352/10000 [============>.................] - ETA: 3s
 4448/10000 [============>.................] - ETA: 3s
 4544/10000 [============>.................] - ETA: 3s
 4672/10000 [=============>................] - ETA: 3s
 4800/10000 [=============>................] - ETA: 2s
 4928/10000 [=============>................] - ETA: 2s
 5024/10000 [==============>...............] - ETA: 2s
 5120/10000 [==============>...............] - ETA: 2s
 5248/10000 [==============>...............] - ETA: 2s
 5376/10000 [===============>..............] - ETA: 2s
 5504/10000 [===============>..............] - ETA: 2s
 5600/10000 [===============>..............] - ETA: 2s
 5696/10000 [================>.............] - ETA: 2s
 5856/10000 [================>.............] - ETA: 2s
 5984/10000 [================>.............] - ETA: 2s
 6080/10000 [=================>............] - ETA: 2s
 6176/10000 [=================>............] - ETA: 2s
 6272/10000 [=================>............] - ETA: 2s
 6400/10000 [==================>...........] - ETA: 1s
 6528/10000 [==================>...........] - ETA: 1s
 6624/10000 [==================>...........] - ETA: 1s
 6720/10000 [===================>..........] - ETA: 1s
 6816/10000 [===================>..........] - ETA: 1s
 6944/10000 [===================>..........] - ETA: 1s
 7104/10000 [====================>.........] - ETA: 1s
 7168/10000 [====================>.........] - ETA: 1s
 7296/10000 [====================>.........] - ETA: 1s
 7424/10000 [=====================>........] - ETA: 1s
 7552/10000 [=====================>........] - ETA: 1s
 7616/10000 [=====================>........] - ETA: 1s
 7744/10000 [======================>.......] - ETA: 1s
 7840/10000 [======================>.......] - ETA: 1s
 7968/10000 [======================>.......] - ETA: 1s
 8064/10000 [=======================>......] - ETA: 1s
 8192/10000 [=======================>......] - ETA: 0s
 8288/10000 [=======================>......] - ETA: 0s
 8384/10000 [========================>.....] - ETA: 0s
 8512/10000 [========================>.....] - ETA: 0s
 8640/10000 [========================>.....] - ETA: 0s
 8736/10000 [=========================>....] - ETA: 0s
 8864/10000 [=========================>....] - ETA: 0s
 8960/10000 [=========================>....] - ETA: 0s
 9088/10000 [==========================>...] - ETA: 0s
 9216/10000 [==========================>...] - ETA: 0s
 9344/10000 [===========================>..] - ETA: 0s
 9472/10000 [===========================>..] - ETA: 0s
 9568/10000 [===========================>..] - ETA: 0s
 9664/10000 [===========================>..] - ETA: 0s
 9760/10000 [============================>.] - ETA: 0s
 9856/10000 [============================>.] - ETA: 0s
 9952/10000 [============================>.] - ETA: 0s
10000/10000 [==============================] - 6s 551us/step
loss: 0.092008
acc: 0.978300

import matplotlib.pyplot as plt

fig, ax = plt.subplots()
fig.set_size_inches((5,5))
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
display(fig)

Next, let's try adding another convolutional layer:

model = Sequential()

model.add(Conv2D(8, # number of kernels 
						(4, 4), # kernel size
                        padding='valid',
                        input_shape=(28, 28, 1)))

model.add(Activation('relu'))

model.add(Conv2D(8, (4, 4))) # <-- additional Conv layer
model.add(Activation('relu'))

model.add(MaxPooling2D(pool_size=(2,2)))

model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))

model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

history = model.fit(X_train, y_train, batch_size=128, epochs=15, verbose=2, validation_split=0.1)

scores = model.evaluate(X_test, y_test, verbose=1)

print
for i in range(len(model.metrics_names)):
	print("%s: %f" % (model.metrics_names[i], scores[i]))

Train on 54000 samples, validate on 6000 samples
Epoch 1/15
 - 104s - loss: 0.2522 - acc: 0.9276 - val_loss: 0.0790 - val_acc: 0.9803
Epoch 2/15
 - 103s - loss: 0.0719 - acc: 0.9780 - val_loss: 0.0605 - val_acc: 0.9827
Epoch 3/15
 - 103s - loss: 0.0488 - acc: 0.9848 - val_loss: 0.0586 - val_acc: 0.9842
Epoch 4/15
 - 95s - loss: 0.0393 - acc: 0.9879 - val_loss: 0.0468 - val_acc: 0.9885
Epoch 5/15
 - 86s - loss: 0.0309 - acc: 0.9903 - val_loss: 0.0451 - val_acc: 0.9888
Epoch 6/15
 - 94s - loss: 0.0252 - acc: 0.9923 - val_loss: 0.0449 - val_acc: 0.9883
Epoch 7/15
 - 91s - loss: 0.0195 - acc: 0.9938 - val_loss: 0.0626 - val_acc: 0.9875
Epoch 8/15
 - 93s - loss: 0.0146 - acc: 0.9954 - val_loss: 0.0500 - val_acc: 0.9885
Epoch 9/15
 - 94s - loss: 0.0122 - acc: 0.9964 - val_loss: 0.0478 - val_acc: 0.9897
Epoch 10/15
 - 95s - loss: 0.0113 - acc: 0.9962 - val_loss: 0.0515 - val_acc: 0.9895
Epoch 11/15
 - 96s - loss: 0.0085 - acc: 0.9971 - val_loss: 0.0796 - val_acc: 0.9860
Epoch 12/15
 - 94s - loss: 0.0091 - acc: 0.9967 - val_loss: 0.0535 - val_acc: 0.9878
Epoch 13/15
 - 94s - loss: 0.0066 - acc: 0.9979 - val_loss: 0.0672 - val_acc: 0.9873
Epoch 14/15
 - 95s - loss: 0.0064 - acc: 0.9977 - val_loss: 0.0594 - val_acc: 0.9883
Epoch 15/15
 - 88s - loss: 0.0055 - acc: 0.9983 - val_loss: 0.0679 - val_acc: 0.9877

   32/10000 [..............................] - ETA: 6s
  128/10000 [..............................] - ETA: 6s
  224/10000 [..............................] - ETA: 6s
  320/10000 [..............................] - ETA: 5s
  448/10000 [>.............................] - ETA: 5s
  512/10000 [>.............................] - ETA: 6s
  608/10000 [>.............................] - ETA: 6s
  704/10000 [=>............................] - ETA: 6s
  768/10000 [=>............................] - ETA: 6s
  864/10000 [=>............................] - ETA: 6s
  960/10000 [=>............................] - ETA: 6s
 1056/10000 [==>...........................] - ETA: 6s
 1120/10000 [==>...........................] - ETA: 6s
 1184/10000 [==>...........................] - ETA: 6s
 1312/10000 [==>...........................] - ETA: 6s
 1408/10000 [===>..........................] - ETA: 6s
 1472/10000 [===>..........................] - ETA: 6s
 1568/10000 [===>..........................] - ETA: 5s
 1664/10000 [===>..........................] - ETA: 5s
 1728/10000 [====>.........................] - ETA: 5s
 1824/10000 [====>.........................] - ETA: 5s
 1920/10000 [====>.........................] - ETA: 5s
 1984/10000 [====>.........................] - ETA: 5s
 2080/10000 [=====>........................] - ETA: 5s
 2176/10000 [=====>........................] - ETA: 5s
 2240/10000 [=====>........................] - ETA: 5s
 2336/10000 [======>.......................] - ETA: 5s
 2400/10000 [======>.......................] - ETA: 5s
 2464/10000 [======>.......................] - ETA: 5s
 2560/10000 [======>.......................] - ETA: 5s
 2656/10000 [======>.......................] - ETA: 5s
 2752/10000 [=======>......................] - ETA: 5s
 2848/10000 [=======>......................] - ETA: 5s
 2976/10000 [=======>......................] - ETA: 4s
 3072/10000 [========>.....................] - ETA: 4s
 3168/10000 [========>.....................] - ETA: 4s
 3232/10000 [========>.....................] - ETA: 4s
 3296/10000 [========>.....................] - ETA: 4s
 3392/10000 [=========>....................] - ETA: 4s
 3488/10000 [=========>....................] - ETA: 4s
 3552/10000 [=========>....................] - ETA: 4s
 3616/10000 [=========>....................] - ETA: 4s
 3712/10000 [==========>...................] - ETA: 4s
 3808/10000 [==========>...................] - ETA: 4s
 3904/10000 [==========>...................] - ETA: 4s
 3968/10000 [==========>...................] - ETA: 4s
 4064/10000 [===========>..................] - ETA: 4s
 4160/10000 [===========>..................] - ETA: 4s
 4256/10000 [===========>..................] - ETA: 4s
 4352/10000 [============>.................] - ETA: 4s
 4448/10000 [============>.................] - ETA: 3s
 4544/10000 [============>.................] - ETA: 3s
 4640/10000 [============>.................] - ETA: 3s
 4736/10000 [=============>................] - ETA: 3s
 4832/10000 [=============>................] - ETA: 3s
 4928/10000 [=============>................] - ETA: 3s
 5024/10000 [==============>...............] - ETA: 3s
 5120/10000 [==============>...............] - ETA: 3s
 5216/10000 [==============>...............] - ETA: 3s
 5344/10000 [===============>..............] - ETA: 3s
 5408/10000 [===============>..............] - ETA: 3s
 5504/10000 [===============>..............] - ETA: 3s
 5600/10000 [===============>..............] - ETA: 3s
 5696/10000 [================>.............] - ETA: 2s
 5760/10000 [================>.............] - ETA: 2s
 5856/10000 [================>.............] - ETA: 2s
 5952/10000 [================>.............] - ETA: 2s
 6048/10000 [=================>............] - ETA: 2s
 6144/10000 [=================>............] - ETA: 2s
 6240/10000 [=================>............] - ETA: 2s
 6304/10000 [=================>............] - ETA: 2s
 6400/10000 [==================>...........] - ETA: 2s
 6496/10000 [==================>...........] - ETA: 2s
 6592/10000 [==================>...........] - ETA: 2s
 6688/10000 [===================>..........] - ETA: 2s
 6784/10000 [===================>..........] - ETA: 2s
 6848/10000 [===================>..........] - ETA: 2s
 6944/10000 [===================>..........] - ETA: 2s
 7008/10000 [====================>.........] - ETA: 2s
 7072/10000 [====================>.........] - ETA: 2s
 7168/10000 [====================>.........] - ETA: 1s
 7264/10000 [====================>.........] - ETA: 1s
 7360/10000 [=====================>........] - ETA: 1s
 7456/10000 [=====================>........] - ETA: 1s
 7552/10000 [=====================>........] - ETA: 1s
 7648/10000 [=====================>........] - ETA: 1s
 7744/10000 [======================>.......] - ETA: 1s
 7840/10000 [======================>.......] - ETA: 1s
 7936/10000 [======================>.......] - ETA: 1s
 8032/10000 [=======================>......] - ETA: 1s
 8128/10000 [=======================>......] - ETA: 1s
 8224/10000 [=======================>......] - ETA: 1s
 8320/10000 [=======================>......] - ETA: 1s
 8416/10000 [========================>.....] - ETA: 1s
 8512/10000 [========================>.....] - ETA: 1s
 8576/10000 [========================>.....] - ETA: 0s
 8672/10000 [=========================>....] - ETA: 0s
 8768/10000 [=========================>....] - ETA: 0s
 8864/10000 [=========================>....] - ETA: 0s
 8960/10000 [=========================>....] - ETA: 0s
 9056/10000 [==========================>...] - ETA: 0s
 9152/10000 [==========================>...] - ETA: 0s
 9248/10000 [==========================>...] - ETA: 0s
 9344/10000 [===========================>..] - ETA: 0s
 9440/10000 [===========================>..] - ETA: 0s
 9536/10000 [===========================>..] - ETA: 0s
 9632/10000 [===========================>..] - ETA: 0s
 9728/10000 [============================>.] - ETA: 0s
 9824/10000 [============================>.] - ETA: 0s
 9920/10000 [============================>.] - ETA: 0s
 9984/10000 [============================>.] - ETA: 0s
10000/10000 [==============================] - 7s 683us/step
loss: 0.046070
acc: 0.988900

Still Overfitting

We're making progress on our test error -- about 99% -- but just a bit for all the additional time, due to the network overfitting the data.

There are a variety of techniques we can take to counter this -- forms of regularization.

Let's try a relatively simple solution that works surprisingly well: add a pair of Dropout filters, a layer that randomly omits a fraction of neurons from each training batch (thus exposing each neuron to only part of the training data).

We'll add more convolution kernels but shrink them to 3x3 as well.

model = Sequential()

model.add(Conv2D(32, # number of kernels 
						(3, 3), # kernel size
                        padding='valid',
                        input_shape=(28, 28, 1)))

model.add(Activation('relu'))

model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))

model.add(MaxPooling2D(pool_size=(2,2)))

model.add(Dropout(rate=1-0.25)) #new parameter rate added (rate=1-keep_prob)
model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))

model.add(Dropout(rate=1-0.5)) #new parameter rate added (rate=1-keep_prob)
model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
history = model.fit(X_train, y_train, batch_size=128, epochs=15, verbose=2)

scores = model.evaluate(X_test, y_test, verbose=2)

print
for i in range(len(model.metrics_names)):
	print("%s: %f" % (model.metrics_names[i], scores[i]))

WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:3733: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
Epoch 1/15
 - 339s - loss: 0.3906 - acc: 0.8762
Epoch 2/15
 - 334s - loss: 0.1601 - acc: 0.9516
Epoch 3/15
 - 271s - loss: 0.1282 - acc: 0.9610
Epoch 4/15
 - 248s - loss: 0.1108 - acc: 0.9664
Epoch 5/15
 - 245s - loss: 0.0972 - acc: 0.9705
Epoch 6/15
 - 249s - loss: 0.0903 - acc: 0.9720
Epoch 7/15
 - 245s - loss: 0.0859 - acc: 0.9737
Epoch 8/15
 - 245s - loss: 0.0828 - acc: 0.9742
Epoch 9/15
 - 249s - loss: 0.0786 - acc: 0.9756
Epoch 10/15
 - 247s - loss: 0.0763 - acc: 0.9764
Epoch 11/15
 - 247s - loss: 0.0752 - acc: 0.9765
Epoch 12/15
 - 244s - loss: 0.0694 - acc: 0.9782
Epoch 13/15
 - 247s - loss: 0.0693 - acc: 0.9786
Epoch 14/15
 - 171s - loss: 0.0655 - acc: 0.9802
Epoch 15/15
 - 157s - loss: 0.0647 - acc: 0.9801
loss: 0.023367
acc: 0.992200

Lab Wrapup

From the last lab, you should have a test accuracy of over 99.1%

For one more activity, try changing the optimizer to old-school "sgd" -- just to see how far we've come with these modern gradient descent techniques in the last few years.

Accuracy will end up noticeably worse ... about 96-97% test accuracy. Two key takeaways:

Without a good optimizer, even a very powerful network design may not achieve results
In fact, we could replace the word "optimizer" there with
- initialization
- activation
- regularization
- (etc.)
All of these elements we've been working with operate together in a complex way to determine final performance

sds-3.x/ScaDaMaLe

Still Overfitting

Lab Wrapup