058_DLbyABr_04c-CIFAR-10(Python)
Import Notebook

SDS-2.x, Scalable Data Engineering Science

This is a 2019 augmentation and update of Adam Breindel's initial notebooks.

CIFAR 10

Details at: https://www.cs.toronto.edu/~kriz/cifar.html

Summary (taken from that page):

The CIFAR-10 and CIFAR-100 are labeled subsets of the 80 million tiny images dataset. They were collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.

The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.

First, we'll mount the S3 bucket where I'm hosting the data:

# you may have to host the data yourself! - this should not work unless you can descramble
ACCESS="...SPORAA...KIAJZEH...PW46CWPUWUN...QPODO" # scrambled up
SECRET="...P7d7Sp7r1...Q9DuUvV...QAy1D+hjC...NxakJF+PXrAb...MXD1tZwBpGyN...1Ns5r5n1" # scrambled up
BUCKET = "cool-data"
MOUNT = "/mnt/cifar"

try:
  dbutils.fs.mount("s3a://"+ ACCESS + ":" + SECRET + "@" + BUCKET, MOUNT)
except:
  print("Error mounting ... possibly already mounted")

This is in DBFS, which is available (via FUSE) at /dbfs ...

So the CIFAR data can be listed through following regular Linux shell command:

%sh
wget http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
--2019-06-17 14:01:25-- http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz Resolving www.cs.toronto.edu (www.cs.toronto.edu)... 128.100.3.30 Connecting to www.cs.toronto.edu (www.cs.toronto.edu)|128.100.3.30|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 170498071 (163M) [application/x-gzip] Saving to: ‘cifar-10-python.tar.gz’ 0K .......... .......... .......... .......... .......... 0% 391K 7m6s 50K .......... .......... .......... .......... .......... 0% 791K 5m18s 100K .......... .......... .......... .......... .......... 0% 18.8M 3m35s 150K .......... .......... .......... .......... .......... 0% 26.2M 2m43s 200K .......... .......... .......... .......... .......... 0% 811K 2m51s 250K .......... .......... .......... .......... .......... 0% 19.4M 2m24s 300K .......... .......... .......... .......... .......... 0% 23.3M 2m4s 350K .......... .......... .......... .......... .......... 0% 26.9M 1m50s 400K .......... .......... .......... .......... .......... 0% 898K 1m58s 450K .......... .......... .......... .......... .......... 0% 21.3M 1m47s 500K .......... .......... .......... .......... .......... 0% 21.7M 98s 550K .......... .......... .......... .......... .......... 0% 19.1M 90s 600K .......... .......... .......... .......... .......... 0% 46.4M 84s 650K .......... .......... .......... .......... .......... 0% 95.3M 78s 700K .......... .......... .......... .......... .......... 0% 186M 73s 750K .......... .......... .......... .......... .......... 0% 129M 68s 800K .......... .......... .......... .......... .......... 0% 222M 64s 850K .......... .......... .......... .......... .......... 0% 926K 70s 900K .......... .......... .......... .......... .......... 0% 35.4M 67s 950K .......... .......... .......... .......... .......... 0% 36.2M 64s 1000K .......... .......... .......... .......... .......... 0% 49.1M 61s 1050K .......... .......... .......... .......... .......... 0% 60.1M 58s 1100K .......... .......... .......... .......... .......... 0% 34.8M 56s 1150K .......... .......... .......... .......... .......... 0% 42.4M 54s 1200K .......... .......... .......... .......... .......... 0% 41.5M 52s 1250K .......... .......... .......... .......... .......... 0% 78.3M 50s 1300K .......... .......... .......... .......... .......... 0% 79.7M 48s 1350K .......... .......... .......... .......... .......... 0% 63.7M 46s 1400K .......... .......... .......... .......... .......... 0% 74.9M 45s 1450K .......... .......... .......... .......... .......... 0% 64.0M 43s 1500K .......... .......... .......... .......... .......... 0% 49.8M 42s 1550K .......... .......... .......... .......... .......... 0% 172M 41s 1600K .......... .......... .......... .......... .......... 0% 256M 40s 1650K .......... .......... .......... .......... .......... 1% 268M 38s 1700K .......... .......... .......... .......... .......... 1% 268M 37s 1750K .......... .......... .......... .......... .......... 1% 995K 41s 1800K .......... .......... .......... .......... .......... 1% 54.9M 40s 1850K .......... .......... .......... .......... .......... 1% 57.9M 39s 1900K .......... .......... .......... .......... .......... 1% 93.9M 38s 1950K .......... .......... .......... .......... .......... 1% 68.3M 37s 2000K .......... .......... .......... .......... .......... 1% 50.0M 36s 2050K .......... .......... .......... .......... .......... 1% 60.6M 35s 2100K .......... .......... .......... .......... .......... 1% 47.3M 35s 2150K .......... .......... .......... .......... .......... 1% 157M 34s 2200K .......... .......... .......... .......... .......... 1% 109M 33s 2250K .......... .......... .......... .......... .......... 1% 125M 32s 2300K .......... .......... .......... .......... .......... 1% 260M 32s 2350K .......... .......... .......... .......... .......... 1% 71.9M 31s 2400K .......... .......... .......... .......... .......... 1% 78.7M 30s 2450K .......... .......... .......... .......... .......... 1% 78.0M 30s 2500K .......... .......... .......... .......... .......... 1% 88.6M 29s 2550K .......... .......... .......... .......... .......... 1% 56.8M 29s 2600K .......... .......... .......... .......... .......... 1% 65.8M 28s 2650K .......... .......... .......... .......... .......... 1% 52.3M 28s 2700K .......... .......... .......... .......... .......... 1% 80.7M 27s 2750K .......... .......... .......... .......... .......... 1% 47.4M 27s 2800K .......... .......... .......... .......... .......... 1% 197M 26s 2850K .......... .......... .......... .......... .......... 1% 48.8M 26s 2900K .......... .......... .......... .......... .......... 1% 114M 26s 2950K .......... .......... .......... .......... .......... 1% 69.0M 25s 3000K .......... .......... .......... .......... .......... 1% 167M 25s 3050K .......... .......... .......... .......... .......... 1% 264M 24s 3100K .......... .......... .......... .......... .......... 1% 249M 24s 3150K .......... .......... .......... .......... .......... 1% 225M 24s 3200K .......... .......... .......... .......... .......... 1% 262M 23s 3250K .......... .......... .......... .......... .......... 1% 254M 23s 3300K .......... .......... .......... .......... .......... 2% 1.03M 25s 3350K .......... .......... .......... .......... .......... 2% 73.3M 25s 3400K .......... .......... .......... .......... .......... 2% 71.4M 24s 3450K .......... .......... .......... .......... .......... 2% 73.9M 24s 3500K .......... .......... .......... .......... .......... 2% 39.3M 24s 3550K .......... .......... .......... .......... .......... 2% 97.7M 23s 3600K .......... .......... .......... .......... .......... 2% 82.1M 23s 3650K .......... .......... .......... .......... .......... 2% 91.3M 23s 3700K .......... .......... .......... .......... .......... 2% 57.1M 22s 3750K .......... .......... .......... .......... .......... 2% 263M 22s 3800K .......... .......... .......... .......... .......... 2% 86.5M 22s 3850K .......... .......... .......... .......... .......... 2% 109M 22s 3900K .......... .......... .......... .......... .......... 2% 88.7M 21s 3950K .......... .......... .......... .......... .......... 2% 63.5M 21s 4000K .......... .......... .......... .......... .......... 2% 65.4M 21s 4050K .......... .......... .......... .......... .......... 2% 70.8M 21s 4100K .......... .......... .......... .......... .......... 2% 54.1M 20s 4150K .......... .......... .......... .......... .......... 2% 148M 20s 4200K .......... .......... .......... .......... .......... 2% 60.4M 20s 4250K .......... .......... .......... .......... .......... 2% 66.0M 20s 4300K .......... .......... .......... .......... .......... 2% 62.5M 20s 4350K .......... .......... .......... .......... .......... 2% 216M 19s 4400K .......... .......... .......... .......... .......... 2% 49.9M 19s 4450K .......... .......... .......... .......... .......... 2% 77.7M 19s 4500K .......... .......... .......... .......... .......... 2% 79.0M 19s 4550K .......... .......... .......... .......... .......... 2% 79.7M 19s 4600K .......... .......... .......... .......... .......... 2% 230M 18s 4650K .......... .......... .......... .......... .......... 2% 254M 18s 4700K .......... .......... .......... .......... .......... 2% 260M 18s 4750K .......... .......... .......... .......... .......... 2% 265M 18s 4800K .......... .......... .......... .......... .......... 2% 1.65M 19s 4850K .......... .......... .......... .......... .......... 2% 2.77M 19s 4900K .......... .......... .......... .......... .......... 2% 36.9M 19s 4950K .......... .......... .......... .......... .......... 3% 65.1M 19s 5000K .......... .......... .......... .......... .......... 3% 87.7M 18s 5050K .......... .......... .......... .......... .......... 3% 71.9M 18s 5100K .......... .......... .......... .......... .......... 3% 105M 18s 5150K .......... .......... .......... .......... .......... 3% 70.9M 18s 5200K .......... .......... .......... .......... .......... 3% 80.9M 18s 5250K .......... .......... .......... .......... .......... 3% 91.9M 18s 5300K .......... .......... .......... .......... .......... 3% 76.8M 18s 5350K .......... .......... .......... .......... .......... 3% 138M 17s 5400K .......... .......... .......... .......... .......... 3% 78.9M 17s 5450K .......... .......... .......... .......... .......... 3% 122M 17s 5500K .......... .......... .......... .......... .......... 3% 59.8M 17s 5550K .......... .......... .......... .......... .......... 3% 92.8M 17s 5600K .......... .......... .......... .......... .......... 3% 103M 17s 5650K .......... .......... .......... .......... .......... 3% 74.8M 17s 5700K .......... .......... .......... .......... .......... 3% 88.9M 16s 5750K .......... .......... .......... .......... .......... 3% 164M 16s 5800K .......... .......... .......... .......... .......... 3% 82.6M 16s 5850K .......... .......... .......... .......... .......... 3% 89.5M 16s 5900K .......... .......... .......... .......... .......... 3% 74.5M 16s 5950K .......... .......... .......... .......... .......... 3% 119M 16s 6000K .......... .......... .......... .......... .......... 3% 114M 16s 6050K .......... .......... .......... .......... .......... 3% 74.7M 16s 6100K .......... .......... .......... .......... .......... 3% 87.7M 15s 6150K .......... .......... .......... .......... .......... 3% 57.6M 15s 6200K .......... .......... .......... .......... .......... 3% 55.9M 15s 6250K .......... .......... .......... .......... .......... 3% 224M 15s 6300K .......... .......... .......... .......... .......... 3% 261M 15s 6350K .......... .......... .......... .......... .......... 3% 1.66M 16s 6400K .......... .......... .......... .......... .......... 3% 2.63M 16s 6450K .......... .......... .......... .......... .......... 3% 72.6M 16s 6500K .......... .......... .......... .......... .......... 3% 61.6M 16s 6550K .......... .......... .......... .......... .......... 3% 67.2M 16s 6600K .......... .......... .......... .......... .......... 3% 79.1M 15s 6650K .......... .......... .......... .......... .......... 4% 81.7M 15s 6700K .......... .......... .......... .......... .......... 4% 83.8M 15s 6750K .......... .......... .......... .......... .......... 4% 71.2M 15s 6800K .......... .......... .......... .......... .......... 4% 55.8M 15s 6850K .......... .......... .......... .......... .......... 4% 69.3M 15s 6900K .......... .......... .......... .......... .......... 4% 189M 15s 6950K .......... .......... .......... .......... .......... 4% 129M 15s 7000K .......... .......... .......... .......... .......... 4% 77.1M 15s 7050K .......... .......... .......... .......... .......... 4% 68.4M 15s 7100K .......... .......... .......... .......... .......... 4% 86.1M 14s 7150K .......... .......... .......... .......... .......... 4% 109M 14s 7200K .......... .......... .......... .......... .......... 4% 187M 14s 7250K .......... .......... .......... .......... .......... 4% 77.7M 14s 7300K .......... .......... .......... .......... .......... 4% 108M 14s 7350K .......... .......... .......... .......... .......... 4% 65.7M 14s 7400K .......... .......... .......... .......... .......... 4% 74.1M 14s 7450K .......... .......... .......... .......... .......... 4% 69.1M 14s 7500K .......... .......... .......... .......... .......... 4% 276M 14s 7550K .......... .......... .......... .......... .......... 4% 56.7M 14s 7600K .......... .......... .......... .......... .......... 4% 76.4M 14s 7650K .......... .......... .......... .......... .......... 4% 69.2M 14s 7700K .......... .......... .......... .......... .......... 4% 74.0M 13s 7750K .......... .......... .......... .......... .......... 4% 234M 13s 7800K .......... .......... .......... .......... .......... 4% 272M 13s 7850K .......... .......... .......... .......... .......... 4% 275M 13s 7900K .......... .......... .......... .......... .......... 4% 1.05M 14s 7950K .......... .......... .......... .......... .......... 4% 32.9M 14s 8000K .......... .......... .......... .......... .......... 4% 42.3M 14s 8050K .......... .......... .......... .......... .......... 4% 80.9M 14s 8100K .......... .......... .......... .......... .......... 4% 96.4M 14s 8150K .......... .......... .......... .......... .......... 4% 72.7M 14s 8200K .......... .......... .......... .......... .......... 4% 73.7M 14s 8250K .......... .......... .......... .......... .......... 4% 63.6M 14s 8300K .......... .......... .......... .......... .......... 5% 148M 13s 8350K .......... .......... .......... .......... .......... 5% 101M 13s 8400K .......... .......... .......... .......... .......... 5% 66.5M 13s 8450K .......... .......... .......... .......... .......... 5% 176M 13s 8500K .......... .......... .......... .......... .......... 5% 72.8M 13s 8550K .......... .......... .......... .......... .......... 5% 116M 13s 8600K .......... .......... .......... .......... .......... 5% 73.8M 13s 8650K .......... .......... .......... .......... .......... 5% 88.2M 13s 8700K .......... .......... .......... .......... .......... 5% 94.9M 13s 8750K .......... .......... .......... .......... .......... 5% 113M 13s 8800K .......... .......... .......... .......... .......... 5% 98.8M 13s 8850K .......... .......... .......... .......... .......... 5% 76.1M 13s 8900K .......... .......... .......... .......... .......... 5% 86.6M 13s 8950K .......... .......... .......... .......... .......... 5% 52.9M 13s 9000K .......... .......... .......... .......... .......... 5% 126M 12s 9050K .......... .......... .......... .......... .......... 5% 234M 12s 9100K .......... .......... .......... .......... .......... 5% 92.9M 12s 9150K .......... .......... .......... .......... .......... 5% 63.2M 12s 9200K .......... .......... .......... .......... .......... 5% 66.7M 12s 9250K .......... .......... .......... .......... .......... 5% 130M 12s 9300K .......... .......... .......... .......... .......... 5% 67.6M 12s 9350K .......... .......... .......... .......... .......... 5% 239M 12s 9400K .......... .......... .......... .......... .......... 5% 1.70M 12s 9450K .......... .......... .......... .......... .......... 5% 2.58M 13s 9500K .......... .......... .......... .......... .......... 5% 77.0M 13s 9550K .......... .......... .......... .......... .......... 5% 34.4M 13s 9600K .......... .......... .......... .......... .......... 5% 82.1M 13s 9650K .......... .......... .......... .......... .......... 5% 60.3M 12s 9700K .......... .......... .......... .......... .......... 5% 94.1M 12s 9750K .......... .......... .......... .......... .......... 5% 87.1M 12s 9800K .......... .......... .......... .......... .......... 5% 73.2M 12s 9850K .......... .......... .......... .......... .......... 5% 83.1M 12s 9900K .......... .......... .......... .......... .......... 5% 108M 12s 9950K .......... .......... .......... .......... .......... 6% 67.5M 12s 10000K .......... .......... .......... .......... .......... 6% 146M 12s 10050K .......... .......... .......... .......... .......... 6% 55.8M 12s 10100K .......... .......... .......... .......... .......... 6% 79.6M 12s 10150K .......... .......... .......... .......... .......... 6% 94.5M 12s 10200K .......... .......... .......... .......... .......... 6% 107M 12s 10250K .......... .......... .......... .......... .......... 6% 99.1M 12s 10300K .......... .......... .......... .......... .......... 6% 78.2M 12s 10350K .......... .......... .......... .......... .......... 6% 98.0M 12s 10400K .......... .......... .......... .......... .......... 6% 74.4M 12s 10450K .......... .......... .......... .......... .......... 6% 104M 12s 10500K .......... .......... .......... .......... .......... 6% 53.9M 12s 10550K .......... .......... .......... .......... .......... 6% 56.6M 12s 10600K .......... .......... .......... .......... .......... 6% 88.5M 11s 10650K .......... .......... .......... .......... .......... 6% 86.8M 11s 10700K .......... .......... .......... .......... .......... 6% 117M 11s 10750K .......... .......... .......... .......... .......... 6% 252M 11s 10800K .......... .......... .......... .......... .......... 6% 90.3M 11s 10850K .......... .......... .......... .......... .......... 6% 71.4M 11s 10900K .......... .......... .......... .......... .......... 6% 264M 11s 10950K .......... .......... .......... .......... .......... 6% 1.07M 12s 11000K .......... .......... .......... .......... .......... 6% 27.9M 12s 11050K .......... .......... .......... .......... .......... 6% 22.8M 12s 11100K .......... .......... .......... .......... .......... 6% 97.0M 12s 11150K .......... .......... .......... .......... .......... 6% 64.1M 12s 11200K .......... .......... .......... .......... .......... 6% 63.2M 12s 11250K .......... .......... .......... .......... .......... 6% 267M 12s 11300K .......... .......... .......... .......... .......... 6% 106M 11s 11350K .......... .......... .......... .......... .......... 6% 110M 11s 11400K .......... .......... .......... .......... .......... 6% 67.2M 11s 11450K .......... .......... .......... .......... .......... 6% 53.2M 11s 11500K .......... .......... .......... .......... .......... 6% 91.7M 11s 11550K .......... .......... .......... .......... .......... 6% 65.3M 11s 11600K .......... .......... .......... .......... .......... 6% 73.6M 11s 11650K .......... .......... .......... .......... .......... 7% 249M 11s 11700K .......... .......... .......... .......... .......... 7% 118M 11s 11750K .......... .......... .......... .......... .......... 7% 128M 11s 11800K .......... .......... .......... .......... .......... 7% 162M 11s 11850K .......... .......... .......... .......... .......... 7% 93.2M 11s 11900K .......... .......... .......... .......... .......... 7% 74.3M 11s 11950K .......... .......... .......... .......... .......... 7% 64.4M 11s 12000K .......... .......... .......... .......... .......... 7% 81.3M 11s 12050K .......... .......... .......... .......... .......... 7% 97.6M 11s 12100K .......... .......... .......... .......... .......... 7% 77.5M 11s 12150K .......... .......... .......... .......... .......... 7% 84.4M 11s 12200K .......... .......... .......... .......... .......... 7% 67.2M 11s 12250K .......... .......... .......... .......... .......... 7% 91.1M 11s 12300K .......... .......... .......... .......... .......... 7% 81.6M 11s 12350K .......... .......... .......... .......... .......... 7% 178M 11s 12400K .......... .......... .......... .......... .......... 7% 253M 11s 12450K .......... .......... .......... .......... .......... 7% 1.78M 11s 12500K .......... .......... .......... .......... .......... 7% 2.58M 11s 12550K .......... .......... .......... .......... .......... 7% 38.3M 11s 12600K .......... .......... .......... .......... .......... 7% 25.6M 11s 12650K .......... .......... .......... .......... .......... 7% 83.6M 11s 12700K .......... .......... .......... .......... .......... 7% 81.6M 11s 12750K .......... .......... .......... .......... .......... 7% 79.4M 11s 12800K .......... .......... .......... .......... .......... 7% 112M 11s 12850K .......... .......... .......... .......... .......... 7% 66.1M 11s 12900K .......... .......... .......... .......... .......... 7% 51.3M 11s 12950K .......... .......... .......... .......... .......... 7% 124M 11s 13000K .......... .......... .......... .......... .......... 7% 62.8M 11s 13050K .......... .......... .......... .......... .......... 7% 85.3M 11s 13100K .......... .......... .......... .......... .......... 7% 101M 11s 13150K .......... .......... .......... .......... .......... 7% 67.1M 11s 13200K .......... .......... .......... .......... .......... 7% 83.7M 10s 13250K .......... .......... .......... .......... .......... 7% 220M 10s 13300K .......... .......... .......... .......... .......... 8% 267M 10s 13350K .......... .......... .......... .......... .......... 8% 101M 10s 13400K .......... .......... .......... .......... .......... 8% 74.9M 10s 13450K .......... .......... .......... .......... .......... 8% 60.3M 10s 13500K .......... .......... .......... .......... .......... 8% 112M 10s 13550K .......... .......... .......... .......... .......... 8% 74.3M 10s 13600K .......... .......... .......... .......... .......... 8% 79.2M 10s 13650K .......... .......... .......... .......... .......... 8% 71.0M 10s 13700K .......... .......... .......... .......... .......... 8% 64.2M 10s 13750K .......... .......... .......... .......... .......... 8% 67.2M 10s 13800K .......... .......... .......... .......... .......... 8% 63.2M 10s 13850K .......... .......... .......... .......... .......... 8% 112M 10s 13900K .......... .......... .......... .......... .......... 8% 264M 10s 13950K .......... .......... .......... .......... .......... 8% 275M 10s 14000K .......... .......... .......... .......... .......... 8% 1.10M 10s 14050K .......... .......... .......... .......... .......... 8% 21.5M 10s 14100K .......... .......... .......... .......... .......... 8% 27.9M 10s 14150K .......... .......... .......... .......... .......... 8% 90.9M 10s 14200K .......... .......... .......... .......... .......... 8% 51.2M 10s 14250K .......... .......... .......... .......... .......... 8% 98.4M 10s 14300K .......... .......... .......... .......... .......... 8% 73.8M 10s 14350K .......... .......... .......... .......... .......... 8% 124M 10s 14400K .......... .......... .......... .......... .......... 8% 127M 10s 14450K .......... .......... .......... .......... .......... 8% 25.2M 10s 14500K .......... .......... .......... .......... .......... 8% 280M 10s 14550K .......... .......... .......... .......... .......... 8% 288M 10s 14600K .......... .......... .......... .......... .......... 8% 285M 10s 14650K .......... .......... .......... .......... .......... 8% 91.0M 10s 14700K .......... .......... .......... .......... .......... 8% 110M 10s 14750K .......... .......... .......... .......... .......... 8% 42.9M 10s 14800K .......... .......... .......... .......... .......... 8% 127M 10s 14850K .......... .......... .......... .......... .......... 8% 85.4M 10s 14900K .......... .......... .......... .......... .......... 8% 97.5M 10s 14950K .......... .......... .......... .......... .......... 9% 115M 10s 15000K .......... .......... .......... .......... .......... 9% 77.9M 10s 15050K .......... .......... .......... .......... .......... 9% 190M 10s 15100K .......... .......... .......... .......... .......... 9% 46.3M 10s 15150K .......... .......... .......... .......... .......... 9% 202M 10s 15200K .......... .......... .......... .......... .......... 9% 74.6M 10s 15250K .......... .......... .......... .......... .......... 9% 42.2M 10s 15300K .......... .......... .......... .......... .......... 9% 151M 10s 15350K .......... .......... .......... .......... .......... 9% 79.1M 10s 15400K .......... .......... .......... .......... .......... 9% 84.1M 10s 15450K .......... .......... .......... .......... .......... 9% 264M 10s 15500K .......... .......... .......... .......... .......... 9% 1.86M 10s 15550K .......... .......... .......... .......... .......... 9% 2.67M 10s 15600K .......... .......... .......... .......... .......... 9% 23.1M 10s 15650K .......... .......... .......... .......... .......... 9% 21.0M 10s 15700K .......... .......... .......... .......... .......... 9% 76.7M 10s 15750K .......... .......... .......... .......... .......... 9% 77.3M 10s 15800K .......... .......... .......... .......... .......... 9% 101M 10s 15850K .......... .......... .......... .......... .......... 9% 104M 10s 15900K .......... .......... .......... .......... .......... 9% 70.2M 10s 15950K .......... .......... .......... .......... .......... 9% 32.3M 10s *** WARNING: skipped 203991 bytes of output *** 150200K .......... .......... .......... .......... .......... 90% 28.0M 1s 150250K .......... .......... .......... .......... .......... 90% 9.04M 1s 150300K .......... .......... .......... .......... .......... 90% 18.5M 1s 150350K .......... .......... .......... .......... .......... 90% 22.4M 1s 150400K .......... .......... .......... .......... .......... 90% 13.4M 1s 150450K .......... .......... .......... .......... .......... 90% 109M 1s 150500K .......... .......... .......... .......... .......... 90% 46.4M 1s 150550K .......... .......... .......... .......... .......... 90% 29.5M 1s 150600K .......... .......... .......... .......... .......... 90% 36.4M 1s 150650K .......... .......... .......... .......... .......... 90% 23.9M 1s 150700K .......... .......... .......... .......... .......... 90% 124M 1s 150750K .......... .......... .......... .......... .......... 90% 36.9M 1s 150800K .......... .......... .......... .......... .......... 90% 22.2M 1s 150850K .......... .......... .......... .......... .......... 90% 81.6M 1s 150900K .......... .......... .......... .......... .......... 90% 93.0M 1s 150950K .......... .......... .......... .......... .......... 90% 19.6M 1s 151000K .......... .......... .......... .......... .......... 90% 89.8M 1s 151050K .......... .......... .......... .......... .......... 90% 111M 1s 151100K .......... .......... .......... .......... .......... 90% 119M 1s 151150K .......... .......... .......... .......... .......... 90% 105M 1s 151200K .......... .......... .......... .......... .......... 90% 113M 1s 151250K .......... .......... .......... .......... .......... 90% 10.3M 1s 151300K .......... .......... .......... .......... .......... 90% 30.2M 1s 151350K .......... .......... .......... .......... .......... 90% 112M 1s 151400K .......... .......... .......... .......... .......... 90% 98.2M 1s 151450K .......... .......... .......... .......... .......... 90% 29.6M 1s 151500K .......... .......... .......... .......... .......... 91% 3.88M 1s 151550K .......... .......... .......... .......... .......... 91% 22.3M 1s 151600K .......... .......... .......... .......... .......... 91% 81.7M 1s 151650K .......... .......... .......... .......... .......... 91% 10.8M 1s 151700K .......... .......... .......... .......... .......... 91% 9.90M 1s 151750K .......... .......... .......... .......... .......... 91% 9.69M 1s 151800K .......... .......... .......... .......... .......... 91% 58.4M 1s 151850K .......... .......... .......... .......... .......... 91% 20.6M 1s 151900K .......... .......... .......... .......... .......... 91% 22.1M 1s 151950K .......... .......... .......... .......... .......... 91% 13.3M 1s 152000K .......... .......... .......... .......... .......... 91% 45.4M 1s 152050K .......... .......... .......... .......... .......... 91% 25.3M 1s 152100K .......... .......... .......... .......... .......... 91% 109M 1s 152150K .......... .......... .......... .......... .......... 91% 30.7M 1s 152200K .......... .......... .......... .......... .......... 91% 32.8M 1s 152250K .......... .......... .......... .......... .......... 91% 37.6M 1s 152300K .......... .......... .......... .......... .......... 91% 120M 1s 152350K .......... .......... .......... .......... .......... 91% 19.1M 1s 152400K .......... .......... .......... .......... .......... 91% 120M 1s 152450K .......... .......... .......... .......... .......... 91% 20.2M 1s 152500K .......... .......... .......... .......... .......... 91% 59.0M 1s 152550K .......... .......... .......... .......... .......... 91% 110M 1s 152600K .......... .......... .......... .......... .......... 91% 126M 1s 152650K .......... .......... .......... .......... .......... 91% 101M 1s 152700K .......... .......... .......... .......... .......... 91% 121M 1s 152750K .......... .......... .......... .......... .......... 91% 13.3M 1s 152800K .......... .......... .......... .......... .......... 91% 38.7M 1s 152850K .......... .......... .......... .......... .......... 91% 25.2M 1s 152900K .......... .......... .......... .......... .......... 91% 73.5M 1s 152950K .......... .......... .......... .......... .......... 91% 119M 1s 153000K .......... .......... .......... .......... .......... 91% 34.6M 1s 153050K .......... .......... .......... .......... .......... 91% 4.03M 1s 153100K .......... .......... .......... .......... .......... 91% 18.1M 1s 153150K .......... .......... .......... .......... .......... 92% 11.5M 1s 153200K .......... .......... .......... .......... .......... 92% 9.59M 1s 153250K .......... .......... .......... .......... .......... 92% 103M 1s 153300K .......... .......... .......... .......... .......... 92% 8.76M 1s 153350K .......... .......... .......... .......... .......... 92% 20.5M 1s 153400K .......... .......... .......... .......... .......... 92% 26.9M 1s 153450K .......... .......... .......... .......... .......... 92% 12.3M 1s 153500K .......... .......... .......... .......... .......... 92% 133M 1s 153550K .......... .......... .......... .......... .......... 92% 37.2M 1s 153600K .......... .......... .......... .......... .......... 92% 28.4M 1s 153650K .......... .......... .......... .......... .......... 92% 82.1M 1s 153700K .......... .......... .......... .......... .......... 92% 41.2M 1s 153750K .......... .......... .......... .......... .......... 92% 26.6M 1s 153800K .......... .......... .......... .......... .......... 92% 38.3M 1s 153850K .......... .......... .......... .......... .......... 92% 19.5M 1s 153900K .......... .......... .......... .......... .......... 92% 122M 1s 153950K .......... .......... .......... .......... .......... 92% 107M 1s 154000K .......... .......... .......... .......... .......... 92% 21.6M 1s 154050K .......... .......... .......... .......... .......... 92% 59.7M 1s 154100K .......... .......... .......... .......... .......... 92% 81.7M 1s 154150K .......... .......... .......... .......... .......... 92% 97.9M 1s 154200K .......... .......... .......... .......... .......... 92% 94.2M 1s 154250K .......... .......... .......... .......... .......... 92% 75.3M 1s 154300K .......... .......... .......... .......... .......... 92% 11.9M 1s 154350K .......... .......... .......... .......... .......... 92% 22.7M 1s 154400K .......... .......... .......... .......... .......... 92% 189M 1s 154450K .......... .......... .......... .......... .......... 92% 62.8M 1s 154500K .......... .......... .......... .......... .......... 92% 38.2M 1s 154550K .......... .......... .......... .......... .......... 92% 4.05M 1s 154600K .......... .......... .......... .......... .......... 92% 20.9M 1s 154650K .......... .......... .......... .......... .......... 92% 67.3M 1s 154700K .......... .......... .......... .......... .......... 92% 11.4M 1s 154750K .......... .......... .......... .......... .......... 92% 9.93M 1s 154800K .......... .......... .......... .......... .......... 93% 8.16M 1s 154850K .......... .......... .......... .......... .......... 93% 22.1M 1s 154900K .......... .......... .......... .......... .......... 93% 115M 1s 154950K .......... .......... .......... .......... .......... 93% 28.2M 1s 155000K .......... .......... .......... .......... .......... 93% 12.6M 1s 155050K .......... .......... .......... .......... .......... 93% 28.0M 1s 155100K .......... .......... .......... .......... .......... 93% 108M 1s 155150K .......... .......... .......... .......... .......... 93% 28.9M 1s 155200K .......... .......... .......... .......... .......... 93% 43.4M 0s 155250K .......... .......... .......... .......... .......... 93% 25.7M 0s 155300K .......... .......... .......... .......... .......... 93% 55.1M 0s 155350K .......... .......... .......... .......... .......... 93% 96.4M 0s 155400K .......... .......... .......... .......... .......... 93% 17.7M 0s 155450K .......... .......... .......... .......... .......... 93% 114M 0s 155500K .......... .......... .......... .......... .......... 93% 23.0M 0s 155550K .......... .......... .......... .......... .......... 93% 53.7M 0s 155600K .......... .......... .......... .......... .......... 93% 107M 0s 155650K .......... .......... .......... .......... .......... 93% 113M 0s 155700K .......... .......... .......... .......... .......... 93% 80.5M 0s 155750K .......... .......... .......... .......... .......... 93% 47.0M 0s 155800K .......... .......... .......... .......... .......... 93% 130M 0s 155850K .......... .......... .......... .......... .......... 93% 12.7M 0s 155900K .......... .......... .......... .......... .......... 93% 22.7M 0s 155950K .......... .......... .......... .......... .......... 93% 93.2M 0s 156000K .......... .......... .......... .......... .......... 93% 35.4M 0s 156050K .......... .......... .......... .......... .......... 93% 118M 0s 156100K .......... .......... .......... .......... .......... 93% 4.06M 0s 156150K .......... .......... .......... .......... .......... 93% 22.1M 0s 156200K .......... .......... .......... .......... .......... 93% 11.5M 0s 156250K .......... .......... .......... .......... .......... 93% 9.82M 0s 156300K .......... .......... .......... .......... .......... 93% 50.6M 0s 156350K .......... .......... .......... .......... .......... 93% 8.51M 0s 156400K .......... .......... .......... .......... .......... 93% 19.7M 0s 156450K .......... .......... .......... .......... .......... 93% 34.3M 0s 156500K .......... .......... .......... .......... .......... 94% 101M 0s 156550K .......... .......... .......... .......... .......... 94% 12.7M 0s 156600K .......... .......... .......... .......... .......... 94% 15.7M 0s 156650K .......... .......... .......... .......... .......... 94% 88.1M 0s 156700K .......... .......... .......... .......... .......... 94% 108M 0s 156750K .......... .......... .......... .......... .......... 94% 41.1M 0s 156800K .......... .......... .......... .......... .......... 94% 26.3M 0s 156850K .......... .......... .......... .......... .......... 94% 53.3M 0s 156900K .......... .......... .......... .......... .......... 94% 15.3M 0s 156950K .......... .......... .......... .......... .......... 94% 177M 0s 157000K .......... .......... .......... .......... .......... 94% 136M 0s 157050K .......... .......... .......... .......... .......... 94% 24.9M 0s 157100K .......... .......... .......... .......... .......... 94% 47.5M 0s 157150K .......... .......... .......... .......... .......... 94% 124M 0s 157200K .......... .......... .......... .......... .......... 94% 122M 0s 157250K .......... .......... .......... .......... .......... 94% 97.5M 0s 157300K .......... .......... .......... .......... .......... 94% 49.0M 0s 157350K .......... .......... .......... .......... .......... 94% 13.0M 0s 157400K .......... .......... .......... .......... .......... 94% 21.6M 0s 157450K .......... .......... .......... .......... .......... 94% 107M 0s 157500K .......... .......... .......... .......... .......... 94% 106M 0s 157550K .......... .......... .......... .......... .......... 94% 36.3M 0s 157600K .......... .......... .......... .......... .......... 94% 4.11M 0s 157650K .......... .......... .......... .......... .......... 94% 19.1M 0s 157700K .......... .......... .......... .......... .......... 94% 91.8M 0s 157750K .......... .......... .......... .......... .......... 94% 11.7M 0s 157800K .......... .......... .......... .......... .......... 94% 9.41M 0s 157850K .......... .......... .......... .......... .......... 94% 11.9M 0s 157900K .......... .......... .......... .......... .......... 94% 23.2M 0s 157950K .......... .......... .......... .......... .......... 94% 19.1M 0s 158000K .......... .......... .......... .......... .......... 94% 31.9M 0s 158050K .......... .......... .......... .......... .......... 94% 13.4M 0s 158100K .......... .......... .......... .......... .......... 94% 15.4M 0s 158150K .......... .......... .......... .......... .......... 95% 114M 0s 158200K .......... .......... .......... .......... .......... 95% 95.9M 0s 158250K .......... .......... .......... .......... .......... 95% 39.2M 0s 158300K .......... .......... .......... .......... .......... 95% 24.9M 0s 158350K .......... .......... .......... .......... .......... 95% 110M 0s 158400K .......... .......... .......... .......... .......... 95% 52.4M 0s 158450K .......... .......... .......... .......... .......... 95% 16.3M 0s 158500K .......... .......... .......... .......... .......... 95% 93.2M 0s 158550K .......... .......... .......... .......... .......... 95% 29.1M 0s 158600K .......... .......... .......... .......... .......... 95% 96.1M 0s 158650K .......... .......... .......... .......... .......... 95% 49.6M 0s 158700K .......... .......... .......... .......... .......... 95% 117M 0s 158750K .......... .......... .......... .......... .......... 95% 129M 0s 158800K .......... .......... .......... .......... .......... 95% 42.0M 0s 158850K .......... .......... .......... .......... .......... 95% 111M 0s 158900K .......... .......... .......... .......... .......... 95% 13.8M 0s 158950K .......... .......... .......... .......... .......... 95% 19.6M 0s 159000K .......... .......... .......... .......... .......... 95% 108M 0s 159050K .......... .......... .......... .......... .......... 95% 36.5M 0s 159100K .......... .......... .......... .......... .......... 95% 94.6M 0s 159150K .......... .......... .......... .......... .......... 95% 4.21M 0s 159200K .......... .......... .......... .......... .......... 95% 18.0M 0s 159250K .......... .......... .......... .......... .......... 95% 13.5M 0s 159300K .......... .......... .......... .......... .......... 95% 51.7M 0s 159350K .......... .......... .......... .......... .......... 95% 8.73M 0s 159400K .......... .......... .......... .......... .......... 95% 9.39M 0s 159450K .......... .......... .......... .......... .......... 95% 17.2M 0s 159500K .......... .......... .......... .......... .......... 95% 38.5M 0s 159550K .......... .......... .......... .......... .......... 95% 72.0M 0s 159600K .......... .......... .......... .......... .......... 95% 13.6M 0s 159650K .......... .......... .......... .......... .......... 95% 14.5M 0s 159700K .......... .......... .......... .......... .......... 95% 101M 0s 159750K .......... .......... .......... .......... .......... 95% 34.3M 0s 159800K .......... .......... .......... .......... .......... 96% 111M 0s 159850K .......... .......... .......... .......... .......... 96% 27.0M 0s 159900K .......... .......... .......... .......... .......... 96% 90.0M 0s 159950K .......... .......... .......... .......... .......... 96% 14.8M 0s 160000K .......... .......... .......... .......... .......... 96% 87.0M 0s 160050K .......... .......... .......... .......... .......... 96% 160M 0s 160100K .......... .......... .......... .......... .......... 96% 26.0M 0s 160150K .......... .......... .......... .......... .......... 96% 55.2M 0s 160200K .......... .......... .......... .......... .......... 96% 101M 0s 160250K .......... .......... .......... .......... .......... 96% 91.6M 0s 160300K .......... .......... .......... .......... .......... 96% 179M 0s 160350K .......... .......... .......... .......... .......... 96% 34.1M 0s 160400K .......... .......... .......... .......... .......... 96% 14.7M 0s 160450K .......... .......... .......... .......... .......... 96% 19.9M 0s 160500K .......... .......... .......... .......... .......... 96% 115M 0s 160550K .......... .......... .......... .......... .......... 96% 99.7M 0s 160600K .......... .......... .......... .......... .......... 96% 37.3M 0s 160650K .......... .......... .......... .......... .......... 96% 5.11M 0s 160700K .......... .......... .......... .......... .......... 96% 20.3M 0s 160750K .......... .......... .......... .......... .......... 96% 9.46M 0s 160800K .......... .......... .......... .......... .......... 96% 31.9M 0s 160850K .......... .......... .......... .......... .......... 96% 8.44M 0s 160900K .......... .......... .......... .......... .......... 96% 13.8M 0s 160950K .......... .......... .......... .......... .......... 96% 19.6M 0s 161000K .......... .......... .......... .......... .......... 96% 18.5M 0s 161050K .......... .......... .......... .......... .......... 96% 36.5M 0s 161100K .......... .......... .......... .......... .......... 96% 13.2M 0s 161150K .......... .......... .......... .......... .......... 96% 14.0M 0s 161200K .......... .......... .......... .......... .......... 96% 54.4M 0s 161250K .......... .......... .......... .......... .......... 96% 56.5M 0s 161300K .......... .......... .......... .......... .......... 96% 29.5M 0s 161350K .......... .......... .......... .......... .......... 96% 42.5M 0s 161400K .......... .......... .......... .......... .......... 96% 55.1M 0s 161450K .......... .......... .......... .......... .......... 96% 49.2M 0s 161500K .......... .......... .......... .......... .......... 97% 21.8M 0s 161550K .......... .......... .......... .......... .......... 97% 50.7M 0s 161600K .......... .......... .......... .......... .......... 97% 44.0M 0s 161650K .......... .......... .......... .......... .......... 97% 62.1M 0s 161700K .......... .......... .......... .......... .......... 97% 55.2M 0s 161750K .......... .......... .......... .......... .......... 97% 47.2M 0s 161800K .......... .......... .......... .......... .......... 97% 62.2M 0s 161850K .......... .......... .......... .......... .......... 97% 51.0M 0s 161900K .......... .......... .......... .......... .......... 97% 51.4M 0s 161950K .......... .......... .......... .......... .......... 97% 27.7M 0s 162000K .......... .......... .......... .......... .......... 97% 18.4M 0s 162050K .......... .......... .......... .......... .......... 97% 47.0M 0s 162100K .......... .......... .......... .......... .......... 97% 55.8M 0s 162150K .......... .......... .......... .......... .......... 97% 47.8M 0s 162200K .......... .......... .......... .......... .......... 97% 4.68M 0s 162250K .......... .......... .......... .......... .......... 97% 9.38M 0s 162300K .......... .......... .......... .......... .......... 97% 34.7M 0s 162350K .......... .......... .......... .......... .......... 97% 75.1M 0s 162400K .......... .......... .......... .......... .......... 97% 8.42M 0s 162450K .......... .......... .......... .......... .......... 97% 9.05M 0s 162500K .......... .......... .......... .......... .......... 97% 15.4M 0s 162550K .......... .......... .......... .......... .......... 97% 52.0M 0s 162600K .......... .......... .......... .......... .......... 97% 163M 0s 162650K .......... .......... .......... .......... .......... 97% 12.4M 0s 162700K .......... .......... .......... .......... .......... 97% 12.5M 0s 162750K .......... .......... .......... .......... .......... 97% 42.1M 0s 162800K .......... .......... .......... .......... .......... 97% 128M 0s 162850K .......... .......... .......... .......... .......... 97% 29.8M 0s 162900K .......... .......... .......... .......... .......... 97% 49.7M 0s 162950K .......... .......... .......... .......... .......... 97% 30.0M 0s 163000K .......... .......... .......... .......... .......... 97% 25.2M 0s 163050K .......... .......... .......... .......... .......... 97% 79.8M 0s 163100K .......... .......... .......... .......... .......... 97% 42.2M 0s 163150K .......... .......... .......... .......... .......... 98% 35.0M 0s 163200K .......... .......... .......... .......... .......... 98% 60.6M 0s 163250K .......... .......... .......... .......... .......... 98% 116M 0s 163300K .......... .......... .......... .......... .......... 98% 52.9M 0s 163350K .......... .......... .......... .......... .......... 98% 40.2M 0s 163400K .......... .......... .......... .......... .......... 98% 42.7M 0s 163450K .......... .......... .......... .......... .......... 98% 46.7M 0s 163500K .......... .......... .......... .......... .......... 98% 40.4M 0s 163550K .......... .......... .......... .......... .......... 98% 18.7M 0s 163600K .......... .......... .......... .......... .......... 98% 34.8M 0s 163650K .......... .......... .......... .......... .......... 98% 51.2M 0s 163700K .......... .......... .......... .......... .......... 98% 6.20M 0s 163750K .......... .......... .......... .......... .......... 98% 18.0M 0s 163800K .......... .......... .......... .......... .......... 98% 8.82M 0s 163850K .......... .......... .......... .......... .......... 98% 30.8M 0s 163900K .......... .......... .......... .......... .......... 98% 14.1M 0s 163950K .......... .......... .......... .......... .......... 98% 9.44M 0s 164000K .......... .......... .......... .......... .......... 98% 15.6M 0s 164050K .......... .......... .......... .......... .......... 98% 15.7M 0s 164100K .......... .......... .......... .......... .......... 98% 65.5M 0s 164150K .......... .......... .......... .......... .......... 98% 12.7M 0s 164200K .......... .......... .......... .......... .......... 98% 118M 0s 164250K .......... .......... .......... .......... .......... 98% 12.3M 0s 164300K .......... .......... .......... .......... .......... 98% 40.9M 0s 164350K .......... .......... .......... .......... .......... 98% 30.6M 0s 164400K .......... .......... .......... .......... .......... 98% 65.4M 0s 164450K .......... .......... .......... .......... .......... 98% 99.2M 0s 164500K .......... .......... .......... .......... .......... 98% 23.8M 0s 164550K .......... .......... .......... .......... .......... 98% 25.5M 0s 164600K .......... .......... .......... .......... .......... 98% 37.6M 0s 164650K .......... .......... .......... .......... .......... 98% 36.1M 0s 164700K .......... .......... .......... .......... .......... 98% 114M 0s 164750K .......... .......... .......... .......... .......... 98% 68.6M 0s 164800K .......... .......... .......... .......... .......... 99% 57.2M 0s 164850K .......... .......... .......... .......... .......... 99% 31.2M 0s 164900K .......... .......... .......... .......... .......... 99% 90.8M 0s 164950K .......... .......... .......... .......... .......... 99% 61.4M 0s 165000K .......... .......... .......... .......... .......... 99% 28.4M 0s 165050K .......... .......... .......... .......... .......... 99% 18.7M 0s 165100K .......... .......... .......... .......... .......... 99% 29.4M 0s 165150K .......... .......... .......... .......... .......... 99% 118M 0s 165200K .......... .......... .......... .......... .......... 99% 65.8M 0s 165250K .......... .......... .......... .......... .......... 99% 5.33M 0s 165300K .......... .......... .......... .......... .......... 99% 7.55M 0s 165350K .......... .......... .......... .......... .......... 99% 41.4M 0s 165400K .......... .......... .......... .......... .......... 99% 67.9M 0s 165450K .......... .......... .......... .......... .......... 99% 8.83M 0s 165500K .......... .......... .......... .......... .......... 99% 8.29M 0s 165550K .......... .......... .......... .......... .......... 99% 15.0M 0s 165600K .......... .......... .......... .......... .......... 99% 128M 0s 165650K .......... .......... .......... .......... .......... 99% 74.4M 0s 165700K .......... .......... .......... .......... .......... 99% 12.6M 0s 165750K .......... .......... .......... .......... .......... 99% 12.7M 0s 165800K .......... .......... .......... .......... .......... 99% 30.9M 0s 165850K .......... .......... .......... .......... .......... 99% 130M 0s 165900K .......... .......... .......... .......... .......... 99% 33.8M 0s 165950K .......... .......... .......... .......... .......... 99% 65.4M 0s 166000K .......... .......... .......... .......... .......... 99% 24.3M 0s 166050K .......... .......... .......... .......... .......... 99% 28.8M 0s 166100K .......... .......... .......... .......... .......... 99% 78.1M 0s 166150K .......... .......... .......... .......... .......... 99% 32.7M 0s 166200K .......... .......... .......... .......... .......... 99% 35.1M 0s 166250K .......... .......... .......... .......... .......... 99% 82.9M 0s 166300K .......... .......... .......... .......... .......... 99% 107M 0s 166350K .......... .......... .......... .......... .......... 99% 60.4M 0s 166400K .......... .......... .......... .......... .......... 99% 25.1M 0s 166450K .......... .......... .......... .......... .......... 99% 71.7M 0s 166500K .. 100% 3858G=7.3s 2019-06-17 14:01:33 (22.1 MB/s) - ‘cifar-10-python.tar.gz’ saved [170498071/170498071] 
%sh
tar zxvf cifar-10-python.tar.gz
cifar-10-batches-py/ cifar-10-batches-py/data_batch_4 cifar-10-batches-py/readme.html cifar-10-batches-py/test_batch cifar-10-batches-py/data_batch_3 cifar-10-batches-py/batches.meta cifar-10-batches-py/data_batch_2 cifar-10-batches-py/data_batch_5 cifar-10-batches-py/data_batch_1 
%sh ls -la /dbfs/mnt/cifar/batches
ls: cannot access '/dbfs/mnt/cifar/batches': Bad address

Recall the classes are: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck

Here is the code to unpickle the batches.

Loaded in this way, each of the batch files contains a dictionary with the following elements:

  • data - a 10000x3072 numpy array of uint8s. Each row of the array stores a 32x32 colour image. The first 1024 entries contain the red channel values, the next 1024 the green, and the final 1024 the blue. The image is stored in row-major order, so that the first 32 entries of the array are the red channel values of the first row of the image.
  • labels - a list of 10000 numbers in the range 0-9. The number at index i indicates the label of the ith image in the array data.
def unpickle(file):
    import pickle
    with open(file, 'rb') as fo:
        dict = pickle.load(fo)# for Python 3, add the following param: encoding='bytes'
    return dict

dir = '/dbfs/mnt/cifar/batches/'

batches = [unpickle(dir + 'data_batch_' + str(1+n)) for n in range(5)]

Now we need to reshape the data batches and concatenate the training batches into one big tensor.

import numpy as np

def decode(xy):
  x_train = np.reshape(xy[b'data'], (10000, 3, 32, 32)).transpose(0, 2, 3, 1)
  y_train = np.reshape(xy[b'labels'], (10000, 1))
  return (x_train, y_train)

decoded = [decode(data) for data in batches]

x_train = np.concatenate([data[0] for data in decoded])
y_train = np.concatenate([data[1] for data in decoded])

(x_test, y_test) = decode(unpickle(dir + 'test_batch'))

print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
('x_train shape:', (50000, 32, 32, 3)) (50000, 'train samples') (10000, 'test samples')

Let's visualize some of the images:

import matplotlib.pyplot as plt

fig = plt.figure()
for i in range(36):
  fig.add_subplot(6, 6, i+1)
  plt.imshow(x_train[i])

display(fig)

Recall that we are getting a categorical output via softmax across 10 neurons, corresponding to the output categories.

So we want to reshape our target values (training labels) to be 1-hot encoded, and Keras can calculate categorical crossentropy between its output layer and the target:

import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Conv2D, MaxPooling2D

num_classes = 10

# Convert class vectors to binary class matrices.
y_train_1hot = keras.utils.to_categorical(y_train, num_classes)
y_test_1hot = keras.utils.to_categorical(y_test, num_classes)
Using TensorFlow backend.

Here's a simple convolutional net to get you started. It will get you to over 57% accuracy in 5 epochs.

As inspiration, with a suitable network and parameters, it's possible to get over 99% test accuracy, although you won't have time to get there in today's session on this hardware.

note: if your network is not learning anything at all -- meaning regardless of settings, you're seeing a loss that doesn't decrease and a validation accuracy that is 10% (i.e., random chance) -- then restart your cluster

model = Sequential()

model.add(Conv2D(32, (3, 3), padding='same', input_shape=x_train.shape[1:]))
model.add(Activation('relu'))

model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dense(num_classes))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer="adam", metrics=['accuracy'])

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255

history = model.fit(x_train, y_train_1hot,
              batch_size=64,
              epochs=5,
              validation_data=(x_test, y_test_1hot),
              verbose=2)
Train on 50000 samples, validate on 10000 samples Epoch 1/5 42s - loss: 1.6253 - acc: 0.4202 - val_loss: 1.3802 - val_acc: 0.5053 Epoch 2/5 42s - loss: 1.2808 - acc: 0.5467 - val_loss: 1.2657 - val_acc: 0.5493 Epoch 3/5 42s - loss: 1.1541 - acc: 0.5940 - val_loss: 1.2050 - val_acc: 0.5742 Epoch 4/5 42s - loss: 1.0697 - acc: 0.6235 - val_loss: 1.2062 - val_acc: 0.5742 Epoch 5/5 42s - loss: 0.9984 - acc: 0.6496 - val_loss: 1.1442 - val_acc: 0.6008

In this session, you probably won't have time to run each experiment for too many epochs ... but you can use this code to plot the training and validation losses:

fig, ax = plt.subplots()
fig.set_size_inches((5,5))
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
display(fig)