058_DLbyABr_04c-CIFAR-10(Python)

Import Notebook

ScaDaMaLe Course site and book

This is a 2019-2021 augmentation and update of Adam Breindel's initial notebooks.

Thanks to Christian von Koch and William Anzén for their contributions towards making these materials Spark 3.0.1 and Python 3+ compliant.

CIFAR 10

Details at: https://www.cs.toronto.edu/~kriz/cifar.html

Summary (taken from that page):

The CIFAR-10 and CIFAR-100 are labeled subsets of the 80 million tiny images dataset. They were collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.

The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.

First, we'll mount the S3 bucket where I'm hosting the data:

# you may have to host the data yourself! - this should not work unless you can descramble
ACCESS="...SPORAA...KIAJZEH...PW46CWPUWUN...QPODO" # scrambled up
SECRET="...P7d7Sp7r1...Q9DuUvV...QAy1D+hjC...NxakJF+PXrAb...MXD1tZwBpGyN...1Ns5r5n1" # scrambled up
BUCKET = "cool-data"
MOUNT = "/mnt/cifar"
 
try:
  dbutils.fs.mount("s3a://"+ ACCESS + ":" + SECRET + "@" + BUCKET, MOUNT)
except:
  print("Error mounting ... possibly already mounted")
Error mounting ... possibly already mounted

This is in DBFS, which is available (via FUSE) at /dbfs ...

So the CIFAR data can be listed through following regular Linux shell command:

%sh
wget http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
--2021-01-18 14:38:29-- http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz Resolving www.cs.toronto.edu (www.cs.toronto.edu)... 128.100.3.30 Connecting to www.cs.toronto.edu (www.cs.toronto.edu)|128.100.3.30|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 170498071 (163M) [application/x-gzip] Saving to: ‘cifar-10-python.tar.gz’ 0K .......... .......... .......... .......... .......... 0% 340K 8m10s 50K .......... .......... .......... .......... .......... 0% 712K 6m2s 100K .......... .......... .......... .......... .......... 0% 677K 5m23s 150K .......... .......... .......... .......... .......... 0% 9.41M 4m6s 200K .......... .......... .......... .......... .......... 0% 727K 4m3s 250K .......... .......... .......... .......... .......... 0% 123M 3m23s 300K .......... .......... .......... .......... .......... 0% 33.2M 2m54s 350K .......... .......... .......... .......... .......... 0% 25.4M 2m33s 400K .......... .......... .......... .......... .......... 0% 723K 2m42s 450K .......... .......... .......... .......... .......... 0% 12.0M 2m27s 500K .......... .......... .......... .......... .......... 0% 9.97M 2m15s 550K .......... .......... .......... .......... .......... 0% 10.1M 2m5s 600K .......... .......... .......... .......... .......... 0% 13.8M 1m56s 650K .......... .......... .......... .......... .......... 0% 10.1M 1m49s 700K .......... .......... .......... .......... .......... 0% 1.01M 1m52s 750K .......... .......... .......... .......... .......... 0% 13.4M 1m46s 800K .......... .......... .......... .......... .......... 0% 13.0M 1m41s 850K .......... .......... .......... .......... .......... 0% 101M 95s 900K .......... .......... .......... .......... .......... 0% 25.4M 90s 950K .......... .......... .......... .......... .......... 0% 95.4M 86s 1000K .......... .......... .......... .......... .......... 0% 178M 82s 1050K .......... .......... .......... .......... .......... 0% 168M 78s 1100K .......... .......... .......... .......... .......... 0% 141M 75s 1150K .......... .......... .......... .......... .......... 0% 11.4M 72s 1200K .......... .......... .......... .......... .......... 0% 52.0M 69s 1250K .......... .......... .......... .......... .......... 0% 17.1M 67s 1300K .......... .......... .......... .......... .......... 0% 45.0M 65s 1350K .......... .......... .......... .......... .......... 0% 1.05M 68s 1400K .......... .......... .......... .......... .......... 0% 28.4M 66s 1450K .......... .......... .......... .......... .......... 0% 18.0M 64s 1500K .......... .......... .......... .......... .......... 0% 18.1M 62s 1550K .......... .......... .......... .......... .......... 0% 76.4M 60s 1600K .......... .......... .......... .......... .......... 0% 11.6M 59s 1650K .......... .......... .......... .......... .......... 1% 32.7M 57s 1700K .......... .......... .......... .......... .......... 1% 64.9M 56s 1750K .......... .......... .......... .......... .......... 1% 34.7M 54s 1800K .......... .......... .......... .......... .......... 1% 30.9M 53s 1850K .......... .......... .......... .......... .......... 1% 30.2M 51s 1900K .......... .......... .......... .......... .......... 1% 62.4M 50s 1950K .......... .......... .......... .......... .......... 1% 35.9M 49s 2000K .......... .......... .......... .......... .......... 1% 28.7M 48s 2050K .......... .......... .......... .......... .......... 1% 23.6M 47s 2100K .......... .......... .......... .......... .......... 1% 146M 46s 2150K .......... .......... .......... .......... .......... 1% 25.9M 45s 2200K .......... .......... .......... .......... .......... 1% 15.4M 44s 2250K .......... .......... .......... .......... .......... 1% 48.2M 43s 2300K .......... .......... .......... .......... .......... 1% 21.6M 43s 2350K .......... .......... .......... .......... .......... 1% 25.2M 42s 2400K .......... .......... .......... .......... .......... 1% 34.2M 41s 2450K .......... .......... .......... .......... .......... 1% 22.4M 40s 2500K .......... .......... .......... .......... .......... 1% 29.8M 40s 2550K .......... .......... .......... .......... .......... 1% 58.4M 39s 2600K .......... .......... .......... .......... .......... 1% 21.6M 38s 2650K .......... .......... .......... .......... .......... 1% 21.9M 38s 2700K .......... .......... .......... .......... .......... 1% 32.0M 37s 2750K .......... .......... .......... .......... .......... 1% 1.93M 38s 2800K .......... .......... .......... .......... .......... 1% 50.3M 37s 2850K .......... .......... .......... .......... .......... 1% 21.5M 37s 2900K .......... .......... .......... .......... .......... 1% 48.3M 36s 2950K .......... .......... .......... .......... .......... 1% 42.8M 36s 3000K .......... .......... .......... .......... .......... 1% 27.7M 35s 3050K .......... .......... .......... .......... .......... 1% 123M 35s 3100K .......... .......... .......... .......... .......... 1% 13.8M 34s 3150K .......... .......... .......... .......... .......... 1% 22.5M 34s 3200K .......... .......... .......... .......... .......... 1% 189M 33s 3250K .......... .......... .......... .......... .......... 1% 46.0M 33s 3300K .......... .......... .......... .......... .......... 2% 22.5M 32s 3350K .......... .......... .......... .......... .......... 2% 30.3M 32s 3400K .......... .......... .......... .......... .......... 2% 41.9M 32s 3450K .......... .......... .......... .......... .......... 2% 36.1M 31s 3500K .......... .......... .......... .......... .......... 2% 103M 31s 3550K .......... .......... .......... .......... .......... 2% 13.1M 30s 3600K .......... .......... .......... .......... .......... 2% 44.0M 30s 3650K .......... .......... .......... .......... .......... 2% 43.4M 30s 3700K .......... .......... .......... .......... .......... 2% 20.4M 29s 3750K .......... .......... .......... .......... .......... 2% 40.8M 29s 3800K .......... .......... .......... .......... .......... 2% 34.0M 29s 3850K .......... .......... .......... .......... .......... 2% 21.9M 28s 3900K .......... .......... .......... .......... .......... 2% 76.5M 28s 3950K .......... .......... .......... .......... .......... 2% 24.7M 28s 4000K .......... .......... .......... .......... .......... 2% 47.5M 28s 4050K .......... .......... .......... .......... .......... 2% 20.4M 27s 4100K .......... .......... .......... .......... .......... 2% 161M 27s 4150K .......... .......... .......... .......... .......... 2% 18.5M 27s 4200K .......... .......... .......... .......... .......... 2% 105M 26s 4250K .......... .......... .......... .......... .......... 2% 2.18M 27s 4300K .......... .......... .......... .......... .......... 2% 8.37M 27s 4350K .......... .......... .......... .......... .......... 2% 155M 27s 4400K .......... .......... .......... .......... .......... 2% 27.0M 26s 4450K .......... .......... .......... .......... .......... 2% 32.7M 26s 4500K .......... .......... .......... .......... .......... 2% 68.8M 26s 4550K .......... .......... .......... .......... .......... 2% 14.5M 26s 4600K .......... .......... .......... .......... .......... 2% 120M 25s 4650K .......... .......... .......... .......... .......... 2% 22.6M 25s 4700K .......... .......... .......... .......... .......... 2% 30.9M 25s 4750K .......... .......... .......... .......... .......... 2% 27.7M 25s 4800K .......... .......... .......... .......... .......... 2% 73.8M 24s 4850K .......... .......... .......... .......... .......... 2% 24.3M 24s 4900K .......... .......... .......... .......... .......... 2% 99.2M 24s 4950K .......... .......... .......... .......... .......... 3% 28.0M 24s 5000K .......... .......... .......... .......... .......... 3% 49.4M 24s 5050K .......... .......... .......... .......... .......... 3% 114M 23s 5100K .......... .......... .......... .......... .......... 3% 22.3M 23s 5150K .......... .......... .......... .......... .......... 3% 27.0M 23s 5200K .......... .......... .......... .......... .......... 3% 13.5M 23s 5250K .......... .......... .......... .......... .......... 3% 81.3M 23s 5300K .......... .......... .......... .......... .......... 3% 30.1M 23s 5350K .......... .......... .......... .......... .......... 3% 52.0M 22s 5400K .......... .......... .......... .......... .......... 3% 23.5M 22s 5450K .......... .......... .......... .......... .......... 3% 20.9M 22s 5500K .......... .......... .......... .......... .......... 3% 117M 22s 5550K .......... .......... .......... .......... .......... 3% 22.7M 22s 5600K .......... .......... .......... .......... .......... 3% 20.8M 22s 5650K .......... .......... .......... .......... .......... 3% 25.7M 22s 5700K .......... .......... .......... .......... .......... 3% 31.2M 21s 5750K .......... .......... .......... .......... .......... 3% 103M 21s 5800K .......... .......... .......... .......... .......... 3% 1.88M 22s 5850K .......... .......... .......... .......... .......... 3% 21.6M 22s 5900K .......... .......... .......... .......... .......... 3% 27.9M 21s 5950K .......... .......... .......... .......... .......... 3% 20.9M 21s 6000K .......... .......... .......... .......... .......... 3% 94.0M 21s 6050K .......... .......... .......... .......... .......... 3% 31.1M 21s 6100K .......... .......... .......... .......... .......... 3% 18.2M 21s 6150K .......... .......... .......... .......... .......... 3% 20.6M 21s 6200K .......... .......... .......... .......... .......... 3% 34.0M 21s 6250K .......... .......... .......... .......... .......... 3% 54.4M 20s 6300K .......... .......... .......... .......... .......... 3% 14.2M 20s 6350K .......... .......... .......... .......... .......... 3% 42.1M 20s 6400K .......... .......... .......... .......... .......... 3% 22.6M 20s 6450K .......... .......... .......... .......... .......... 3% 135M 20s 6500K .......... .......... .......... .......... .......... 3% 21.5M 20s 6550K .......... .......... .......... .......... .......... 3% 22.6M 20s 6600K .......... .......... .......... .......... .......... 3% 30.4M 20s 6650K .......... .......... .......... .......... .......... 4% 30.8M 20s 6700K .......... .......... .......... .......... .......... 4% 61.1M 19s 6750K .......... .......... .......... .......... .......... 4% 32.9M 19s 6800K .......... .......... .......... .......... .......... 4% 32.4M 19s 6850K .......... .......... .......... .......... .......... 4% 21.8M 19s 6900K .......... .......... .......... .......... .......... 4% 60.3M 19s 6950K .......... .......... .......... .......... .......... 4% 99.1M 19s 7000K .......... .......... .......... .......... .......... 4% 63.6M 19s 7050K .......... .......... .......... .......... .......... 4% 59.7M 19s 7100K .......... .......... .......... .......... .......... 4% 21.8M 19s 7150K .......... .......... .......... .......... .......... 4% 11.5M 19s 7200K .......... .......... .......... .......... .......... 4% 103M 18s 7250K .......... .......... .......... .......... .......... 4% 155M 18s 7300K .......... .......... .......... .......... .......... 4% 2.34M 19s 7350K .......... .......... .......... .......... .......... 4% 9.40M 19s 7400K .......... .......... .......... .......... .......... 4% 88.7M 18s 7450K .......... .......... .......... .......... .......... 4% 14.5M 18s 7500K .......... .......... .......... .......... .......... 4% 82.8M 18s 7550K .......... .......... .......... .......... .......... 4% 38.5M 18s 7600K .......... .......... .......... .......... .......... 4% 71.5M 18s 7650K .......... .......... .......... .......... .......... 4% 16.4M 18s 7700K .......... .......... .......... .......... .......... 4% 15.8M 18s 7750K .......... .......... .......... .......... .......... 4% 91.1M 18s 7800K .......... .......... .......... .......... .......... 4% 13.1M 18s 7850K .......... .......... .......... .......... .......... 4% 87.1M 18s 7900K .......... .......... .......... .......... .......... 4% 50.4M 18s 7950K .......... .......... .......... .......... .......... 4% 21.3M 18s 8000K .......... .......... .......... .......... .......... 4% 23.4M 17s 8050K .......... .......... .......... .......... .......... 4% 22.5M 17s 8100K .......... .......... .......... .......... .......... 4% 42.8M 17s 8150K .......... .......... .......... .......... .......... 4% 47.4M 17s 8200K .......... .......... .......... .......... .......... 4% 26.0M 17s 8250K .......... .......... .......... .......... .......... 4% 36.7M 17s 8300K .......... .......... .......... .......... .......... 5% 101M 17s 8350K .......... .......... .......... .......... .......... 5% 16.6M 17s 8400K .......... .......... .......... .......... .......... 5% 81.7M 17s 8450K .......... .......... .......... .......... .......... 5% 65.8M 17s 8500K .......... .......... .......... .......... .......... 5% 44.3M 17s 8550K .......... .......... .......... .......... .......... 5% 133M 17s 8600K .......... .......... .......... .......... .......... 5% 50.0M 16s 8650K .......... .......... .......... .......... .......... 5% 19.4M 16s 8700K .......... .......... .......... .......... .......... 5% 27.6M 16s 8750K .......... .......... .......... .......... .......... 5% 20.4M 16s 8800K .......... .......... .......... .......... .......... 5% 79.5M 16s 8850K .......... .......... .......... .......... .......... 5% 2.03M 17s 8900K .......... .......... .......... .......... .......... 5% 17.4M 16s 8950K .......... .......... .......... .......... .......... 5% 12.0M 16s 9000K .......... .......... .......... .......... .......... 5% 83.2M 16s 9050K .......... .......... .......... .......... .......... 5% 51.4M 16s 9100K .......... .......... .......... .......... .......... 5% 51.7M 16s 9150K .......... .......... .......... .......... .......... 5% 17.2M 16s 9200K .......... .......... .......... .......... .......... 5% 30.0M 16s 9250K .......... .......... .......... .......... .......... 5% 60.3M 16s 9300K .......... .......... .......... .......... .......... 5% 48.9M 16s 9350K .......... .......... .......... .......... .......... 5% 13.0M 16s 9400K .......... .......... .......... .......... .......... 5% 23.0M 16s 9450K .......... .......... .......... .......... .......... 5% 31.2M 16s 9500K .......... .......... .......... .......... .......... 5% 85.2M 16s 9550K .......... .......... .......... .......... .......... 5% 24.7M 16s 9600K .......... .......... .......... .......... .......... 5% 20.9M 16s 9650K .......... .......... .......... .......... .......... 5% 23.1M 16s 9700K .......... .......... .......... .......... .......... 5% 59.1M 15s 9750K .......... .......... .......... .......... .......... 5% 27.9M 15s 9800K .......... .......... .......... .......... .......... 5% 14.5M 15s 9850K .......... .......... .......... .......... .......... 5% 72.8M 15s 9900K .......... .......... .......... .......... .......... 5% 26.2M 15s 9950K .......... .......... .......... .......... .......... 6% 97.1M 15s 10000K .......... .......... .......... .......... .......... 6% 13.8M 15s 10050K .......... .......... .......... .......... .......... 6% 54.0M 15s 10100K .......... .......... .......... .......... .......... 6% 33.4M 15s 10150K .......... .......... .......... .......... .......... 6% 23.1M 15s 10200K .......... .......... .......... .......... .......... 6% 43.3M 15s 10250K .......... .......... .......... .......... .......... 6% 21.2M 15s 10300K .......... .......... .......... .......... .......... 6% 26.7M 15s 10350K .......... .......... .......... .......... .......... 6% 3.01M 15s 10400K .......... .......... .......... .......... .......... 6% 19.5M 15s 10450K .......... .......... .......... .......... .......... 6% 11.5M 15s 10500K .......... .......... .......... .......... .......... 6% 14.7M 15s 10550K .......... .......... .......... .......... .......... 6% 24.1M 15s 10600K .......... .......... .......... .......... .......... 6% 31.5M 15s 10650K .......... .......... .......... .......... .......... 6% 111M 15s 10700K .......... .......... .......... .......... .......... 6% 24.4M 15s 10750K .......... .......... .......... .......... .......... 6% 29.9M 15s 10800K .......... .......... .......... .......... .......... 6% 45.4M 15s 10850K .......... .......... .......... .......... .......... 6% 11.4M 15s 10900K .......... .......... .......... .......... .......... 6% 47.3M 15s 10950K .......... .......... .......... .......... .......... 6% 49.5M 14s 11000K .......... .......... .......... .......... .......... 6% 23.1M 14s 11050K .......... .......... .......... .......... .......... 6% 29.2M 14s 11100K .......... .......... .......... .......... .......... 6% 67.0M 14s 11150K .......... .......... .......... .......... .......... 6% 24.0M 14s 11200K .......... .......... .......... .......... .......... 6% 22.9M 14s 11250K .......... .......... .......... .......... .......... 6% 33.2M 14s 11300K .......... .......... .......... .......... .......... 6% 29.7M 14s 11350K .......... .......... .......... .......... .......... 6% 26.2M 14s 11400K .......... .......... .......... .......... .......... 6% 109M 14s 11450K .......... .......... .......... .......... .......... 6% 32.4M 14s 11500K .......... .......... .......... .......... .......... 6% 12.9M 14s 11550K .......... .......... .......... .......... .......... 6% 59.2M 14s 11600K .......... .......... .......... .......... .......... 6% 71.9M 14s 11650K .......... .......... .......... .......... .......... 7% 40.7M 14s 11700K .......... .......... .......... .......... .......... 7% 21.6M 14s 11750K .......... .......... .......... .......... .......... 7% 23.9M 14s 11800K .......... .......... .......... .......... .......... 7% 28.7M 14s 11850K .......... .......... .......... .......... .......... 7% 24.0M 14s 11900K .......... .......... .......... .......... .......... 7% 2.81M 14s 11950K .......... .......... .......... .......... .......... 7% 12.3M 14s 12000K .......... .......... .......... .......... .......... 7% 16.6M 14s 12050K .......... .......... .......... .......... .......... 7% 28.9M 14s 12100K .......... .......... .......... .......... .......... 7% 23.6M 14s 12150K .......... .......... .......... .......... .......... 7% 46.2M 14s 12200K .......... .......... .......... .......... .......... 7% 20.0M 14s 12250K .......... .......... .......... .......... .......... 7% 48.5M 14s 12300K .......... .......... .......... .......... .......... 7% 70.2M 14s 12350K .......... .......... .......... .......... .......... 7% 43.2M 14s 12400K .......... .......... .......... .......... .......... 7% 10.2M 14s 12450K .......... .......... .......... .......... .......... 7% 45.6M 14s 12500K .......... .......... .......... .......... .......... 7% 112M 13s 12550K .......... .......... .......... .......... .......... 7% 15.1M 13s 12600K .......... .......... .......... .......... .......... 7% 30.3M 13s 12650K .......... .......... .......... .......... .......... 7% 19.0M 13s 12700K .......... .......... .......... .......... .......... 7% 30.7M 13s 12750K .......... .......... .......... .......... .......... 7% 73.2M 13s 12800K .......... .......... .......... .......... .......... 7% 20.8M 13s 12850K .......... .......... .......... .......... .......... 7% 34.2M 13s 12900K .......... .......... .......... .......... .......... 7% 30.2M 13s 12950K .......... .......... .......... .......... .......... 7% 33.7M 13s 13000K .......... .......... .......... .......... .......... 7% 66.8M 13s 13050K .......... .......... .......... .......... .......... 7% 13.1M 13s 13100K .......... .......... .......... .......... .......... 7% 56.5M 13s 13150K .......... .......... .......... .......... .......... 7% 53.4M 13s 13200K .......... .......... .......... .......... .......... 7% 126M 13s 13250K .......... .......... .......... .......... .......... 7% 28.1M 13s 13300K .......... .......... .......... .......... .......... 8% 16.7M 13s 13350K .......... .......... .......... .......... .......... 8% 23.8M 13s 13400K .......... .......... .......... .......... .......... 8% 3.20M 13s 13450K .......... .......... .......... .......... .......... 8% 13.0M 13s 13500K .......... .......... .......... .......... .......... 8% 10.5M 13s 13550K .......... .......... .......... .......... .......... 8% 24.9M 13s 13600K .......... .......... .......... .......... .......... 8% 49.6M 13s 13650K .......... .......... .......... .......... .......... 8% 14.6M 13s 13700K .......... .......... .......... .......... .......... 8% 148M 13s 13750K .......... .......... .......... .......... .......... 8% 15.5M 13s 13800K .......... .......... .......... .......... .......... 8% 21.3M 13s 13850K .......... .......... .......... .......... .......... 8% 6.80M 13s 13900K .......... .......... .......... .......... .......... 8% 115M 13s 13950K .......... .......... .......... .......... .......... 8% 78.6M 13s 14000K .......... .......... .......... .......... .......... 8% 73.9M 13s 14050K .......... .......... .......... .......... .......... 8% 64.7M 13s 14100K .......... .......... .......... .......... .......... 8% 39.2M 13s 14150K .......... .......... .......... .......... .......... 8% 60.3M 13s 14200K .......... .......... .......... .......... .......... 8% 34.7M 13s 14250K .......... .......... .......... .......... .......... 8% 38.8M 13s 14300K .......... .......... .......... .......... .......... 8% 15.7M 13s 14350K .......... .......... .......... .......... .......... 8% 32.6M 12s 14400K .......... .......... .......... .......... .......... 8% 45.4M 12s 14450K .......... .......... .......... .......... .......... 8% 28.2M 12s 14500K .......... .......... .......... .......... .......... 8% 47.0M 12s 14550K .......... .......... .......... .......... .......... 8% 48.2M 12s 14600K .......... .......... .......... .......... .......... 8% 16.8M 12s 14650K .......... .......... .......... .......... .......... 8% 40.9M 12s 14700K .......... .......... .......... .......... .......... 8% 33.6M 12s 14750K .......... .......... .......... .......... .......... 8% 21.8M 12s 14800K .......... .......... .......... .......... .......... 8% 21.1M 12s 14850K .......... .......... .......... .......... .......... 8% 81.7M 12s 14900K .......... .......... .......... .......... .......... 8% 19.4M 12s 14950K .......... .......... .......... .......... .......... 9% 2.91M 12s 15000K .......... .......... .......... .......... .......... 9% 95.3M 12s 15050K .......... .......... .......... .......... .......... 9% 4.33M 12s 15100K .......... .......... .......... .......... .......... 9% 166M 12s 15150K .......... .......... .......... .......... .......... 9% 138M 12s 15200K .......... .......... .......... .......... .......... 9% 164M 12s 15250K .......... .......... .......... .......... .......... 9% 63.8M 12s 15300K .......... .......... .......... .......... .......... 9% 49.4M 12s 15350K .......... .......... .......... .......... .......... 9% 20.9M 12s 15400K .......... .......... .......... .......... .......... 9% 17.4M 12s 15450K .......... .......... .......... .......... .......... 9% 24.1M 12s 15500K .......... .......... .......... .......... .......... 9% 50.1M 12s 15550K .......... .......... .......... .......... .......... 9% 31.5M 12s 15600K .......... .......... .......... .......... .......... 9% 14.7M 12s 15650K .......... .......... .......... .......... .......... 9% 78.0M 12s 15700K .......... .......... .......... .......... .......... 9% 48.0M 12s 15750K .......... .......... .......... .......... .......... 9% 27.1M 12s 15800K .......... .......... .......... .......... .......... 9% 41.9M 12s 15850K .......... .......... .......... .......... .......... 9% 22.0M 12s 15900K .......... .......... .......... .......... .......... 9% 18.9M 12s 15950K .......... .......... .......... .......... .......... 9% 33.0M 12s *** WARNING: skipped 204162 bytes of output *** 150200K .......... .......... .......... .......... .......... 90% 22.1M 1s 150250K .......... .......... .......... .......... .......... 90% 19.7M 1s 150300K .......... .......... .......... .......... .......... 90% 20.7M 1s 150350K .......... .......... .......... .......... .......... 90% 18.6M 1s 150400K .......... .......... .......... .......... .......... 90% 19.5M 1s 150450K .......... .......... .......... .......... .......... 90% 46.1M 1s 150500K .......... .......... .......... .......... .......... 90% 19.1M 1s 150550K .......... .......... .......... .......... .......... 90% 12.4M 1s 150600K .......... .......... .......... .......... .......... 90% 16.3M 1s 150650K .......... .......... .......... .......... .......... 90% 122M 1s 150700K .......... .......... .......... .......... .......... 90% 16.8M 1s 150750K .......... .......... .......... .......... .......... 90% 47.7M 1s 150800K .......... .......... .......... .......... .......... 90% 44.2M 1s 150850K .......... .......... .......... .......... .......... 90% 21.8M 1s 150900K .......... .......... .......... .......... .......... 90% 73.6M 1s 150950K .......... .......... .......... .......... .......... 90% 18.4M 1s 151000K .......... .......... .......... .......... .......... 90% 12.4M 1s 151050K .......... .......... .......... .......... .......... 90% 18.7M 1s 151100K .......... .......... .......... .......... .......... 90% 12.7M 1s 151150K .......... .......... .......... .......... .......... 90% 23.4M 1s 151200K .......... .......... .......... .......... .......... 90% 12.2M 1s 151250K .......... .......... .......... .......... .......... 90% 16.0M 1s 151300K .......... .......... .......... .......... .......... 90% 19.5M 1s 151350K .......... .......... .......... .......... .......... 90% 64.8M 1s 151400K .......... .......... .......... .......... .......... 90% 15.9M 1s 151450K .......... .......... .......... .......... .......... 90% 37.7M 1s 151500K .......... .......... .......... .......... .......... 91% 23.2M 1s 151550K .......... .......... .......... .......... .......... 91% 13.7M 1s 151600K .......... .......... .......... .......... .......... 91% 24.5M 1s 151650K .......... .......... .......... .......... .......... 91% 20.1M 1s 151700K .......... .......... .......... .......... .......... 91% 30.8M 1s 151750K .......... .......... .......... .......... .......... 91% 18.0M 1s 151800K .......... .......... .......... .......... .......... 91% 26.8M 1s 151850K .......... .......... .......... .......... .......... 91% 78.1M 1s 151900K .......... .......... .......... .......... .......... 91% 18.2M 1s 151950K .......... .......... .......... .......... .......... 91% 14.7M 1s 152000K .......... .......... .......... .......... .......... 91% 119M 1s 152050K .......... .......... .......... .......... .......... 91% 19.9M 1s 152100K .......... .......... .......... .......... .......... 91% 11.7M 1s 152150K .......... .......... .......... .......... .......... 91% 17.9M 1s 152200K .......... .......... .......... .......... .......... 91% 13.1M 1s 152250K .......... .......... .......... .......... .......... 91% 33.8M 1s 152300K .......... .......... .......... .......... .......... 91% 106M 1s 152350K .......... .......... .......... .......... .......... 91% 43.1M 1s 152400K .......... .......... .......... .......... .......... 91% 24.5M 1s 152450K .......... .......... .......... .......... .......... 91% 18.5M 1s 152500K .......... .......... .......... .......... .......... 91% 12.4M 1s 152550K .......... .......... .......... .......... .......... 91% 104M 1s 152600K .......... .......... .......... .......... .......... 91% 22.3M 1s 152650K .......... .......... .......... .......... .......... 91% 8.15M 1s 152700K .......... .......... .......... .......... .......... 91% 14.8M 1s 152750K .......... .......... .......... .......... .......... 91% 47.3M 1s 152800K .......... .......... .......... .......... .......... 91% 16.7M 1s 152850K .......... .......... .......... .......... .......... 91% 26.9M 1s 152900K .......... .......... .......... .......... .......... 91% 16.6M 1s 152950K .......... .......... .......... .......... .......... 91% 25.4M 1s 153000K .......... .......... .......... .......... .......... 91% 53.0M 1s 153050K .......... .......... .......... .......... .......... 91% 46.3M 1s 153100K .......... .......... .......... .......... .......... 91% 8.79M 1s 153150K .......... .......... .......... .......... .......... 92% 20.4M 1s 153200K .......... .......... .......... .......... .......... 92% 34.2M 1s 153250K .......... .......... .......... .......... .......... 92% 89.4M 1s 153300K .......... .......... .......... .......... .......... 92% 17.4M 1s 153350K .......... .......... .......... .......... .......... 92% 27.5M 1s 153400K .......... .......... .......... .......... .......... 92% 15.6M 1s 153450K .......... .......... .......... .......... .......... 92% 88.0M 1s 153500K .......... .......... .......... .......... .......... 92% 14.8M 1s 153550K .......... .......... .......... .......... .......... 92% 22.4M 1s 153600K .......... .......... .......... .......... .......... 92% 13.7M 1s 153650K .......... .......... .......... .......... .......... 92% 15.1M 1s 153700K .......... .......... .......... .......... .......... 92% 91.5M 1s 153750K .......... .......... .......... .......... .......... 92% 13.6M 1s 153800K .......... .......... .......... .......... .......... 92% 27.9M 1s 153850K .......... .......... .......... .......... .......... 92% 55.8M 1s 153900K .......... .......... .......... .......... .......... 92% 32.6M 1s 153950K .......... .......... .......... .......... .......... 92% 59.6M 1s 154000K .......... .......... .......... .......... .......... 92% 16.3M 1s 154050K .......... .......... .......... .......... .......... 92% 7.24M 1s 154100K .......... .......... .......... .......... .......... 92% 102M 1s 154150K .......... .......... .......... .......... .......... 92% 114M 1s 154200K .......... .......... .......... .......... .......... 92% 9.56M 1s 154250K .......... .......... .......... .......... .......... 92% 9.85M 1s 154300K .......... .......... .......... .......... .......... 92% 23.1M 1s 154350K .......... .......... .......... .......... .......... 92% 63.5M 1s 154400K .......... .......... .......... .......... .......... 92% 62.1M 1s 154450K .......... .......... .......... .......... .......... 92% 15.5M 1s 154500K .......... .......... .......... .......... .......... 92% 21.2M 1s 154550K .......... .......... .......... .......... .......... 92% 46.8M 1s 154600K .......... .......... .......... .......... .......... 92% 8.33M 1s 154650K .......... .......... .......... .......... .......... 92% 77.7M 1s 154700K .......... .......... .......... .......... .......... 92% 24.6M 1s 154750K .......... .......... .......... .......... .......... 92% 35.6M 1s 154800K .......... .......... .......... .......... .......... 93% 20.2M 1s 154850K .......... .......... .......... .......... .......... 93% 61.5M 1s 154900K .......... .......... .......... .......... .......... 93% 23.9M 1s 154950K .......... .......... .......... .......... .......... 93% 15.1M 1s 155000K .......... .......... .......... .......... .......... 93% 16.9M 1s 155050K .......... .......... .......... .......... .......... 93% 21.0M 1s 155100K .......... .......... .......... .......... .......... 93% 135M 1s 155150K .......... .......... .......... .......... .......... 93% 11.1M 1s 155200K .......... .......... .......... .......... .......... 93% 17.3M 1s 155250K .......... .......... .......... .......... .......... 93% 14.6M 1s 155300K .......... .......... .......... .......... .......... 93% 35.1M 1s 155350K .......... .......... .......... .......... .......... 93% 35.9M 1s 155400K .......... .......... .......... .......... .......... 93% 51.8M 1s 155450K .......... .......... .......... .......... .......... 93% 27.8M 1s 155500K .......... .......... .......... .......... .......... 93% 20.5M 1s 155550K .......... .......... .......... .......... .......... 93% 56.7M 1s 155600K .......... .......... .......... .......... .......... 93% 13.0M 1s 155650K .......... .......... .......... .......... .......... 93% 24.2M 1s 155700K .......... .......... .......... .......... .......... 93% 14.8M 1s 155750K .......... .......... .......... .......... .......... 93% 7.81M 1s 155800K .......... .......... .......... .......... .......... 93% 64.4M 1s 155850K .......... .......... .......... .......... .......... 93% 13.4M 1s 155900K .......... .......... .......... .......... .......... 93% 19.7M 1s 155950K .......... .......... .......... .......... .......... 93% 19.6M 1s 156000K .......... .......... .......... .......... .......... 93% 30.1M 1s 156050K .......... .......... .......... .......... .......... 93% 50.1M 1s 156100K .......... .......... .......... .......... .......... 93% 27.8M 1s 156150K .......... .......... .......... .......... .......... 93% 10.5M 1s 156200K .......... .......... .......... .......... .......... 93% 18.4M 1s 156250K .......... .......... .......... .......... .......... 93% 95.9M 1s 156300K .......... .......... .......... .......... .......... 93% 22.7M 1s 156350K .......... .......... .......... .......... .......... 93% 22.8M 1s 156400K .......... .......... .......... .......... .......... 93% 22.0M 1s 156450K .......... .......... .......... .......... .......... 93% 20.8M 0s 156500K .......... .......... .......... .......... .......... 94% 62.2M 0s 156550K .......... .......... .......... .......... .......... 94% 16.5M 0s 156600K .......... .......... .......... .......... .......... 94% 23.0M 0s 156650K .......... .......... .......... .......... .......... 94% 16.9M 0s 156700K .......... .......... .......... .......... .......... 94% 14.9M 0s 156750K .......... .......... .......... .......... .......... 94% 99.6M 0s 156800K .......... .......... .......... .......... .......... 94% 11.9M 0s 156850K .......... .......... .......... .......... .......... 94% 30.2M 0s 156900K .......... .......... .......... .......... .......... 94% 22.3M 0s 156950K .......... .......... .......... .......... .......... 94% 92.4M 0s 157000K .......... .......... .......... .......... .......... 94% 58.2M 0s 157050K .......... .......... .......... .......... .......... 94% 17.7M 0s 157100K .......... .......... .......... .......... .......... 94% 7.60M 0s 157150K .......... .......... .......... .......... .......... 94% 99.6M 0s 157200K .......... .......... .......... .......... .......... 94% 174M 0s 157250K .......... .......... .......... .......... .......... 94% 10.8M 0s 157300K .......... .......... .......... .......... .......... 94% 10.9M 0s 157350K .......... .......... .......... .......... .......... 94% 12.1M 0s 157400K .......... .......... .......... .......... .......... 94% 21.7M 0s 157450K .......... .......... .......... .......... .......... 94% 107M 0s 157500K .......... .......... .......... .......... .......... 94% 20.5M 0s 157550K .......... .......... .......... .......... .......... 94% 24.6M 0s 157600K .......... .......... .......... .......... .......... 94% 20.8M 0s 157650K .......... .......... .......... .......... .......... 94% 60.3M 0s 157700K .......... .......... .......... .......... .......... 94% 13.0M 0s 157750K .......... .......... .......... .......... .......... 94% 19.2M 0s 157800K .......... .......... .......... .......... .......... 94% 21.7M 0s 157850K .......... .......... .......... .......... .......... 94% 19.5M 0s 157900K .......... .......... .......... .......... .......... 94% 60.4M 0s 157950K .......... .......... .......... .......... .......... 94% 19.3M 0s 158000K .......... .......... .......... .......... .......... 94% 25.5M 0s 158050K .......... .......... .......... .......... .......... 94% 16.8M 0s 158100K .......... .......... .......... .......... .......... 94% 28.7M 0s 158150K .......... .......... .......... .......... .......... 95% 55.7M 0s 158200K .......... .......... .......... .......... .......... 95% 15.0M 0s 158250K .......... .......... .......... .......... .......... 95% 16.3M 0s 158300K .......... .......... .......... .......... .......... 95% 56.1M 0s 158350K .......... .......... .......... .......... .......... 95% 9.03M 0s 158400K .......... .......... .......... .......... .......... 95% 130M 0s 158450K .......... .......... .......... .......... .......... 95% 31.4M 0s 158500K .......... .......... .......... .......... .......... 95% 53.9M 0s 158550K .......... .......... .......... .......... .......... 95% 15.8M 0s 158600K .......... .......... .......... .......... .......... 95% 165M 0s 158650K .......... .......... .......... .......... .......... 95% 11.3M 0s 158700K .......... .......... .......... .......... .......... 95% 27.8M 0s 158750K .......... .......... .......... .......... .......... 95% 21.9M 0s 158800K .......... .......... .......... .......... .......... 95% 7.42M 0s 158850K .......... .......... .......... .......... .......... 95% 55.6M 0s 158900K .......... .......... .......... .......... .......... 95% 11.7M 0s 158950K .......... .......... .......... .......... .......... 95% 20.0M 0s 159000K .......... .......... .......... .......... .......... 95% 23.1M 0s 159050K .......... .......... .......... .......... .......... 95% 55.3M 0s 159100K .......... .......... .......... .......... .......... 95% 27.0M 0s 159150K .......... .......... .......... .......... .......... 95% 18.9M 0s 159200K .......... .......... .......... .......... .......... 95% 14.1M 0s 159250K .......... .......... .......... .......... .......... 95% 17.9M 0s 159300K .......... .......... .......... .......... .......... 95% 94.0M 0s 159350K .......... .......... .......... .......... .......... 95% 18.8M 0s 159400K .......... .......... .......... .......... .......... 95% 22.0M 0s 159450K .......... .......... .......... .......... .......... 95% 24.5M 0s 159500K .......... .......... .......... .......... .......... 95% 19.1M 0s 159550K .......... .......... .......... .......... .......... 95% 65.9M 0s 159600K .......... .......... .......... .......... .......... 95% 11.8M 0s 159650K .......... .......... .......... .......... .......... 95% 25.2M 0s 159700K .......... .......... .......... .......... .......... 95% 15.3M 0s 159750K .......... .......... .......... .......... .......... 95% 76.7M 0s 159800K .......... .......... .......... .......... .......... 96% 15.6M 0s 159850K .......... .......... .......... .......... .......... 96% 11.4M 0s 159900K .......... .......... .......... .......... .......... 96% 95.4M 0s 159950K .......... .......... .......... .......... .......... 96% 16.4M 0s 160000K .......... .......... .......... .......... .......... 96% 65.6M 0s 160050K .......... .......... .......... .......... .......... 96% 41.2M 0s 160100K .......... .......... .......... .......... .......... 96% 21.1M 0s 160150K .......... .......... .......... .......... .......... 96% 15.5M 0s 160200K .......... .......... .......... .......... .......... 96% 21.6M 0s 160250K .......... .......... .......... .......... .......... 96% 41.7M 0s 160300K .......... .......... .......... .......... .......... 96% 12.8M 0s 160350K .......... .......... .......... .......... .......... 96% 13.5M 0s 160400K .......... .......... .......... .......... .......... 96% 11.5M 0s 160450K .......... .......... .......... .......... .......... 96% 65.7M 0s 160500K .......... .......... .......... .......... .......... 96% 19.1M 0s 160550K .......... .......... .......... .......... .......... 96% 19.5M 0s 160600K .......... .......... .......... .......... .......... 96% 26.1M 0s 160650K .......... .......... .......... .......... .......... 96% 19.5M 0s 160700K .......... .......... .......... .......... .......... 96% 85.4M 0s 160750K .......... .......... .......... .......... .......... 96% 14.8M 0s 160800K .......... .......... .......... .......... .......... 96% 16.2M 0s 160850K .......... .......... .......... .......... .......... 96% 27.5M 0s 160900K .......... .......... .......... .......... .......... 96% 13.6M 0s 160950K .......... .......... .......... .......... .......... 96% 94.8M 0s 161000K .......... .......... .......... .......... .......... 96% 17.6M 0s 161050K .......... .......... .......... .......... .......... 96% 20.0M 0s 161100K .......... .......... .......... .......... .......... 96% 24.3M 0s 161150K .......... .......... .......... .......... .......... 96% 31.8M 0s 161200K .......... .......... .......... .......... .......... 96% 19.8M 0s 161250K .......... .......... .......... .......... .......... 96% 19.6M 0s 161300K .......... .......... .......... .......... .......... 96% 18.7M 0s 161350K .......... .......... .......... .......... .......... 96% 12.4M 0s 161400K .......... .......... .......... .......... .......... 96% 21.7M 0s 161450K .......... .......... .......... .......... .......... 96% 28.8M 0s 161500K .......... .......... .......... .......... .......... 97% 20.6M 0s 161550K .......... .......... .......... .......... .......... 97% 28.5M 0s 161600K .......... .......... .......... .......... .......... 97% 18.5M 0s 161650K .......... .......... .......... .......... .......... 97% 89.5M 0s 161700K .......... .......... .......... .......... .......... 97% 13.2M 0s 161750K .......... .......... .......... .......... .......... 97% 58.5M 0s 161800K .......... .......... .......... .......... .......... 97% 12.4M 0s 161850K .......... .......... .......... .......... .......... 97% 16.3M 0s 161900K .......... .......... .......... .......... .......... 97% 35.4M 0s 161950K .......... .......... .......... .......... .......... 97% 9.31M 0s 162000K .......... .......... .......... .......... .......... 97% 14.8M 0s 162050K .......... .......... .......... .......... .......... 97% 32.5M 0s 162100K .......... .......... .......... .......... .......... 97% 41.4M 0s 162150K .......... .......... .......... .......... .......... 97% 21.4M 0s 162200K .......... .......... .......... .......... .......... 97% 38.4M 0s 162250K .......... .......... .......... .......... .......... 97% 14.5M 0s 162300K .......... .......... .......... .......... .......... 97% 10.7M 0s 162350K .......... .......... .......... .......... .......... 97% 32.5M 0s 162400K .......... .......... .......... .......... .......... 97% 48.7M 0s 162450K .......... .......... .......... .......... .......... 97% 15.6M 0s 162500K .......... .......... .......... .......... .......... 97% 23.8M 0s 162550K .......... .......... .......... .......... .......... 97% 32.3M 0s 162600K .......... .......... .......... .......... .......... 97% 29.4M 0s 162650K .......... .......... .......... .......... .......... 97% 16.8M 0s 162700K .......... .......... .......... .......... .......... 97% 26.7M 0s 162750K .......... .......... .......... .......... .......... 97% 17.8M 0s 162800K .......... .......... .......... .......... .......... 97% 63.2M 0s 162850K .......... .......... .......... .......... .......... 97% 15.8M 0s 162900K .......... .......... .......... .......... .......... 97% 15.9M 0s 162950K .......... .......... .......... .......... .......... 97% 12.5M 0s 163000K .......... .......... .......... .......... .......... 97% 24.2M 0s 163050K .......... .......... .......... .......... .......... 97% 50.5M 0s 163100K .......... .......... .......... .......... .......... 97% 24.5M 0s 163150K .......... .......... .......... .......... .......... 98% 27.2M 0s 163200K .......... .......... .......... .......... .......... 98% 22.4M 0s 163250K .......... .......... .......... .......... .......... 98% 11.0M 0s 163300K .......... .......... .......... .......... .......... 98% 72.7M 0s 163350K .......... .......... .......... .......... .......... 98% 18.7M 0s 163400K .......... .......... .......... .......... .......... 98% 14.4M 0s 163450K .......... .......... .......... .......... .......... 98% 11.6M 0s 163500K .......... .......... .......... .......... .......... 98% 29.2M 0s 163550K .......... .......... .......... .......... .......... 98% 14.4M 0s 163600K .......... .......... .......... .......... .......... 98% 25.7M 0s 163650K .......... .......... .......... .......... .......... 98% 25.8M 0s 163700K .......... .......... .......... .......... .......... 98% 26.6M 0s 163750K .......... .......... .......... .......... .......... 98% 97.2M 0s 163800K .......... .......... .......... .......... .......... 98% 13.0M 0s 163850K .......... .......... .......... .......... .......... 98% 10.8M 0s 163900K .......... .......... .......... .......... .......... 98% 31.2M 0s 163950K .......... .......... .......... .......... .......... 98% 18.8M 0s 164000K .......... .......... .......... .......... .......... 98% 31.9M 0s 164050K .......... .......... .......... .......... .......... 98% 22.9M 0s 164100K .......... .......... .......... .......... .......... 98% 28.9M 0s 164150K .......... .......... .......... .......... .......... 98% 18.3M 0s 164200K .......... .......... .......... .......... .......... 98% 79.7M 0s 164250K .......... .......... .......... .......... .......... 98% 19.7M 0s 164300K .......... .......... .......... .......... .......... 98% 18.0M 0s 164350K .......... .......... .......... .......... .......... 98% 17.4M 0s 164400K .......... .......... .......... .......... .......... 98% 9.15M 0s 164450K .......... .......... .......... .......... .......... 98% 122M 0s 164500K .......... .......... .......... .......... .......... 98% 20.5M 0s 164550K .......... .......... .......... .......... .......... 98% 21.8M 0s 164600K .......... .......... .......... .......... .......... 98% 25.9M 0s 164650K .......... .......... .......... .......... .......... 98% 21.8M 0s 164700K .......... .......... .......... .......... .......... 98% 80.9M 0s 164750K .......... .......... .......... .......... .......... 98% 7.10M 0s 164800K .......... .......... .......... .......... .......... 99% 117M 0s 164850K .......... .......... .......... .......... .......... 99% 36.1M 0s 164900K .......... .......... .......... .......... .......... 99% 52.0M 0s 164950K .......... .......... .......... .......... .......... 99% 13.7M 0s 165000K .......... .......... .......... .......... .......... 99% 9.19M 0s 165050K .......... .......... .......... .......... .......... 99% 23.1M 0s 165100K .......... .......... .......... .......... .......... 99% 13.2M 0s 165150K .......... .......... .......... .......... .......... 99% 77.8M 0s 165200K .......... .......... .......... .......... .......... 99% 29.2M 0s 165250K .......... .......... .......... .......... .......... 99% 23.6M 0s 165300K .......... .......... .......... .......... .......... 99% 26.8M 0s 165350K .......... .......... .......... .......... .......... 99% 11.2M 0s 165400K .......... .......... .......... .......... .......... 99% 31.1M 0s 165450K .......... .......... .......... .......... .......... 99% 21.1M 0s 165500K .......... .......... .......... .......... .......... 99% 23.9M 0s 165550K .......... .......... .......... .......... .......... 99% 12.3M 0s 165600K .......... .......... .......... .......... .......... 99% 96.0M 0s 165650K .......... .......... .......... .......... .......... 99% 20.9M 0s 165700K .......... .......... .......... .......... .......... 99% 24.4M 0s 165750K .......... .......... .......... .......... .......... 99% 26.3M 0s 165800K .......... .......... .......... .......... .......... 99% 22.7M 0s 165850K .......... .......... .......... .......... .......... 99% 35.1M 0s 165900K .......... .......... .......... .......... .......... 99% 16.1M 0s 165950K .......... .......... .......... .......... .......... 99% 10.2M 0s 166000K .......... .......... .......... .......... .......... 99% 26.6M 0s 166050K .......... .......... .......... .......... .......... 99% 17.2M 0s 166100K .......... .......... .......... .......... .......... 99% 115M 0s 166150K .......... .......... .......... .......... .......... 99% 28.9M 0s 166200K .......... .......... .......... .......... .......... 99% 20.9M 0s 166250K .......... .......... .......... .......... .......... 99% 25.0M 0s 166300K .......... .......... .......... .......... .......... 99% 19.4M 0s 166350K .......... .......... .......... .......... .......... 99% 18.9M 0s 166400K .......... .......... .......... .......... .......... 99% 20.2M 0s 166450K .......... .......... .......... .......... .......... 99% 16.5M 0s 166500K .. 100% 3858G=8.3s 2021-01-18 14:38:37 (19.7 MB/s) - ‘cifar-10-python.tar.gz’ saved [170498071/170498071] 
%sh
tar zxvf cifar-10-python.tar.gz
cifar-10-batches-py/ cifar-10-batches-py/data_batch_4 cifar-10-batches-py/readme.html cifar-10-batches-py/test_batch cifar-10-batches-py/data_batch_3 cifar-10-batches-py/batches.meta cifar-10-batches-py/data_batch_2 cifar-10-batches-py/data_batch_5 cifar-10-batches-py/data_batch_1 
%sh ls -la cifar-10-batches-py
total 181884 drwxr-xr-x 2 2156 1103 4096 Jun 4 2009 . drwxr-xr-x 1 root root 4096 Jan 18 14:38 .. -rw-r--r-- 1 2156 1103 158 Mar 31 2009 batches.meta -rw-r--r-- 1 2156 1103 31035704 Mar 31 2009 data_batch_1 -rw-r--r-- 1 2156 1103 31035320 Mar 31 2009 data_batch_2 -rw-r--r-- 1 2156 1103 31035999 Mar 31 2009 data_batch_3 -rw-r--r-- 1 2156 1103 31035696 Mar 31 2009 data_batch_4 -rw-r--r-- 1 2156 1103 31035623 Mar 31 2009 data_batch_5 -rw-r--r-- 1 2156 1103 88 Jun 4 2009 readme.html -rw-r--r-- 1 2156 1103 31035526 Mar 31 2009 test_batch

Recall the classes are: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck

Here is the code to unpickle the batches.

Loaded in this way, each of the batch files contains a dictionary with the following elements:

  • data - a 10000x3072 numpy array of uint8s. Each row of the array stores a 32x32 colour image. The first 1024 entries contain the red channel values, the next 1024 the green, and the final 1024 the blue. The image is stored in row-major order, so that the first 32 entries of the array are the red channel values of the first row of the image.
  • labels - a list of 10000 numbers in the range 0-9. The number at index i indicates the label of the ith image in the array data.
def unpickle(file):
    import pickle
    with open(file, 'rb') as fo:
        dict = pickle.load(fo, encoding='bytes')# for Python 3, add the following param: encoding='bytes'
    return dict
 
dir = 'cifar-10-batches-py/'
 
batches = [unpickle(dir + 'data_batch_' + str(1+n)) for n in range(5)]

Now we need to reshape the data batches and concatenate the training batches into one big tensor.

import numpy as np
 
def decode(xy):
  x_train = np.reshape(xy[b'data'], (10000, 3, 32, 32)).transpose(0, 2, 3, 1)
  y_train = np.reshape(xy[b'labels'], (10000, 1))
  return (x_train, y_train)
 
decoded = [decode(data) for data in batches]
 
x_train = np.concatenate([data[0] for data in decoded])
y_train = np.concatenate([data[1] for data in decoded])
 
(x_test, y_test) = decode(unpickle(dir + 'test_batch'))
 
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
x_train shape: (50000, 32, 32, 3) 50000 train samples 10000 test samples

Let's visualize some of the images:

import matplotlib.pyplot as plt
 
fig = plt.figure()
for i in range(36):
  fig.add_subplot(6, 6, i+1)
  plt.imshow(x_train[i])
 
display(fig)

Recall that we are getting a categorical output via softmax across 10 neurons, corresponding to the output categories.

So we want to reshape our target values (training labels) to be 1-hot encoded, and Keras can calculate categorical crossentropy between its output layer and the target:

import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Conv2D, MaxPooling2D
 
num_classes = 10
 
# Convert class vectors to binary class matrices.
y_train_1hot = keras.utils.to_categorical(y_train, num_classes)
y_test_1hot = keras.utils.to_categorical(y_test, num_classes)
Using TensorFlow backend.

Here's a simple convolutional net to get you started. It will get you to over 57% accuracy in 5 epochs.

As inspiration, with a suitable network and parameters, it's possible to get over 99% test accuracy, although you won't have time to get there in today's session on this hardware.

note: if your network is not learning anything at all -- meaning regardless of settings, you're seeing a loss that doesn't decrease and a validation accuracy that is 10% (i.e., random chance) -- then restart your cluster

model = Sequential()
 
model.add(Conv2D(32, (3, 3), padding='same', input_shape=x_train.shape[1:]))
model.add(Activation('relu'))
 
model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dense(num_classes))
model.add(Activation('softmax'))
 
model.compile(loss='categorical_crossentropy', optimizer="adam", metrics=['accuracy'])
 
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
 
history = model.fit(x_train, y_train_1hot,
              batch_size=64,
              epochs=5,
              validation_data=(x_test, y_test_1hot),
              verbose=2)
WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. Train on 50000 samples, validate on 10000 samples Epoch 1/5
Cancelled

In this session, you probably won't have time to run each experiment for too many epochs ... but you can use this code to plot the training and validation losses:

fig, ax = plt.subplots()
fig.set_size_inches((5,5))
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
display(fig)