ScaDaMaLe Course site and book
This is a 2019-2021 augmentation and update of Adam Breindel's initial notebooks.
Thanks to Christian von Koch and William Anzén for their contributions towards making these materials Spark 3.0.1 and Python 3+ compliant.
CIFAR 10
Details at: https://www.cs.toronto.edu/~kriz/cifar.html
Summary (taken from that page):
The CIFAR-10 and CIFAR-100 are labeled subsets of the 80 million tiny images dataset. They were collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.
First, we'll mount the S3 bucket where I'm hosting the data:
# you may have to host the data yourself! - this should not work unless you can descramble
ACCESS="...SPORAA...KIAJZEH...PW46CWPUWUN...QPODO" # scrambled up
SECRET="...P7d7Sp7r1...Q9DuUvV...QAy1D+hjC...NxakJF+PXrAb...MXD1tZwBpGyN...1Ns5r5n1" # scrambled up
BUCKET = "cool-data"
MOUNT = "/mnt/cifar"
try:
dbutils.fs.mount("s3a://"+ ACCESS + ":" + SECRET + "@" + BUCKET, MOUNT)
except:
print("Error mounting ... possibly already mounted")
Error mounting ... possibly already mounted
This is in DBFS, which is available (via FUSE) at /dbfs ...
So the CIFAR data can be listed through following regular Linux shell command:
wget http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
--2021-01-18 14:38:29-- http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
Resolving www.cs.toronto.edu (www.cs.toronto.edu)... 128.100.3.30
Connecting to www.cs.toronto.edu (www.cs.toronto.edu)|128.100.3.30|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 170498071 (163M) [application/x-gzip]
Saving to: ‘cifar-10-python.tar.gz’
0K .......... .......... .......... .......... .......... 0% 340K 8m10s
50K .......... .......... .......... .......... .......... 0% 712K 6m2s
100K .......... .......... .......... .......... .......... 0% 677K 5m23s
150K .......... .......... .......... .......... .......... 0% 9.41M 4m6s
200K .......... .......... .......... .......... .......... 0% 727K 4m3s
250K .......... .......... .......... .......... .......... 0% 123M 3m23s
300K .......... .......... .......... .......... .......... 0% 33.2M 2m54s
350K .......... .......... .......... .......... .......... 0% 25.4M 2m33s
400K .......... .......... .......... .......... .......... 0% 723K 2m42s
450K .......... .......... .......... .......... .......... 0% 12.0M 2m27s
500K .......... .......... .......... .......... .......... 0% 9.97M 2m15s
550K .......... .......... .......... .......... .......... 0% 10.1M 2m5s
600K .......... .......... .......... .......... .......... 0% 13.8M 1m56s
650K .......... .......... .......... .......... .......... 0% 10.1M 1m49s
700K .......... .......... .......... .......... .......... 0% 1.01M 1m52s
750K .......... .......... .......... .......... .......... 0% 13.4M 1m46s
800K .......... .......... .......... .......... .......... 0% 13.0M 1m41s
850K .......... .......... .......... .......... .......... 0% 101M 95s
900K .......... .......... .......... .......... .......... 0% 25.4M 90s
950K .......... .......... .......... .......... .......... 0% 95.4M 86s
1000K .......... .......... .......... .......... .......... 0% 178M 82s
1050K .......... .......... .......... .......... .......... 0% 168M 78s
1100K .......... .......... .......... .......... .......... 0% 141M 75s
1150K .......... .......... .......... .......... .......... 0% 11.4M 72s
1200K .......... .......... .......... .......... .......... 0% 52.0M 69s
1250K .......... .......... .......... .......... .......... 0% 17.1M 67s
1300K .......... .......... .......... .......... .......... 0% 45.0M 65s
1350K .......... .......... .......... .......... .......... 0% 1.05M 68s
1400K .......... .......... .......... .......... .......... 0% 28.4M 66s
1450K .......... .......... .......... .......... .......... 0% 18.0M 64s
1500K .......... .......... .......... .......... .......... 0% 18.1M 62s
1550K .......... .......... .......... .......... .......... 0% 76.4M 60s
1600K .......... .......... .......... .......... .......... 0% 11.6M 59s
1650K .......... .......... .......... .......... .......... 1% 32.7M 57s
1700K .......... .......... .......... .......... .......... 1% 64.9M 56s
1750K .......... .......... .......... .......... .......... 1% 34.7M 54s
1800K .......... .......... .......... .......... .......... 1% 30.9M 53s
1850K .......... .......... .......... .......... .......... 1% 30.2M 51s
1900K .......... .......... .......... .......... .......... 1% 62.4M 50s
1950K .......... .......... .......... .......... .......... 1% 35.9M 49s
2000K .......... .......... .......... .......... .......... 1% 28.7M 48s
2050K .......... .......... .......... .......... .......... 1% 23.6M 47s
2100K .......... .......... .......... .......... .......... 1% 146M 46s
2150K .......... .......... .......... .......... .......... 1% 25.9M 45s
2200K .......... .......... .......... .......... .......... 1% 15.4M 44s
2250K .......... .......... .......... .......... .......... 1% 48.2M 43s
2300K .......... .......... .......... .......... .......... 1% 21.6M 43s
2350K .......... .......... .......... .......... .......... 1% 25.2M 42s
2400K .......... .......... .......... .......... .......... 1% 34.2M 41s
2450K .......... .......... .......... .......... .......... 1% 22.4M 40s
2500K .......... .......... .......... .......... .......... 1% 29.8M 40s
2550K .......... .......... .......... .......... .......... 1% 58.4M 39s
2600K .......... .......... .......... .......... .......... 1% 21.6M 38s
2650K .......... .......... .......... .......... .......... 1% 21.9M 38s
2700K .......... .......... .......... .......... .......... 1% 32.0M 37s
2750K .......... .......... .......... .......... .......... 1% 1.93M 38s
2800K .......... .......... .......... .......... .......... 1% 50.3M 37s
2850K .......... .......... .......... .......... .......... 1% 21.5M 37s
2900K .......... .......... .......... .......... .......... 1% 48.3M 36s
2950K .......... .......... .......... .......... .......... 1% 42.8M 36s
3000K .......... .......... .......... .......... .......... 1% 27.7M 35s
3050K .......... .......... .......... .......... .......... 1% 123M 35s
3100K .......... .......... .......... .......... .......... 1% 13.8M 34s
3150K .......... .......... .......... .......... .......... 1% 22.5M 34s
3200K .......... .......... .......... .......... .......... 1% 189M 33s
3250K .......... .......... .......... .......... .......... 1% 46.0M 33s
3300K .......... .......... .......... .......... .......... 2% 22.5M 32s
3350K .......... .......... .......... .......... .......... 2% 30.3M 32s
3400K .......... .......... .......... .......... .......... 2% 41.9M 32s
3450K .......... .......... .......... .......... .......... 2% 36.1M 31s
3500K .......... .......... .......... .......... .......... 2% 103M 31s
3550K .......... .......... .......... .......... .......... 2% 13.1M 30s
3600K .......... .......... .......... .......... .......... 2% 44.0M 30s
3650K .......... .......... .......... .......... .......... 2% 43.4M 30s
3700K .......... .......... .......... .......... .......... 2% 20.4M 29s
3750K .......... .......... .......... .......... .......... 2% 40.8M 29s
3800K .......... .......... .......... .......... .......... 2% 34.0M 29s
3850K .......... .......... .......... .......... .......... 2% 21.9M 28s
3900K .......... .......... .......... .......... .......... 2% 76.5M 28s
3950K .......... .......... .......... .......... .......... 2% 24.7M 28s
4000K .......... .......... .......... .......... .......... 2% 47.5M 28s
4050K .......... .......... .......... .......... .......... 2% 20.4M 27s
4100K .......... .......... .......... .......... .......... 2% 161M 27s
4150K .......... .......... .......... .......... .......... 2% 18.5M 27s
4200K .......... .......... .......... .......... .......... 2% 105M 26s
4250K .......... .......... .......... .......... .......... 2% 2.18M 27s
4300K .......... .......... .......... .......... .......... 2% 8.37M 27s
4350K .......... .......... .......... .......... .......... 2% 155M 27s
4400K .......... .......... .......... .......... .......... 2% 27.0M 26s
4450K .......... .......... .......... .......... .......... 2% 32.7M 26s
4500K .......... .......... .......... .......... .......... 2% 68.8M 26s
4550K .......... .......... .......... .......... .......... 2% 14.5M 26s
4600K .......... .......... .......... .......... .......... 2% 120M 25s
4650K .......... .......... .......... .......... .......... 2% 22.6M 25s
4700K .......... .......... .......... .......... .......... 2% 30.9M 25s
4750K .......... .......... .......... .......... .......... 2% 27.7M 25s
4800K .......... .......... .......... .......... .......... 2% 73.8M 24s
4850K .......... .......... .......... .......... .......... 2% 24.3M 24s
4900K .......... .......... .......... .......... .......... 2% 99.2M 24s
4950K .......... .......... .......... .......... .......... 3% 28.0M 24s
5000K .......... .......... .......... .......... .......... 3% 49.4M 24s
5050K .......... .......... .......... .......... .......... 3% 114M 23s
5100K .......... .......... .......... .......... .......... 3% 22.3M 23s
5150K .......... .......... .......... .......... .......... 3% 27.0M 23s
5200K .......... .......... .......... .......... .......... 3% 13.5M 23s
5250K .......... .......... .......... .......... .......... 3% 81.3M 23s
5300K .......... .......... .......... .......... .......... 3% 30.1M 23s
5350K .......... .......... .......... .......... .......... 3% 52.0M 22s
5400K .......... .......... .......... .......... .......... 3% 23.5M 22s
5450K .......... .......... .......... .......... .......... 3% 20.9M 22s
5500K .......... .......... .......... .......... .......... 3% 117M 22s
5550K .......... .......... .......... .......... .......... 3% 22.7M 22s
5600K .......... .......... .......... .......... .......... 3% 20.8M 22s
5650K .......... .......... .......... .......... .......... 3% 25.7M 22s
5700K .......... .......... .......... .......... .......... 3% 31.2M 21s
5750K .......... .......... .......... .......... .......... 3% 103M 21s
5800K .......... .......... .......... .......... .......... 3% 1.88M 22s
5850K .......... .......... .......... .......... .......... 3% 21.6M 22s
5900K .......... .......... .......... .......... .......... 3% 27.9M 21s
5950K .......... .......... .......... .......... .......... 3% 20.9M 21s
6000K .......... .......... .......... .......... .......... 3% 94.0M 21s
6050K .......... .......... .......... .......... .......... 3% 31.1M 21s
6100K .......... .......... .......... .......... .......... 3% 18.2M 21s
6150K .......... .......... .......... .......... .......... 3% 20.6M 21s
6200K .......... .......... .......... .......... .......... 3% 34.0M 21s
6250K .......... .......... .......... .......... .......... 3% 54.4M 20s
6300K .......... .......... .......... .......... .......... 3% 14.2M 20s
6350K .......... .......... .......... .......... .......... 3% 42.1M 20s
6400K .......... .......... .......... .......... .......... 3% 22.6M 20s
6450K .......... .......... .......... .......... .......... 3% 135M 20s
6500K .......... .......... .......... .......... .......... 3% 21.5M 20s
6550K .......... .......... .......... .......... .......... 3% 22.6M 20s
6600K .......... .......... .......... .......... .......... 3% 30.4M 20s
6650K .......... .......... .......... .......... .......... 4% 30.8M 20s
6700K .......... .......... .......... .......... .......... 4% 61.1M 19s
6750K .......... .......... .......... .......... .......... 4% 32.9M 19s
6800K .......... .......... .......... .......... .......... 4% 32.4M 19s
6850K .......... .......... .......... .......... .......... 4% 21.8M 19s
6900K .......... .......... .......... .......... .......... 4% 60.3M 19s
6950K .......... .......... .......... .......... .......... 4% 99.1M 19s
7000K .......... .......... .......... .......... .......... 4% 63.6M 19s
7050K .......... .......... .......... .......... .......... 4% 59.7M 19s
7100K .......... .......... .......... .......... .......... 4% 21.8M 19s
7150K .......... .......... .......... .......... .......... 4% 11.5M 19s
7200K .......... .......... .......... .......... .......... 4% 103M 18s
7250K .......... .......... .......... .......... .......... 4% 155M 18s
7300K .......... .......... .......... .......... .......... 4% 2.34M 19s
7350K .......... .......... .......... .......... .......... 4% 9.40M 19s
7400K .......... .......... .......... .......... .......... 4% 88.7M 18s
7450K .......... .......... .......... .......... .......... 4% 14.5M 18s
7500K .......... .......... .......... .......... .......... 4% 82.8M 18s
7550K .......... .......... .......... .......... .......... 4% 38.5M 18s
7600K .......... .......... .......... .......... .......... 4% 71.5M 18s
7650K .......... .......... .......... .......... .......... 4% 16.4M 18s
7700K .......... .......... .......... .......... .......... 4% 15.8M 18s
7750K .......... .......... .......... .......... .......... 4% 91.1M 18s
7800K .......... .......... .......... .......... .......... 4% 13.1M 18s
7850K .......... .......... .......... .......... .......... 4% 87.1M 18s
7900K .......... .......... .......... .......... .......... 4% 50.4M 18s
7950K .......... .......... .......... .......... .......... 4% 21.3M 18s
8000K .......... .......... .......... .......... .......... 4% 23.4M 17s
8050K .......... .......... .......... .......... .......... 4% 22.5M 17s
8100K .......... .......... .......... .......... .......... 4% 42.8M 17s
8150K .......... .......... .......... .......... .......... 4% 47.4M 17s
8200K .......... .......... .......... .......... .......... 4% 26.0M 17s
8250K .......... .......... .......... .......... .......... 4% 36.7M 17s
8300K .......... .......... .......... .......... .......... 5% 101M 17s
8350K .......... .......... .......... .......... .......... 5% 16.6M 17s
8400K .......... .......... .......... .......... .......... 5% 81.7M 17s
8450K .......... .......... .......... .......... .......... 5% 65.8M 17s
8500K .......... .......... .......... .......... .......... 5% 44.3M 17s
8550K .......... .......... .......... .......... .......... 5% 133M 17s
8600K .......... .......... .......... .......... .......... 5% 50.0M 16s
8650K .......... .......... .......... .......... .......... 5% 19.4M 16s
8700K .......... .......... .......... .......... .......... 5% 27.6M 16s
8750K .......... .......... .......... .......... .......... 5% 20.4M 16s
8800K .......... .......... .......... .......... .......... 5% 79.5M 16s
8850K .......... .......... .......... .......... .......... 5% 2.03M 17s
8900K .......... .......... .......... .......... .......... 5% 17.4M 16s
8950K .......... .......... .......... .......... .......... 5% 12.0M 16s
9000K .......... .......... .......... .......... .......... 5% 83.2M 16s
9050K .......... .......... .......... .......... .......... 5% 51.4M 16s
9100K .......... .......... .......... .......... .......... 5% 51.7M 16s
9150K .......... .......... .......... .......... .......... 5% 17.2M 16s
9200K .......... .......... .......... .......... .......... 5% 30.0M 16s
9250K .......... .......... .......... .......... .......... 5% 60.3M 16s
9300K .......... .......... .......... .......... .......... 5% 48.9M 16s
9350K .......... .......... .......... .......... .......... 5% 13.0M 16s
9400K .......... .......... .......... .......... .......... 5% 23.0M 16s
9450K .......... .......... .......... .......... .......... 5% 31.2M 16s
9500K .......... .......... .......... .......... .......... 5% 85.2M 16s
9550K .......... .......... .......... .......... .......... 5% 24.7M 16s
9600K .......... .......... .......... .......... .......... 5% 20.9M 16s
9650K .......... .......... .......... .......... .......... 5% 23.1M 16s
9700K .......... .......... .......... .......... .......... 5% 59.1M 15s
9750K .......... .......... .......... .......... .......... 5% 27.9M 15s
9800K .......... .......... .......... .......... .......... 5% 14.5M 15s
9850K .......... .......... .......... .......... .......... 5% 72.8M 15s
9900K .......... .......... .......... .......... .......... 5% 26.2M 15s
9950K .......... .......... .......... .......... .......... 6% 97.1M 15s
10000K .......... .......... .......... .......... .......... 6% 13.8M 15s
10050K .......... .......... .......... .......... .......... 6% 54.0M 15s
10100K .......... .......... .......... .......... .......... 6% 33.4M 15s
10150K .......... .......... .......... .......... .......... 6% 23.1M 15s
10200K .......... .......... .......... .......... .......... 6% 43.3M 15s
10250K .......... .......... .......... .......... .......... 6% 21.2M 15s
10300K .......... .......... .......... .......... .......... 6% 26.7M 15s
10350K .......... .......... .......... .......... .......... 6% 3.01M 15s
10400K .......... .......... .......... .......... .......... 6% 19.5M 15s
10450K .......... .......... .......... .......... .......... 6% 11.5M 15s
10500K .......... .......... .......... .......... .......... 6% 14.7M 15s
10550K .......... .......... .......... .......... .......... 6% 24.1M 15s
10600K .......... .......... .......... .......... .......... 6% 31.5M 15s
10650K .......... .......... .......... .......... .......... 6% 111M 15s
10700K .......... .......... .......... .......... .......... 6% 24.4M 15s
10750K .......... .......... .......... .......... .......... 6% 29.9M 15s
10800K .......... .......... .......... .......... .......... 6% 45.4M 15s
10850K .......... .......... .......... .......... .......... 6% 11.4M 15s
10900K .......... .......... .......... .......... .......... 6% 47.3M 15s
10950K .......... .......... .......... .......... .......... 6% 49.5M 14s
11000K .......... .......... .......... .......... .......... 6% 23.1M 14s
11050K .......... .......... .......... .......... .......... 6% 29.2M 14s
11100K .......... .......... .......... .......... .......... 6% 67.0M 14s
11150K .......... .......... .......... .......... .......... 6% 24.0M 14s
11200K .......... .......... .......... .......... .......... 6% 22.9M 14s
11250K .......... .......... .......... .......... .......... 6% 33.2M 14s
11300K .......... .......... .......... .......... .......... 6% 29.7M 14s
11350K .......... .......... .......... .......... .......... 6% 26.2M 14s
11400K .......... .......... .......... .......... .......... 6% 109M 14s
11450K .......... .......... .......... .......... .......... 6% 32.4M 14s
11500K .......... .......... .......... .......... .......... 6% 12.9M 14s
11550K .......... .......... .......... .......... .......... 6% 59.2M 14s
11600K .......... .......... .......... .......... .......... 6% 71.9M 14s
11650K .......... .......... .......... .......... .......... 7% 40.7M 14s
11700K .......... .......... .......... .......... .......... 7% 21.6M 14s
11750K .......... .......... .......... .......... .......... 7% 23.9M 14s
11800K .......... .......... .......... .......... .......... 7% 28.7M 14s
11850K .......... .......... .......... .......... .......... 7% 24.0M 14s
11900K .......... .......... .......... .......... .......... 7% 2.81M 14s
11950K .......... .......... .......... .......... .......... 7% 12.3M 14s
12000K .......... .......... .......... .......... .......... 7% 16.6M 14s
12050K .......... .......... .......... .......... .......... 7% 28.9M 14s
12100K .......... .......... .......... .......... .......... 7% 23.6M 14s
12150K .......... .......... .......... .......... .......... 7% 46.2M 14s
12200K .......... .......... .......... .......... .......... 7% 20.0M 14s
12250K .......... .......... .......... .......... .......... 7% 48.5M 14s
12300K .......... .......... .......... .......... .......... 7% 70.2M 14s
12350K .......... .......... .......... .......... .......... 7% 43.2M 14s
12400K .......... .......... .......... .......... .......... 7% 10.2M 14s
12450K .......... .......... .......... .......... .......... 7% 45.6M 14s
12500K .......... .......... .......... .......... .......... 7% 112M 13s
12550K .......... .......... .......... .......... .......... 7% 15.1M 13s
12600K .......... .......... .......... .......... .......... 7% 30.3M 13s
12650K .......... .......... .......... .......... .......... 7% 19.0M 13s
12700K .......... .......... .......... .......... .......... 7% 30.7M 13s
12750K .......... .......... .......... .......... .......... 7% 73.2M 13s
12800K .......... .......... .......... .......... .......... 7% 20.8M 13s
12850K .......... .......... .......... .......... .......... 7% 34.2M 13s
12900K .......... .......... .......... .......... .......... 7% 30.2M 13s
12950K .......... .......... .......... .......... .......... 7% 33.7M 13s
13000K .......... .......... .......... .......... .......... 7% 66.8M 13s
13050K .......... .......... .......... .......... .......... 7% 13.1M 13s
13100K .......... .......... .......... .......... .......... 7% 56.5M 13s
13150K .......... .......... .......... .......... .......... 7% 53.4M 13s
13200K .......... .......... .......... .......... .......... 7% 126M 13s
13250K .......... .......... .......... .......... .......... 7% 28.1M 13s
13300K .......... .......... .......... .......... .......... 8% 16.7M 13s
13350K .......... .......... .......... .......... .......... 8% 23.8M 13s
13400K .......... .......... .......... .......... .......... 8% 3.20M 13s
13450K .......... .......... .......... .......... .......... 8% 13.0M 13s
13500K .......... .......... .......... .......... .......... 8% 10.5M 13s
13550K .......... .......... .......... .......... .......... 8% 24.9M 13s
13600K .......... .......... .......... .......... .......... 8% 49.6M 13s
13650K .......... .......... .......... .......... .......... 8% 14.6M 13s
13700K .......... .......... .......... .......... .......... 8% 148M 13s
13750K .......... .......... .......... .......... .......... 8% 15.5M 13s
13800K .......... .......... .......... .......... .......... 8% 21.3M 13s
13850K .......... .......... .......... .......... .......... 8% 6.80M 13s
13900K .......... .......... .......... .......... .......... 8% 115M 13s
13950K .......... .......... .......... .......... .......... 8% 78.6M 13s
14000K .......... .......... .......... .......... .......... 8% 73.9M 13s
14050K .......... .......... .......... .......... .......... 8% 64.7M 13s
14100K .......... .......... .......... .......... .......... 8% 39.2M 13s
14150K .......... .......... .......... .......... .......... 8% 60.3M 13s
14200K .......... .......... .......... .......... .......... 8% 34.7M 13s
14250K .......... .......... .......... .......... .......... 8% 38.8M 13s
14300K .......... .......... .......... .......... .......... 8% 15.7M 13s
14350K .......... .......... .......... .......... .......... 8% 32.6M 12s
14400K .......... .......... .......... .......... .......... 8% 45.4M 12s
14450K .......... .......... .......... .......... .......... 8% 28.2M 12s
14500K .......... .......... .......... .......... .......... 8% 47.0M 12s
14550K .......... .......... .......... .......... .......... 8% 48.2M 12s
14600K .......... .......... .......... .......... .......... 8% 16.8M 12s
14650K .......... .......... .......... .......... .......... 8% 40.9M 12s
14700K .......... .......... .......... .......... .......... 8% 33.6M 12s
14750K .......... .......... .......... .......... .......... 8% 21.8M 12s
14800K .......... .......... .......... .......... .......... 8% 21.1M 12s
14850K .......... .......... .......... .......... .......... 8% 81.7M 12s
14900K .......... .......... .......... .......... .......... 8% 19.4M 12s
14950K .......... .......... .......... .......... .......... 9% 2.91M 12s
15000K .......... .......... .......... .......... .......... 9% 95.3M 12s
15050K .......... .......... .......... .......... .......... 9% 4.33M 12s
15100K .......... .......... .......... .......... .......... 9% 166M 12s
15150K .......... .......... .......... .......... .......... 9% 138M 12s
15200K .......... .......... .......... .......... .......... 9% 164M 12s
15250K .......... .......... .......... .......... .......... 9% 63.8M 12s
15300K .......... .......... .......... .......... .......... 9% 49.4M 12s
15350K .......... .......... .......... .......... .......... 9% 20.9M 12s
15400K .......... .......... .......... .......... .......... 9% 17.4M 12s
15450K .......... .......... .......... .......... .......... 9% 24.1M 12s
15500K .......... .......... .......... .......... .......... 9% 50.1M 12s
15550K .......... .......... .......... .......... .......... 9% 31.5M 12s
15600K .......... .......... .......... .......... .......... 9% 14.7M 12s
15650K .......... .......... .......... .......... .......... 9% 78.0M 12s
15700K .......... .......... .......... .......... .......... 9% 48.0M 12s
15750K .......... .......... .......... .......... .......... 9% 27.1M 12s
15800K .......... .......... .......... .......... .......... 9% 41.9M 12s
15850K .......... .......... .......... .......... .......... 9% 22.0M 12s
15900K .......... .......... .......... .......... .......... 9% 18.9M 12s
15950K .......... .......... .......... .......... .......... 9% 33.0M 12s
*** WARNING: skipped 204162 bytes of output ***
150200K .......... .......... .......... .......... .......... 90% 22.1M 1s
150250K .......... .......... .......... .......... .......... 90% 19.7M 1s
150300K .......... .......... .......... .......... .......... 90% 20.7M 1s
150350K .......... .......... .......... .......... .......... 90% 18.6M 1s
150400K .......... .......... .......... .......... .......... 90% 19.5M 1s
150450K .......... .......... .......... .......... .......... 90% 46.1M 1s
150500K .......... .......... .......... .......... .......... 90% 19.1M 1s
150550K .......... .......... .......... .......... .......... 90% 12.4M 1s
150600K .......... .......... .......... .......... .......... 90% 16.3M 1s
150650K .......... .......... .......... .......... .......... 90% 122M 1s
150700K .......... .......... .......... .......... .......... 90% 16.8M 1s
150750K .......... .......... .......... .......... .......... 90% 47.7M 1s
150800K .......... .......... .......... .......... .......... 90% 44.2M 1s
150850K .......... .......... .......... .......... .......... 90% 21.8M 1s
150900K .......... .......... .......... .......... .......... 90% 73.6M 1s
150950K .......... .......... .......... .......... .......... 90% 18.4M 1s
151000K .......... .......... .......... .......... .......... 90% 12.4M 1s
151050K .......... .......... .......... .......... .......... 90% 18.7M 1s
151100K .......... .......... .......... .......... .......... 90% 12.7M 1s
151150K .......... .......... .......... .......... .......... 90% 23.4M 1s
151200K .......... .......... .......... .......... .......... 90% 12.2M 1s
151250K .......... .......... .......... .......... .......... 90% 16.0M 1s
151300K .......... .......... .......... .......... .......... 90% 19.5M 1s
151350K .......... .......... .......... .......... .......... 90% 64.8M 1s
151400K .......... .......... .......... .......... .......... 90% 15.9M 1s
151450K .......... .......... .......... .......... .......... 90% 37.7M 1s
151500K .......... .......... .......... .......... .......... 91% 23.2M 1s
151550K .......... .......... .......... .......... .......... 91% 13.7M 1s
151600K .......... .......... .......... .......... .......... 91% 24.5M 1s
151650K .......... .......... .......... .......... .......... 91% 20.1M 1s
151700K .......... .......... .......... .......... .......... 91% 30.8M 1s
151750K .......... .......... .......... .......... .......... 91% 18.0M 1s
151800K .......... .......... .......... .......... .......... 91% 26.8M 1s
151850K .......... .......... .......... .......... .......... 91% 78.1M 1s
151900K .......... .......... .......... .......... .......... 91% 18.2M 1s
151950K .......... .......... .......... .......... .......... 91% 14.7M 1s
152000K .......... .......... .......... .......... .......... 91% 119M 1s
152050K .......... .......... .......... .......... .......... 91% 19.9M 1s
152100K .......... .......... .......... .......... .......... 91% 11.7M 1s
152150K .......... .......... .......... .......... .......... 91% 17.9M 1s
152200K .......... .......... .......... .......... .......... 91% 13.1M 1s
152250K .......... .......... .......... .......... .......... 91% 33.8M 1s
152300K .......... .......... .......... .......... .......... 91% 106M 1s
152350K .......... .......... .......... .......... .......... 91% 43.1M 1s
152400K .......... .......... .......... .......... .......... 91% 24.5M 1s
152450K .......... .......... .......... .......... .......... 91% 18.5M 1s
152500K .......... .......... .......... .......... .......... 91% 12.4M 1s
152550K .......... .......... .......... .......... .......... 91% 104M 1s
152600K .......... .......... .......... .......... .......... 91% 22.3M 1s
152650K .......... .......... .......... .......... .......... 91% 8.15M 1s
152700K .......... .......... .......... .......... .......... 91% 14.8M 1s
152750K .......... .......... .......... .......... .......... 91% 47.3M 1s
152800K .......... .......... .......... .......... .......... 91% 16.7M 1s
152850K .......... .......... .......... .......... .......... 91% 26.9M 1s
152900K .......... .......... .......... .......... .......... 91% 16.6M 1s
152950K .......... .......... .......... .......... .......... 91% 25.4M 1s
153000K .......... .......... .......... .......... .......... 91% 53.0M 1s
153050K .......... .......... .......... .......... .......... 91% 46.3M 1s
153100K .......... .......... .......... .......... .......... 91% 8.79M 1s
153150K .......... .......... .......... .......... .......... 92% 20.4M 1s
153200K .......... .......... .......... .......... .......... 92% 34.2M 1s
153250K .......... .......... .......... .......... .......... 92% 89.4M 1s
153300K .......... .......... .......... .......... .......... 92% 17.4M 1s
153350K .......... .......... .......... .......... .......... 92% 27.5M 1s
153400K .......... .......... .......... .......... .......... 92% 15.6M 1s
153450K .......... .......... .......... .......... .......... 92% 88.0M 1s
153500K .......... .......... .......... .......... .......... 92% 14.8M 1s
153550K .......... .......... .......... .......... .......... 92% 22.4M 1s
153600K .......... .......... .......... .......... .......... 92% 13.7M 1s
153650K .......... .......... .......... .......... .......... 92% 15.1M 1s
153700K .......... .......... .......... .......... .......... 92% 91.5M 1s
153750K .......... .......... .......... .......... .......... 92% 13.6M 1s
153800K .......... .......... .......... .......... .......... 92% 27.9M 1s
153850K .......... .......... .......... .......... .......... 92% 55.8M 1s
153900K .......... .......... .......... .......... .......... 92% 32.6M 1s
153950K .......... .......... .......... .......... .......... 92% 59.6M 1s
154000K .......... .......... .......... .......... .......... 92% 16.3M 1s
154050K .......... .......... .......... .......... .......... 92% 7.24M 1s
154100K .......... .......... .......... .......... .......... 92% 102M 1s
154150K .......... .......... .......... .......... .......... 92% 114M 1s
154200K .......... .......... .......... .......... .......... 92% 9.56M 1s
154250K .......... .......... .......... .......... .......... 92% 9.85M 1s
154300K .......... .......... .......... .......... .......... 92% 23.1M 1s
154350K .......... .......... .......... .......... .......... 92% 63.5M 1s
154400K .......... .......... .......... .......... .......... 92% 62.1M 1s
154450K .......... .......... .......... .......... .......... 92% 15.5M 1s
154500K .......... .......... .......... .......... .......... 92% 21.2M 1s
154550K .......... .......... .......... .......... .......... 92% 46.8M 1s
154600K .......... .......... .......... .......... .......... 92% 8.33M 1s
154650K .......... .......... .......... .......... .......... 92% 77.7M 1s
154700K .......... .......... .......... .......... .......... 92% 24.6M 1s
154750K .......... .......... .......... .......... .......... 92% 35.6M 1s
154800K .......... .......... .......... .......... .......... 93% 20.2M 1s
154850K .......... .......... .......... .......... .......... 93% 61.5M 1s
154900K .......... .......... .......... .......... .......... 93% 23.9M 1s
154950K .......... .......... .......... .......... .......... 93% 15.1M 1s
155000K .......... .......... .......... .......... .......... 93% 16.9M 1s
155050K .......... .......... .......... .......... .......... 93% 21.0M 1s
155100K .......... .......... .......... .......... .......... 93% 135M 1s
155150K .......... .......... .......... .......... .......... 93% 11.1M 1s
155200K .......... .......... .......... .......... .......... 93% 17.3M 1s
155250K .......... .......... .......... .......... .......... 93% 14.6M 1s
155300K .......... .......... .......... .......... .......... 93% 35.1M 1s
155350K .......... .......... .......... .......... .......... 93% 35.9M 1s
155400K .......... .......... .......... .......... .......... 93% 51.8M 1s
155450K .......... .......... .......... .......... .......... 93% 27.8M 1s
155500K .......... .......... .......... .......... .......... 93% 20.5M 1s
155550K .......... .......... .......... .......... .......... 93% 56.7M 1s
155600K .......... .......... .......... .......... .......... 93% 13.0M 1s
155650K .......... .......... .......... .......... .......... 93% 24.2M 1s
155700K .......... .......... .......... .......... .......... 93% 14.8M 1s
155750K .......... .......... .......... .......... .......... 93% 7.81M 1s
155800K .......... .......... .......... .......... .......... 93% 64.4M 1s
155850K .......... .......... .......... .......... .......... 93% 13.4M 1s
155900K .......... .......... .......... .......... .......... 93% 19.7M 1s
155950K .......... .......... .......... .......... .......... 93% 19.6M 1s
156000K .......... .......... .......... .......... .......... 93% 30.1M 1s
156050K .......... .......... .......... .......... .......... 93% 50.1M 1s
156100K .......... .......... .......... .......... .......... 93% 27.8M 1s
156150K .......... .......... .......... .......... .......... 93% 10.5M 1s
156200K .......... .......... .......... .......... .......... 93% 18.4M 1s
156250K .......... .......... .......... .......... .......... 93% 95.9M 1s
156300K .......... .......... .......... .......... .......... 93% 22.7M 1s
156350K .......... .......... .......... .......... .......... 93% 22.8M 1s
156400K .......... .......... .......... .......... .......... 93% 22.0M 1s
156450K .......... .......... .......... .......... .......... 93% 20.8M 0s
156500K .......... .......... .......... .......... .......... 94% 62.2M 0s
156550K .......... .......... .......... .......... .......... 94% 16.5M 0s
156600K .......... .......... .......... .......... .......... 94% 23.0M 0s
156650K .......... .......... .......... .......... .......... 94% 16.9M 0s
156700K .......... .......... .......... .......... .......... 94% 14.9M 0s
156750K .......... .......... .......... .......... .......... 94% 99.6M 0s
156800K .......... .......... .......... .......... .......... 94% 11.9M 0s
156850K .......... .......... .......... .......... .......... 94% 30.2M 0s
156900K .......... .......... .......... .......... .......... 94% 22.3M 0s
156950K .......... .......... .......... .......... .......... 94% 92.4M 0s
157000K .......... .......... .......... .......... .......... 94% 58.2M 0s
157050K .......... .......... .......... .......... .......... 94% 17.7M 0s
157100K .......... .......... .......... .......... .......... 94% 7.60M 0s
157150K .......... .......... .......... .......... .......... 94% 99.6M 0s
157200K .......... .......... .......... .......... .......... 94% 174M 0s
157250K .......... .......... .......... .......... .......... 94% 10.8M 0s
157300K .......... .......... .......... .......... .......... 94% 10.9M 0s
157350K .......... .......... .......... .......... .......... 94% 12.1M 0s
157400K .......... .......... .......... .......... .......... 94% 21.7M 0s
157450K .......... .......... .......... .......... .......... 94% 107M 0s
157500K .......... .......... .......... .......... .......... 94% 20.5M 0s
157550K .......... .......... .......... .......... .......... 94% 24.6M 0s
157600K .......... .......... .......... .......... .......... 94% 20.8M 0s
157650K .......... .......... .......... .......... .......... 94% 60.3M 0s
157700K .......... .......... .......... .......... .......... 94% 13.0M 0s
157750K .......... .......... .......... .......... .......... 94% 19.2M 0s
157800K .......... .......... .......... .......... .......... 94% 21.7M 0s
157850K .......... .......... .......... .......... .......... 94% 19.5M 0s
157900K .......... .......... .......... .......... .......... 94% 60.4M 0s
157950K .......... .......... .......... .......... .......... 94% 19.3M 0s
158000K .......... .......... .......... .......... .......... 94% 25.5M 0s
158050K .......... .......... .......... .......... .......... 94% 16.8M 0s
158100K .......... .......... .......... .......... .......... 94% 28.7M 0s
158150K .......... .......... .......... .......... .......... 95% 55.7M 0s
158200K .......... .......... .......... .......... .......... 95% 15.0M 0s
158250K .......... .......... .......... .......... .......... 95% 16.3M 0s
158300K .......... .......... .......... .......... .......... 95% 56.1M 0s
158350K .......... .......... .......... .......... .......... 95% 9.03M 0s
158400K .......... .......... .......... .......... .......... 95% 130M 0s
158450K .......... .......... .......... .......... .......... 95% 31.4M 0s
158500K .......... .......... .......... .......... .......... 95% 53.9M 0s
158550K .......... .......... .......... .......... .......... 95% 15.8M 0s
158600K .......... .......... .......... .......... .......... 95% 165M 0s
158650K .......... .......... .......... .......... .......... 95% 11.3M 0s
158700K .......... .......... .......... .......... .......... 95% 27.8M 0s
158750K .......... .......... .......... .......... .......... 95% 21.9M 0s
158800K .......... .......... .......... .......... .......... 95% 7.42M 0s
158850K .......... .......... .......... .......... .......... 95% 55.6M 0s
158900K .......... .......... .......... .......... .......... 95% 11.7M 0s
158950K .......... .......... .......... .......... .......... 95% 20.0M 0s
159000K .......... .......... .......... .......... .......... 95% 23.1M 0s
159050K .......... .......... .......... .......... .......... 95% 55.3M 0s
159100K .......... .......... .......... .......... .......... 95% 27.0M 0s
159150K .......... .......... .......... .......... .......... 95% 18.9M 0s
159200K .......... .......... .......... .......... .......... 95% 14.1M 0s
159250K .......... .......... .......... .......... .......... 95% 17.9M 0s
159300K .......... .......... .......... .......... .......... 95% 94.0M 0s
159350K .......... .......... .......... .......... .......... 95% 18.8M 0s
159400K .......... .......... .......... .......... .......... 95% 22.0M 0s
159450K .......... .......... .......... .......... .......... 95% 24.5M 0s
159500K .......... .......... .......... .......... .......... 95% 19.1M 0s
159550K .......... .......... .......... .......... .......... 95% 65.9M 0s
159600K .......... .......... .......... .......... .......... 95% 11.8M 0s
159650K .......... .......... .......... .......... .......... 95% 25.2M 0s
159700K .......... .......... .......... .......... .......... 95% 15.3M 0s
159750K .......... .......... .......... .......... .......... 95% 76.7M 0s
159800K .......... .......... .......... .......... .......... 96% 15.6M 0s
159850K .......... .......... .......... .......... .......... 96% 11.4M 0s
159900K .......... .......... .......... .......... .......... 96% 95.4M 0s
159950K .......... .......... .......... .......... .......... 96% 16.4M 0s
160000K .......... .......... .......... .......... .......... 96% 65.6M 0s
160050K .......... .......... .......... .......... .......... 96% 41.2M 0s
160100K .......... .......... .......... .......... .......... 96% 21.1M 0s
160150K .......... .......... .......... .......... .......... 96% 15.5M 0s
160200K .......... .......... .......... .......... .......... 96% 21.6M 0s
160250K .......... .......... .......... .......... .......... 96% 41.7M 0s
160300K .......... .......... .......... .......... .......... 96% 12.8M 0s
160350K .......... .......... .......... .......... .......... 96% 13.5M 0s
160400K .......... .......... .......... .......... .......... 96% 11.5M 0s
160450K .......... .......... .......... .......... .......... 96% 65.7M 0s
160500K .......... .......... .......... .......... .......... 96% 19.1M 0s
160550K .......... .......... .......... .......... .......... 96% 19.5M 0s
160600K .......... .......... .......... .......... .......... 96% 26.1M 0s
160650K .......... .......... .......... .......... .......... 96% 19.5M 0s
160700K .......... .......... .......... .......... .......... 96% 85.4M 0s
160750K .......... .......... .......... .......... .......... 96% 14.8M 0s
160800K .......... .......... .......... .......... .......... 96% 16.2M 0s
160850K .......... .......... .......... .......... .......... 96% 27.5M 0s
160900K .......... .......... .......... .......... .......... 96% 13.6M 0s
160950K .......... .......... .......... .......... .......... 96% 94.8M 0s
161000K .......... .......... .......... .......... .......... 96% 17.6M 0s
161050K .......... .......... .......... .......... .......... 96% 20.0M 0s
161100K .......... .......... .......... .......... .......... 96% 24.3M 0s
161150K .......... .......... .......... .......... .......... 96% 31.8M 0s
161200K .......... .......... .......... .......... .......... 96% 19.8M 0s
161250K .......... .......... .......... .......... .......... 96% 19.6M 0s
161300K .......... .......... .......... .......... .......... 96% 18.7M 0s
161350K .......... .......... .......... .......... .......... 96% 12.4M 0s
161400K .......... .......... .......... .......... .......... 96% 21.7M 0s
161450K .......... .......... .......... .......... .......... 96% 28.8M 0s
161500K .......... .......... .......... .......... .......... 97% 20.6M 0s
161550K .......... .......... .......... .......... .......... 97% 28.5M 0s
161600K .......... .......... .......... .......... .......... 97% 18.5M 0s
161650K .......... .......... .......... .......... .......... 97% 89.5M 0s
161700K .......... .......... .......... .......... .......... 97% 13.2M 0s
161750K .......... .......... .......... .......... .......... 97% 58.5M 0s
161800K .......... .......... .......... .......... .......... 97% 12.4M 0s
161850K .......... .......... .......... .......... .......... 97% 16.3M 0s
161900K .......... .......... .......... .......... .......... 97% 35.4M 0s
161950K .......... .......... .......... .......... .......... 97% 9.31M 0s
162000K .......... .......... .......... .......... .......... 97% 14.8M 0s
162050K .......... .......... .......... .......... .......... 97% 32.5M 0s
162100K .......... .......... .......... .......... .......... 97% 41.4M 0s
162150K .......... .......... .......... .......... .......... 97% 21.4M 0s
162200K .......... .......... .......... .......... .......... 97% 38.4M 0s
162250K .......... .......... .......... .......... .......... 97% 14.5M 0s
162300K .......... .......... .......... .......... .......... 97% 10.7M 0s
162350K .......... .......... .......... .......... .......... 97% 32.5M 0s
162400K .......... .......... .......... .......... .......... 97% 48.7M 0s
162450K .......... .......... .......... .......... .......... 97% 15.6M 0s
162500K .......... .......... .......... .......... .......... 97% 23.8M 0s
162550K .......... .......... .......... .......... .......... 97% 32.3M 0s
162600K .......... .......... .......... .......... .......... 97% 29.4M 0s
162650K .......... .......... .......... .......... .......... 97% 16.8M 0s
162700K .......... .......... .......... .......... .......... 97% 26.7M 0s
162750K .......... .......... .......... .......... .......... 97% 17.8M 0s
162800K .......... .......... .......... .......... .......... 97% 63.2M 0s
162850K .......... .......... .......... .......... .......... 97% 15.8M 0s
162900K .......... .......... .......... .......... .......... 97% 15.9M 0s
162950K .......... .......... .......... .......... .......... 97% 12.5M 0s
163000K .......... .......... .......... .......... .......... 97% 24.2M 0s
163050K .......... .......... .......... .......... .......... 97% 50.5M 0s
163100K .......... .......... .......... .......... .......... 97% 24.5M 0s
163150K .......... .......... .......... .......... .......... 98% 27.2M 0s
163200K .......... .......... .......... .......... .......... 98% 22.4M 0s
163250K .......... .......... .......... .......... .......... 98% 11.0M 0s
163300K .......... .......... .......... .......... .......... 98% 72.7M 0s
163350K .......... .......... .......... .......... .......... 98% 18.7M 0s
163400K .......... .......... .......... .......... .......... 98% 14.4M 0s
163450K .......... .......... .......... .......... .......... 98% 11.6M 0s
163500K .......... .......... .......... .......... .......... 98% 29.2M 0s
163550K .......... .......... .......... .......... .......... 98% 14.4M 0s
163600K .......... .......... .......... .......... .......... 98% 25.7M 0s
163650K .......... .......... .......... .......... .......... 98% 25.8M 0s
163700K .......... .......... .......... .......... .......... 98% 26.6M 0s
163750K .......... .......... .......... .......... .......... 98% 97.2M 0s
163800K .......... .......... .......... .......... .......... 98% 13.0M 0s
163850K .......... .......... .......... .......... .......... 98% 10.8M 0s
163900K .......... .......... .......... .......... .......... 98% 31.2M 0s
163950K .......... .......... .......... .......... .......... 98% 18.8M 0s
164000K .......... .......... .......... .......... .......... 98% 31.9M 0s
164050K .......... .......... .......... .......... .......... 98% 22.9M 0s
164100K .......... .......... .......... .......... .......... 98% 28.9M 0s
164150K .......... .......... .......... .......... .......... 98% 18.3M 0s
164200K .......... .......... .......... .......... .......... 98% 79.7M 0s
164250K .......... .......... .......... .......... .......... 98% 19.7M 0s
164300K .......... .......... .......... .......... .......... 98% 18.0M 0s
164350K .......... .......... .......... .......... .......... 98% 17.4M 0s
164400K .......... .......... .......... .......... .......... 98% 9.15M 0s
164450K .......... .......... .......... .......... .......... 98% 122M 0s
164500K .......... .......... .......... .......... .......... 98% 20.5M 0s
164550K .......... .......... .......... .......... .......... 98% 21.8M 0s
164600K .......... .......... .......... .......... .......... 98% 25.9M 0s
164650K .......... .......... .......... .......... .......... 98% 21.8M 0s
164700K .......... .......... .......... .......... .......... 98% 80.9M 0s
164750K .......... .......... .......... .......... .......... 98% 7.10M 0s
164800K .......... .......... .......... .......... .......... 99% 117M 0s
164850K .......... .......... .......... .......... .......... 99% 36.1M 0s
164900K .......... .......... .......... .......... .......... 99% 52.0M 0s
164950K .......... .......... .......... .......... .......... 99% 13.7M 0s
165000K .......... .......... .......... .......... .......... 99% 9.19M 0s
165050K .......... .......... .......... .......... .......... 99% 23.1M 0s
165100K .......... .......... .......... .......... .......... 99% 13.2M 0s
165150K .......... .......... .......... .......... .......... 99% 77.8M 0s
165200K .......... .......... .......... .......... .......... 99% 29.2M 0s
165250K .......... .......... .......... .......... .......... 99% 23.6M 0s
165300K .......... .......... .......... .......... .......... 99% 26.8M 0s
165350K .......... .......... .......... .......... .......... 99% 11.2M 0s
165400K .......... .......... .......... .......... .......... 99% 31.1M 0s
165450K .......... .......... .......... .......... .......... 99% 21.1M 0s
165500K .......... .......... .......... .......... .......... 99% 23.9M 0s
165550K .......... .......... .......... .......... .......... 99% 12.3M 0s
165600K .......... .......... .......... .......... .......... 99% 96.0M 0s
165650K .......... .......... .......... .......... .......... 99% 20.9M 0s
165700K .......... .......... .......... .......... .......... 99% 24.4M 0s
165750K .......... .......... .......... .......... .......... 99% 26.3M 0s
165800K .......... .......... .......... .......... .......... 99% 22.7M 0s
165850K .......... .......... .......... .......... .......... 99% 35.1M 0s
165900K .......... .......... .......... .......... .......... 99% 16.1M 0s
165950K .......... .......... .......... .......... .......... 99% 10.2M 0s
166000K .......... .......... .......... .......... .......... 99% 26.6M 0s
166050K .......... .......... .......... .......... .......... 99% 17.2M 0s
166100K .......... .......... .......... .......... .......... 99% 115M 0s
166150K .......... .......... .......... .......... .......... 99% 28.9M 0s
166200K .......... .......... .......... .......... .......... 99% 20.9M 0s
166250K .......... .......... .......... .......... .......... 99% 25.0M 0s
166300K .......... .......... .......... .......... .......... 99% 19.4M 0s
166350K .......... .......... .......... .......... .......... 99% 18.9M 0s
166400K .......... .......... .......... .......... .......... 99% 20.2M 0s
166450K .......... .......... .......... .......... .......... 99% 16.5M 0s
166500K .. 100% 3858G=8.3s
2021-01-18 14:38:37 (19.7 MB/s) - ‘cifar-10-python.tar.gz’ saved [170498071/170498071]
tar zxvf cifar-10-python.tar.gz
cifar-10-batches-py/
cifar-10-batches-py/data_batch_4
cifar-10-batches-py/readme.html
cifar-10-batches-py/test_batch
cifar-10-batches-py/data_batch_3
cifar-10-batches-py/batches.meta
cifar-10-batches-py/data_batch_2
cifar-10-batches-py/data_batch_5
cifar-10-batches-py/data_batch_1
ls -la cifar-10-batches-py
total 181884
drwxr-xr-x 2 2156 1103 4096 Jun 4 2009 .
drwxr-xr-x 1 root root 4096 Jan 18 14:38 ..
-rw-r--r-- 1 2156 1103 158 Mar 31 2009 batches.meta
-rw-r--r-- 1 2156 1103 31035704 Mar 31 2009 data_batch_1
-rw-r--r-- 1 2156 1103 31035320 Mar 31 2009 data_batch_2
-rw-r--r-- 1 2156 1103 31035999 Mar 31 2009 data_batch_3
-rw-r--r-- 1 2156 1103 31035696 Mar 31 2009 data_batch_4
-rw-r--r-- 1 2156 1103 31035623 Mar 31 2009 data_batch_5
-rw-r--r-- 1 2156 1103 88 Jun 4 2009 readme.html
-rw-r--r-- 1 2156 1103 31035526 Mar 31 2009 test_batch
Recall the classes are: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck
Here is the code to unpickle the batches.
Loaded in this way, each of the batch files contains a dictionary with the following elements:
- data - a 10000x3072 numpy array of uint8s. Each row of the array stores a 32x32 colour image. The first 1024 entries contain the red channel values, the next 1024 the green, and the final 1024 the blue. The image is stored in row-major order, so that the first 32 entries of the array are the red channel values of the first row of the image.
- labels - a list of 10000 numbers in the range 0-9. The number at index i indicates the label of the ith image in the array data.
def unpickle(file):
import pickle
with open(file, 'rb') as fo:
dict = pickle.load(fo, encoding='bytes')# for Python 3, add the following param: encoding='bytes'
return dict
dir = 'cifar-10-batches-py/'
batches = [unpickle(dir + 'data_batch_' + str(1+n)) for n in range(5)]
Now we need to reshape the data batches and concatenate the training batches into one big tensor.
import numpy as np
def decode(xy):
x_train = np.reshape(xy[b'data'], (10000, 3, 32, 32)).transpose(0, 2, 3, 1)
y_train = np.reshape(xy[b'labels'], (10000, 1))
return (x_train, y_train)
decoded = [decode(data) for data in batches]
x_train = np.concatenate([data[0] for data in decoded])
y_train = np.concatenate([data[1] for data in decoded])
(x_test, y_test) = decode(unpickle(dir + 'test_batch'))
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
x_train shape: (50000, 32, 32, 3)
50000 train samples
10000 test samples
Let's visualize some of the images:
import matplotlib.pyplot as plt
fig = plt.figure()
for i in range(36):
fig.add_subplot(6, 6, i+1)
plt.imshow(x_train[i])
display(fig)
Recall that we are getting a categorical output via softmax across 10 neurons, corresponding to the output categories.
So we want to reshape our target values (training labels) to be 1-hot encoded, and Keras can calculate categorical crossentropy between its output layer and the target:
import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Conv2D, MaxPooling2D
num_classes = 10
# Convert class vectors to binary class matrices.
y_train_1hot = keras.utils.to_categorical(y_train, num_classes)
y_test_1hot = keras.utils.to_categorical(y_test, num_classes)
Using TensorFlow backend.
Here's a simple convolutional net to get you started. It will get you to over 57% accuracy in 5 epochs.
As inspiration, with a suitable network and parameters, it's possible to get over 99% test accuracy, although you won't have time to get there in today's session on this hardware.
note: if your network is not learning anything at all -- meaning regardless of settings, you're seeing a loss that doesn't decrease and a validation accuracy that is 10% (i.e., random chance) -- then restart your cluster
model = Sequential()
model.add(Conv2D(32, (3, 3), padding='same', input_shape=x_train.shape[1:]))
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dense(num_classes))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer="adam", metrics=['accuracy'])
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
history = model.fit(x_train, y_train_1hot,
batch_size=64,
epochs=5,
validation_data=(x_test, y_test_1hot),
verbose=2)
WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /databricks/python/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Train on 50000 samples, validate on 10000 samples
Epoch 1/5
In this session, you probably won't have time to run each experiment for too many epochs ... but you can use this code to plot the training and validation losses:
fig, ax = plt.subplots()
fig.set_size_inches((5,5))
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
display(fig)