exjobbsOfCombientMix2021_04_pspnet_tuning_parallel(Python)

Import Notebook

Implementation of PSPNet with Hyperparameter Tuning in Parallel

William Anzén (Linkedin), Christian von Koch (Linkedin)

2021, Stockholm, Sweden

This project was supported by Combient Mix AB under the industrial supervision of Razesh Sainudiin and Max Fischer.

In this notebook, an implementation of PSPNet is presented which is an architecture which uses scene parsing and evaluates the images at different scales and finally combines the different results to form a final prediction. The architecture is evaluated against the Oxford-IIIT Pet Dataset whilst some hyperparameters of the model are tunead parallely through Spark Trials.

Importing the required packages.

import tensorflow as tf
import tensorflow_datasets as tfds
import matplotlib.pyplot as plt
from tensorflow.keras.layers import AveragePooling2D, Conv2D, BatchNormalization, Activation, Concatenate, UpSampling2D, Reshape, TimeDistributed, ConvLSTM2D
from tensorflow.keras import Model
import numpy as np
from tensorflow.keras.applications.resnet50 import ResNet50
from hyperopt import fmin, tpe, hp, Trials, STATUS_OK, SparkTrials

Defining functions for normalizing and transforming the images.

# Function for normalizing image_size so that pixel intensity is between 0 and 1
def normalize(input_image, input_mask):
  input_image = tf.cast(input_image, tf.float32) / 255.0
  input_mask -= 1
  return input_image, input_mask
 
# Function for resizing the train images to the desired input shape of 128x128 as well as augmenting the training images
def load_image_train(datapoint):
  input_image = tf.image.resize(datapoint['image'], (128, 128))
  input_mask = tf.image.resize(datapoint['segmentation_mask'], (128, 128))
 
  if tf.random.uniform(()) > 0.5:
    input_image = tf.image.flip_left_right(input_image)
    input_mask = tf.image.flip_left_right(input_mask)
 
  input_image, input_mask = normalize(input_image, input_mask)
  input_mask = tf.math.round(input_mask)
 
  return input_image, input_mask
 
# Function for resizing the test images to the desired output shape (no augmenation)
def load_image_test(datapoint):
  input_image = tf.image.resize(datapoint['image'], (128, 128))
  input_mask = tf.image.resize(datapoint['segmentation_mask'], (128, 128))
 
  input_image, input_mask = normalize(input_image, input_mask)
 
  return input_image, input_mask

The Oxford-IIT Pet Dataset from the TensorFlow datasets is loaded and then the images are transformed in desired way and the datasets used for training and inference are created.

dataset, info = tfds.load('oxford_iiit_pet:3.*.*', with_info=True)
 
TRAIN_LENGTH = info.splits['train'].num_examples
TEST_LENGTH = info.splits['test'].num_examples
BATCH_SIZE = 64
BUFFER_SIZE = 1000
STEPS_PER_EPOCH = TRAIN_LENGTH // BATCH_SIZE
 
train = dataset['train'].map(load_image_train)
test = dataset['test'].map(load_image_test)
Downloading and preparing dataset 773.52 MiB (download: 773.52 MiB, generated: 774.69 MiB, total: 1.51 GiB) to /root/tensorflow_datasets/oxford_iiit_pet/3.2.0... Dl Completed...: 0 url [00:00, ? url/s] Dl Size...: 0 MiB [00:00, ? MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/1 [00:00<?, ? url/s] Dl Size...: 0 MiB [00:00, ? MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 0 MiB [00:00, ? MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 0%| | 0/18 [00:00<?, ? MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 0%| | 0/773 [00:00<?, ? MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Size...: 0%| | 1/773 [00:00<02:24, 5.36 MiB/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 0%| | 1/773 [00:00<02:24, 5.36 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 0%| | 2/773 [00:00<02:23, 5.36 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 0%| | 3/773 [00:00<02:23, 5.36 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 1%| | 4/773 [00:00<02:23, 5.36 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 1%| | 5/773 [00:00<02:23, 5.36 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 1%| | 6/773 [00:00<02:23, 5.36 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 1%| | 7/773 [00:00<02:22, 5.36 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 1%| | 8/773 [00:00<02:22, 5.36 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 1%| | 9/773 [00:00<02:22, 5.36 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Size...: 1%|▏ | 10/773 [00:00<01:42, 7.45 MiB/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 1%|▏ | 10/773 [00:00<01:42, 7.45 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 1%|▏ | 11/773 [00:00<01:42, 7.45 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 2%|▏ | 12/773 [00:00<01:42, 7.45 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 2%|▏ | 13/773 [00:00<01:41, 7.45 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 2%|▏ | 14/773 [00:00<01:41, 7.45 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 2%|▏ | 15/773 [00:00<01:41, 7.45 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 2%|▏ | 16/773 [00:00<01:41, 7.45 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 2%|▏ | 17/773 [00:00<01:41, 7.45 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Size...: 2%|▏ | 18/773 [00:00<01:13, 10.23 MiB/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 2%|▏ | 18/773 [00:00<01:13, 10.23 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 2%|▏ | 19/773 [00:00<01:13, 10.23 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 3%|▎ | 20/773 [00:00<01:13, 10.23 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 3%|▎ | 21/773 [00:00<01:13, 10.23 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 3%|▎ | 22/773 [00:00<01:13, 10.23 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 3%|▎ | 23/773 [00:00<01:13, 10.23 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 3%|▎ | 24/773 [00:00<01:13, 10.23 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 3%|▎ | 25/773 [00:00<01:13, 10.23 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Size...: 3%|▎ | 26/773 [00:00<00:53, 13.85 MiB/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 3%|▎ | 26/773 [00:00<00:53, 13.85 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 3%|▎ | 27/773 [00:00<00:53, 13.85 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 4%|▎ | 28/773 [00:00<00:53, 13.85 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 4%|▍ | 29/773 [00:00<00:53, 13.85 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 4%|▍ | 30/773 [00:00<00:53, 13.85 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 0%| | 0/2 [00:00<?, ? url/s] Dl Size...: 4%|▍ | 31/773 [00:00<00:53, 13.85 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 4%|▍ | 31/773 [00:00<00:53, 13.85 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 4%|▍ | 32/773 [00:00<00:53, 13.85 MiB/s] Extraction completed...: 0 file [00:00, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 4%|▍ | 32/773 [00:00<00:53, 13.85 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 4%|▍ | 33/773 [00:00<00:53, 13.85 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Size...: 4%|▍ | 34/773 [00:00<00:40, 18.06 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 4%|▍ | 34/773 [00:00<00:40, 18.06 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 5%|▍ | 35/773 [00:00<00:40, 18.06 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 5%|▍ | 36/773 [00:00<00:40, 18.06 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 5%|▍ | 37/773 [00:00<00:40, 18.06 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 5%|▍ | 38/773 [00:00<00:40, 18.06 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 5%|▌ | 39/773 [00:00<00:40, 18.06 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Size...: 5%|▌ | 40/773 [00:00<00:33, 21.61 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 5%|▌ | 40/773 [00:00<00:33, 21.61 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 5%|▌ | 41/773 [00:00<00:33, 21.61 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 5%|▌ | 42/773 [00:00<00:33, 21.61 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 6%|▌ | 43/773 [00:00<00:33, 21.61 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 6%|▌ | 44/773 [00:00<00:33, 21.61 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 6%|▌ | 45/773 [00:00<00:33, 21.61 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Size...: 6%|▌ | 46/773 [00:00<00:30, 24.14 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 6%|▌ | 46/773 [00:00<00:30, 24.14 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:00<00:00, 1.76 url/s] Dl Size...: 6%|▌ | 47/773 [00:00<00:30, 24.14 MiB/s] Extraction completed...: 0%| | 0/1 [00:00<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 6%|▌ | 48/773 [00:01<00:30, 24.14 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 6%|▋ | 49/773 [00:01<00:29, 24.14 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 6%|▋ | 50/773 [00:01<00:29, 24.14 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Size...: 7%|▋ | 51/773 [00:01<00:26, 26.83 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 7%|▋ | 51/773 [00:01<00:26, 26.83 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 7%|▋ | 52/773 [00:01<00:26, 26.83 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 7%|▋ | 53/773 [00:01<00:26, 26.83 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 7%|▋ | 54/773 [00:01<00:26, 26.83 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 7%|▋ | 55/773 [00:01<00:26, 26.83 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Size...: 7%|▋ | 56/773 [00:01<00:24, 28.97 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 7%|▋ | 56/773 [00:01<00:24, 28.97 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 7%|▋ | 57/773 [00:01<00:24, 28.97 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 8%|▊ | 58/773 [00:01<00:24, 28.97 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 8%|▊ | 59/773 [00:01<00:24, 28.97 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 8%|▊ | 60/773 [00:01<00:24, 28.97 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Size...: 8%|▊ | 61/773 [00:01<00:23, 30.47 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 8%|▊ | 61/773 [00:01<00:23, 30.47 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 8%|▊ | 62/773 [00:01<00:23, 30.47 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 8%|▊ | 63/773 [00:01<00:23, 30.47 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 8%|▊ | 64/773 [00:01<00:23, 30.47 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Size...: 8%|▊ | 65/773 [00:01<00:22, 30.82 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 8%|▊ | 65/773 [00:01<00:22, 30.82 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 9%|▊ | 66/773 [00:01<00:22, 30.82 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 9%|▊ | 67/773 [00:01<00:22, 30.82 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 9%|▉ | 68/773 [00:01<00:22, 30.82 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Size...: 9%|▉ | 69/773 [00:01<00:21, 32.20 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 9%|▉ | 69/773 [00:01<00:21, 32.20 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 9%|▉ | 70/773 [00:01<00:21, 32.20 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 9%|▉ | 71/773 [00:01<00:21, 32.20 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 9%|▉ | 72/773 [00:01<00:21, 32.20 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Size...: 9%|▉ | 73/773 [00:01<00:21, 32.96 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 9%|▉ | 73/773 [00:01<00:21, 32.96 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 10%|▉ | 74/773 [00:01<00:21, 32.96 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 10%|▉ | 75/773 [00:01<00:21, 32.96 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 10%|▉ | 76/773 [00:01<00:21, 32.96 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Size...: 10%|▉ | 77/773 [00:01<00:20, 33.34 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 10%|▉ | 77/773 [00:01<00:20, 33.34 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 10%|█ | 78/773 [00:01<00:20, 33.34 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 10%|█ | 79/773 [00:01<00:20, 33.34 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 10%|█ | 80/773 [00:01<00:20, 33.34 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Size...: 10%|█ | 81/773 [00:01<00:21, 32.86 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:01<00:00, 1.76 url/s] Dl Size...: 10%|█ | 81/773 [00:01<00:21, 32.86 MiB/s] Extraction completed...: 0%| | 0/1 [00:01<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 11%|█ | 82/773 [00:02<00:21, 32.86 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 11%|█ | 83/773 [00:02<00:20, 32.86 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 11%|█ | 84/773 [00:02<00:20, 32.86 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Size...: 11%|█ | 85/773 [00:02<00:20, 32.97 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 11%|█ | 85/773 [00:02<00:20, 32.97 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 11%|█ | 86/773 [00:02<00:20, 32.97 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 11%|█▏ | 87/773 [00:02<00:20, 32.97 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 11%|█▏ | 88/773 [00:02<00:20, 32.97 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Size...: 12%|█▏ | 89/773 [00:02<00:20, 32.80 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 12%|█▏ | 89/773 [00:02<00:20, 32.80 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 12%|█▏ | 90/773 [00:02<00:20, 32.80 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 12%|█▏ | 91/773 [00:02<00:20, 32.80 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 12%|█▏ | 92/773 [00:02<00:20, 32.80 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Size...: 12%|█▏ | 93/773 [00:02<00:19, 34.04 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 12%|█▏ | 93/773 [00:02<00:19, 34.04 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 12%|█▏ | 94/773 [00:02<00:19, 34.04 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 12%|█▏ | 95/773 [00:02<00:19, 34.04 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 12%|█▏ | 96/773 [00:02<00:19, 34.04 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Size...: 13%|█▎ | 97/773 [00:02<00:19, 34.12 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 13%|█▎ | 97/773 [00:02<00:19, 34.12 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 13%|█▎ | 98/773 [00:02<00:19, 34.12 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 13%|█▎ | 99/773 [00:02<00:19, 34.12 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 13%|█▎ | 100/773 [00:02<00:19, 34.12 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Size...: 13%|█▎ | 101/773 [00:02<00:19, 34.12 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 13%|█▎ | 101/773 [00:02<00:19, 34.12 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 13%|█▎ | 102/773 [00:02<00:19, 34.12 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 13%|█▎ | 103/773 [00:02<00:19, 34.12 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 13%|█▎ | 104/773 [00:02<00:19, 34.12 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Size...: 14%|█▎ | 105/773 [00:02<00:19, 34.08 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 14%|█▎ | 105/773 [00:02<00:19, 34.08 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 14%|█▎ | 106/773 [00:02<00:19, 34.08 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 14%|█▍ | 107/773 [00:02<00:19, 34.08 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 14%|█▍ | 108/773 [00:02<00:19, 34.08 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Size...: 14%|█▍ | 109/773 [00:02<00:19, 33.34 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 14%|█▍ | 109/773 [00:02<00:19, 33.34 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 14%|█▍ | 110/773 [00:02<00:19, 33.34 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] Dl Size...: 14%|█▍ | 111/773 [00:02<00:19, 33.34 MiB/s] Extraction completed...: 0%| | 0/1 [00:02<?, ? file/s] Dl Completed...: 50%|█████ | 1/2 [00:02<00:00, 1.76 url/s] *** WARNING: skipped 129057 bytes of output *** Dl Size...: 89%|████████▉ | 688/773 [00:11<00:01, 74.32 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 89%|████████▉ | 689/773 [00:11<00:01, 74.32 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 89%|████████▉ | 690/773 [00:11<00:01, 74.32 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 89%|████████▉ | 691/773 [00:11<00:01, 74.32 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 90%|████████▉ | 692/773 [00:11<00:01, 74.32 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Size...: 90%|████████▉ | 693/773 [00:11<00:01, 74.05 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 90%|████████▉ | 693/773 [00:11<00:01, 74.05 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 90%|████████▉ | 694/773 [00:11<00:01, 74.05 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 90%|████████▉ | 695/773 [00:11<00:01, 74.05 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 90%|█████████ | 696/773 [00:11<00:01, 74.05 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 90%|█████████ | 697/773 [00:11<00:01, 74.05 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 90%|█████████ | 698/773 [00:11<00:01, 74.05 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 90%|█████████ | 699/773 [00:11<00:00, 74.05 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 91%|█████████ | 700/773 [00:11<00:00, 74.05 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Size...: 91%|█████████ | 701/773 [00:11<00:00, 74.08 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 91%|█████████ | 701/773 [00:11<00:00, 74.08 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 91%|█████████ | 702/773 [00:11<00:00, 74.08 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 91%|█████████ | 703/773 [00:11<00:00, 74.08 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 91%|█████████ | 704/773 [00:11<00:00, 74.08 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 91%|█████████ | 705/773 [00:11<00:00, 74.08 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 91%|█████████▏| 706/773 [00:11<00:00, 74.08 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 91%|█████████▏| 707/773 [00:11<00:00, 74.08 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 92%|█████████▏| 708/773 [00:11<00:00, 74.08 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Size...: 92%|█████████▏| 709/773 [00:11<00:00, 74.02 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 92%|█████████▏| 709/773 [00:11<00:00, 74.02 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 92%|█████████▏| 710/773 [00:11<00:00, 74.02 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 92%|█████████▏| 711/773 [00:11<00:00, 74.02 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 92%|█████████▏| 712/773 [00:11<00:00, 74.02 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 92%|█████████▏| 713/773 [00:11<00:00, 74.02 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 92%|█████████▏| 714/773 [00:11<00:00, 74.02 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 92%|█████████▏| 715/773 [00:11<00:00, 74.02 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 93%|█████████▎| 716/773 [00:11<00:00, 74.02 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Size...: 93%|█████████▎| 717/773 [00:11<00:00, 74.07 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 93%|█████████▎| 717/773 [00:11<00:00, 74.07 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 93%|█████████▎| 718/773 [00:11<00:00, 74.07 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 93%|█████████▎| 719/773 [00:11<00:00, 74.07 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 93%|█████████▎| 720/773 [00:11<00:00, 74.07 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 93%|█████████▎| 721/773 [00:11<00:00, 74.07 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 93%|█████████▎| 722/773 [00:11<00:00, 74.07 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 94%|█████████▎| 723/773 [00:11<00:00, 74.07 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 94%|█████████▎| 724/773 [00:11<00:00, 74.07 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Size...: 94%|█████████▍| 725/773 [00:11<00:00, 74.22 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 94%|█████████▍| 725/773 [00:11<00:00, 74.22 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 94%|█████████▍| 726/773 [00:11<00:00, 74.22 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 94%|█████████▍| 727/773 [00:11<00:00, 74.22 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 94%|█████████▍| 728/773 [00:11<00:00, 74.22 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:11<00:00, 1.76 url/s] Dl Size...: 94%|█████████▍| 729/773 [00:11<00:00, 74.22 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:11<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 94%|█████████▍| 730/773 [00:12<00:00, 74.22 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 95%|█████████▍| 731/773 [00:12<00:00, 74.22 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 95%|█████████▍| 732/773 [00:12<00:00, 74.22 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Size...: 95%|█████████▍| 733/773 [00:12<00:00, 74.51 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 95%|█████████▍| 733/773 [00:12<00:00, 74.51 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 95%|█████████▍| 734/773 [00:12<00:00, 74.51 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 95%|█████████▌| 735/773 [00:12<00:00, 74.51 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 95%|█████████▌| 736/773 [00:12<00:00, 74.51 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 95%|█████████▌| 737/773 [00:12<00:00, 74.51 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 95%|█████████▌| 738/773 [00:12<00:00, 74.51 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 96%|█████████▌| 739/773 [00:12<00:00, 74.51 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 96%|█████████▌| 740/773 [00:12<00:00, 74.51 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Size...: 96%|█████████▌| 741/773 [00:12<00:00, 74.34 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 96%|█████████▌| 741/773 [00:12<00:00, 74.34 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 96%|█████████▌| 742/773 [00:12<00:00, 74.34 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 96%|█████████▌| 743/773 [00:12<00:00, 74.34 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 96%|█████████▌| 744/773 [00:12<00:00, 74.34 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 96%|█████████▋| 745/773 [00:12<00:00, 74.34 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 97%|█████████▋| 746/773 [00:12<00:00, 74.34 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 97%|█████████▋| 747/773 [00:12<00:00, 74.34 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 97%|█████████▋| 748/773 [00:12<00:00, 74.34 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Size...: 97%|█████████▋| 749/773 [00:12<00:00, 74.43 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 97%|█████████▋| 749/773 [00:12<00:00, 74.43 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 97%|█████████▋| 750/773 [00:12<00:00, 74.43 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 97%|█████████▋| 751/773 [00:12<00:00, 74.43 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 97%|█████████▋| 752/773 [00:12<00:00, 74.43 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 97%|█████████▋| 753/773 [00:12<00:00, 74.43 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 98%|█████████▊| 754/773 [00:12<00:00, 74.43 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 98%|█████████▊| 755/773 [00:12<00:00, 74.43 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 98%|█████████▊| 756/773 [00:12<00:00, 74.43 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Size...: 98%|█████████▊| 757/773 [00:12<00:00, 74.40 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 98%|█████████▊| 757/773 [00:12<00:00, 74.40 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 98%|█████████▊| 758/773 [00:12<00:00, 74.40 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 98%|█████████▊| 759/773 [00:12<00:00, 74.40 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 98%|█████████▊| 760/773 [00:12<00:00, 74.40 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 98%|█████████▊| 761/773 [00:12<00:00, 74.40 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 99%|█████████▊| 762/773 [00:12<00:00, 74.40 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 99%|█████████▊| 763/773 [00:12<00:00, 74.40 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 99%|█████████▉| 764/773 [00:12<00:00, 74.40 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Size...: 99%|█████████▉| 765/773 [00:12<00:00, 74.33 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 99%|█████████▉| 765/773 [00:12<00:00, 74.33 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 99%|█████████▉| 766/773 [00:12<00:00, 74.33 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 99%|█████████▉| 767/773 [00:12<00:00, 74.33 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 99%|█████████▉| 768/773 [00:12<00:00, 74.33 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 99%|█████████▉| 769/773 [00:12<00:00, 74.33 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 100%|█████████▉| 770/773 [00:12<00:00, 74.33 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 100%|█████████▉| 771/773 [00:12<00:00, 74.33 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 100%|█████████▉| 772/773 [00:12<00:00, 74.33 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Size...: 100%|██████████| 773/773 [00:12<00:00, 74.43 MiB/s] Dl Completed...: 50%|█████ | 1/2 [00:12<00:00, 1.76 url/s] Dl Size...: 100%|██████████| 773/773 [00:12<00:00, 74.43 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 100%|██████████| 2/2 [00:12<00:00, 4.00s/ url] Dl Completed...: 100%|██████████| 2/2 [00:12<00:00, 4.00s/ url] Dl Size...: 100%|██████████| 773/773 [00:12<00:00, 74.43 MiB/s] Extraction completed...: 100%|██████████| 1/1 [00:12<00:00, 4.40s/ file] Dl Completed...: 100%|██████████| 2/2 [00:12<00:00, 4.00s/ url] Dl Size...: 100%|██████████| 773/773 [00:12<00:00, 74.43 MiB/s] Extraction completed...: 50%|█████ | 1/2 [00:12<00:04, 4.40s/ file] Extraction completed...: 100%|██████████| 2/2 [00:18<00:00, 7.39s/ file] Dl Completed...: 100%|██████████| 2/2 [00:18<00:00, 4.00s/ url] Dl Size...: 100%|██████████| 773/773 [00:18<00:00, 74.43 MiB/s] Extraction completed...: 100%|██████████| 2/2 [00:18<00:00, 7.39s/ file] Extraction completed...: 100%|██████████| 2/2 [00:18<00:00, 9.38s/ file] Dl Size...: 100%|██████████| 773/773 [00:18<00:00, 41.20 MiB/s] Dl Completed...: 100%|██████████| 2/2 [00:18<00:00, 9.38s/ url] Generating splits...: 0%| | 0/2 [00:00<?, ? splits/s] Generating train examples...: 0%| | 0/3680 [00:00<?, ? examples/s] Generating train examples...: 7%|▋ | 247/3680 [00:00<00:01, 2462.67 examples/s] Generating train examples...: 13%|█▎ | 494/3680 [00:00<00:01, 2464.04 examples/s] Generating train examples...: 20%|█▉ | 732/3680 [00:00<00:01, 2437.21 examples/s] Generating train examples...: 26%|██▌ | 963/3680 [00:00<00:01, 2397.17 examples/s] Generating train examples...: 33%|███▎ | 1208/3680 [00:00<00:01, 2411.82 examples/s] Generating train examples...: 40%|███▉ | 1461/3680 [00:00<00:00, 2445.64 examples/s] Generating train examples...: 46%|████▌ | 1697/3680 [00:00<00:00, 2417.55 examples/s] Generating train examples...: 53%|█████▎ | 1934/3680 [00:00<00:00, 2402.13 examples/s] Generating train examples...: 59%|█████▉ | 2185/3680 [00:00<00:00, 2432.12 examples/s] Generating train examples...: 66%|██████▌ | 2433/3680 [00:01<00:00, 2443.77 examples/s] Generating train examples...: 73%|███████▎ | 2671/3680 [00:01<00:00, 2417.79 examples/s] Generating train examples...: 79%|███████▉ | 2909/3680 [00:01<00:00, 2400.33 examples/s] Generating train examples...: 86%|████████▌ | 3155/3680 [00:01<00:00, 2417.39 examples/s] Generating train examples...: 92%|█████████▏| 3395/3680 [00:01<00:00, 2404.77 examples/s] Generating train examples...: 99%|█████████▉| 3634/3680 [00:01<00:00, 2349.52 examples/s] Shuffling oxford_iiit_pet-train.tfrecord...: 0%| | 0/3680 [00:00<?, ? examples/s] Shuffling oxford_iiit_pet-train.tfrecord...: 0%| | 1/3680 [00:00<17:28, 3.51 examples/s] Shuffling oxford_iiit_pet-train.tfrecord...: 44%|████▍ | 1613/3680 [00:00<06:52, 5.01 examples/s] Shuffling oxford_iiit_pet-train.tfrecord...: 86%|████████▌ | 3156/3680 [00:00<01:13, 7.16 examples/s] Generating splits...: 50%|█████ | 1/2 [00:02<00:02, 2.06s/ splits] Generating test examples...: 0%| | 0/3669 [00:00<?, ? examples/s] Generating test examples...: 7%|▋ | 261/3669 [00:00<00:01, 2603.25 examples/s] Generating test examples...: 14%|█▎ | 504/3669 [00:00<00:01, 2547.87 examples/s] Generating test examples...: 21%|██ | 765/3669 [00:00<00:01, 2565.17 examples/s] Generating test examples...: 28%|██▊ | 1024/3669 [00:00<00:01, 2571.52 examples/s] Generating test examples...: 35%|███▍ | 1284/3669 [00:00<00:00, 2579.29 examples/s] Generating test examples...: 42%|████▏ | 1532/3669 [00:00<00:00, 2547.19 examples/s] Generating test examples...: 48%|████▊ | 1778/3669 [00:00<00:00, 2518.01 examples/s] Generating test examples...: 55%|█████▌ | 2030/3669 [00:00<00:00, 2518.16 examples/s] Generating test examples...: 62%|██████▏ | 2280/3669 [00:00<00:00, 2510.86 examples/s] Generating test examples...: 69%|██████▉ | 2530/3669 [00:01<00:00, 2507.18 examples/s] Generating test examples...: 76%|███████▌ | 2776/3669 [00:01<00:00, 2492.52 examples/s] Generating test examples...: 82%|████████▏ | 3021/3669 [00:01<00:00, 2407.38 examples/s] Generating test examples...: 89%|████████▉ | 3259/3669 [00:01<00:00, 2395.50 examples/s] Generating test examples...: 95%|█████████▌| 3497/3669 [00:01<00:00, 2388.98 examples/s] Shuffling oxford_iiit_pet-test.tfrecord...: 0%| | 0/3669 [00:00<?, ? examples/s] Shuffling oxford_iiit_pet-test.tfrecord...: 40%|███▉ | 1465/3669 [00:00<00:00, 14649.42 examples/s] Shuffling oxford_iiit_pet-test.tfrecord...: 65%|██████▌ | 2388/3669 [00:00<00:00, 12181.09 examples/s] Shuffling oxford_iiit_pet-test.tfrecord...: 81%|████████ | 2958/3669 [00:00<00:00, 4114.81 examples/s] Shuffling oxford_iiit_pet-test.tfrecord...: 93%|█████████▎| 3406/3669 [00:00<00:00, 2312.63 examples/s] Generating splits...: 100%|██████████| 2/2 [00:04<00:00, 2.26s/ splits] Dataset oxford_iiit_pet downloaded and prepared to /root/tensorflow_datasets/oxford_iiit_pet/3.2.0. Subsequent calls will reuse this data.

The dataset is then converted to numpy format since Spark Trials does not yet handle the TensorFlow Dataset class.

train_numpy = tfds.as_numpy(train)
test_numpy = tfds.as_numpy(test)
X_train = np.array(list(map(lambda x: x[0], train_numpy)))
Y_train = np.array(list(map(lambda x: x[1], train_numpy)))
 
X_test = np.array(list(map(lambda x: x[0], test_numpy)))
Y_test = np.array(list(map(lambda x: x[1], test_numpy)))

An example image is displayed. 

def display(display_list):
  plt.figure(figsize=(15, 15))
 
  title = ['Input Image', 'True Mask', 'Predicted Mask']
 
  for i in range(len(display_list)):
    plt.subplot(1, len(display_list), i+1)
    plt.title(title[i])
    plt.imshow(tf.keras.preprocessing.image.array_to_img(display_list[i]))
    plt.axis('off')
  plt.show()
 
it = iter(train)
#for image, mask in next(train):
sample_image, sample_mask = next(it)
display([sample_image, sample_mask])

Defining the functions needed for the PSPNet.

def pool_block(cur_tensor: tf.Tensor,
               image_width: int,
               image_height: int,
               pooling_factor: int,
               activation: str = 'relu'
              ) -> tf.Tensor:
  """
  Parameters
  ----------
  cur_tensor : tf.Tensor
      Incoming tensor.
 
  image_width : int
      Width of the image.
 
  image_height : int
      Height of the image.
 
  pooling_factor : int
      Pooling factor to scale image.
 
  activation : str, default = 'relu'
      Activation function to be applied after the pooling operations.
 
  Returns
  -------
  tf.Tensor
      2D convolutional block.
  """
  
  #Calculate the strides with image size and pooling factor
  strides = [int(np.round(float(image_width)/pooling_factor)),
            int(np.round(float(image_height)/pooling_factor))]
  
  pooling_size = strides
  
  x = AveragePooling2D(pooling_size, strides=strides, padding='same')(cur_tensor)
  
  x = Conv2D(filters = 512, 
             kernel_size = (1,1),
             padding = 'same')(x)
  
  x = BatchNormalization()(x)
  x = Activation(activation)(x)
  
  # Resizing images to correct shape for future concat
  x = tf.keras.layers.experimental.preprocessing.Resizing(
    image_height, 
    image_width, 
    interpolation="bilinear")(x) 
  
  return x
 
def modify_ResNet_Dilation(
  model: tf.keras.Model
) -> tf.keras.Model:
  """
  Modifies the ResNet to fit the PSPNet Paper with the described dilated strategy.
  
    Parameters
    ----------
    model : tf.keras.Model
        The ResNet-50 model to be modified.
        
    Returns
    -------
    tf.keras.Model
        Modified model.
  """
  for i in range(0,4):
    model.get_layer('conv4_block1_{}_conv'.format(i)).strides = 1
    model.get_layer('conv4_block1_{}_conv'.format(i)).dilation_rate = 2
    model.get_layer('conv5_block1_{}_conv'.format(i)).strides = 1
    model.get_layer('conv5_block1_{}_conv'.format(i)).dilation_rate = 4
  model.save('/tmp/my_model')
  new_model = tf.keras.models.load_model('/tmp/my_model')
  return new_model
 
def PSPNet(image_width: int,
           image_height: int,
           n_classes: int,
           kernel_size: tuple = (3,3),
           activation: str = 'relu',
           weights: str = 'imagenet',
           shallow_tuning: bool = False,
           isICNet: bool = False
          ) -> tf.keras.Model:
  """
  Setting up the PSPNet model
  
    Parameters
    ----------
    image_width : int
        Width of the image.
        
    image_height : int
        Height of the image.
        
    n_classes : int
        Number of classes.
        
    kernel_size : tuple, default = (3, 3)
        Size of the kernel.    
        
    activation : str, default = 'relu'
        Activation function to be applied after the pooling operations.
        
    weights: str, default = 'imagenet'
        String defining which weights to use for the backbone, 'imagenet' for ImageNet weights or None for normalized initialization.
        
    shallow_tuning : bool, default = True
        Boolean for using shallow tuning with pre-trained weights from ImageNet or not.
        
    isICNet : bool,  default = False   
        Boolean to determine if the PSPNet will be part of an ICNet.
        
    Returns
    -------
    tf.keras.Model
        The finished keras model for PSPNet.
  """
  if shallow_tuning and not weights:
    raise ValueError("Shallow tuning can not be performed without loading pre-trained weights. Please input 'imagenet' to argument weights...")
  #If the function is used for the ICNet input_shape is set to none as ICNet takes 3 inputs
  if isICNet:
    input_shape=(None, None, 3)
  else:
    input_shape=(image_height,image_width,3)
    
  #Initializing the ResNet50 Backbone  
  y=ResNet50(include_top=False, weights=weights, input_shape=input_shape)
 
  y=modify_ResNet_Dilation(y)
  if shallow_tuning:
    y.trainable=False
  
  pooling_layer=[]
  output=y.output
 
  pooling_layer.append(output)
  
  h = image_height//8
  w = image_width//8
  
  #Loop for calling the pool block functions for pooling factors [1,2,3,6]
  for i in [1,2,3,6]:
    pool = pool_block(output, h, w, i, activation)
    pooling_layer.append(pool)
    
  x=Concatenate()(pooling_layer)
  
  x=Conv2D(filters=n_classes, kernel_size=(1,1), padding='same')(x)
  
  x=UpSampling2D(size=(8,8), data_format='channels_last', interpolation='bilinear')(x)
  x=Reshape((image_height*image_width, n_classes))(x)
  x=Activation(tf.nn.softmax)(x)
  x=Reshape((image_height,image_width,n_classes))(x)
 
  final_model=tf.keras.Model(inputs=y.input, outputs=x)
                              
  return final_model

Defining the custom data generators used for training the model.

def batch_generator(batch_size):
  indices = np.arange(len(X_train)) 
  batch=[]
  while True:
  # it might be a good idea to shuffle your data before each epoch
    np.random.shuffle(indices) 
    for i in indices:
      batch.append(i)
      if len(batch)==batch_size:
        yield X_train[batch], Y_train[batch]
        batch=[]
        
def batch_generator_eval(batch_size):
  indices = np.arange(len(X_test)) 
  batch=[]
  while True:
    for i in indices:
      batch.append(i)
      if len(batch)==batch_size:
        yield X_test[batch], Y_test[batch]
        batch=[]

To exploit parallelism, Spark Trials require that you define a function to be used for loading the dataset as well as training and evaluating the model.

def train_spark(params):
  
  gpus = tf.config.list_physical_devices('GPU')
  if gpus:
    try:
      # Currently, memory growt*h needs to be the same across GPUs
      for gpu in gpus:
        tf.config.experimental.set_memory_growth(gpu, True)
      logical_gpus = tf.config.experimental.list_logical_devices('GPU')
      print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
    except RuntimeError as e:
      # Memory growth must be set before GPUs have been initialized
      print(e)
  VAL_SUBSPLITS=5
  EPOCHS=50
  VALIDATION_STEPS = TEST_LENGTH//params['batch_size']//VAL_SUBSPLITS
  STEPS_PER_EPOCH = TRAIN_LENGTH // params['batch_size']
  BATCH_SIZE = params['batch_size']
  print(BATCH_SIZE)
  BUFFER_SIZE = 1000
  """
  An example train method that calls into HorovodRunner.
  This method is passed to hyperopt.fmin().
  
  :param params: hyperparameters. Its structure is consistent with how search space is defined. See below.
  :return: dict with fields 'loss' (scalar loss) and 'status' (success/failure status of run)
  """
  train_dataset = batch_generator(BATCH_SIZE)
  test_dataset = batch_generator_eval(BATCH_SIZE)
  model=PSPNet(128,128,3)
  model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=params['learning_rate']),
              loss=tf.keras.losses.SparseCategoricalCrossentropy(),
              metrics=tf.keras.metrics.SparseCategoricalAccuracy())
  
  model_history =  model.fit(train_dataset, epochs=EPOCHS, steps_per_epoch=STEPS_PER_EPOCH)
  loss = model.evaluate(test_dataset, steps=VALIDATION_STEPS)[0]
  return {'loss': loss, 'status': STATUS_OK}

In this experiment, the hyperparameters learning_rate and batch_size were explored through random search in combination with an adaptive algorithm called Tree of Parzen Estimators (TPE). Since parallelism were set to 4 the choice of hyperparameters will be set randomly for the first 4 runs in parallel and then adaptively for the second pass of 4 runs.

space = {
  'learning_rate': hp.loguniform('learning_rate', np.log(1e-4), np.log(1e-1)),
  'batch_size': hp.choice('batch_size', [16, 32, 64]),
}