%md

ScaDaMaLe Course [site](https://lamastex.github.io/scalable-data-science/sds/3/x/) and [book](https://lamastex.github.io/ScaDaMaLe/index.html)

%md

# Write files with animal names continuously for structured streaming

This notebook can be used to write files every 2 seconds into the distributed file system where each of these files contains a row given by the time stamp and two animals chosen at random from six animals in a `animals.txt` file in the driver.

After running the commands in this notebook you should have a a set of files named by the minute and second for easy setting up of structured streaming jobs in another notebook. This is mainly to create a structured streaming of files for learning purposes. In a real situation, you will have such streams coming from more robust ingestion frameworks such as kafka queues.

It is a good idea to understand how to run executibles from the driver to set up a stream of files for ingestion in structured streaming tasks down stream. 

The following *seven steps (Steps 0-6)* can be used in more complex situations like running a more complex simulator from an executible file.

%md

## Step 0

let's get our bearings and prepare for setting up a structured streaming from files.

%md

Just find the working directory using `%sh`.

%md

We are in `databricks/driver` directory.

%md

To run the script and be able to kill it you need a few installs.

%md

## Step 1

Let's first make the `animals.txt` file in the driver.

%md

## Step 2

Now let's make a `bash` shell script that can be executed every two seconds to produce the desired `.log` files with names prepended by minute and second inside the local directory `logsEvery2Secs`. Each line the file `every2SecsRndWordsInFiles.sh` is explained line by line:

* `#!/bin/bash` is how we tell that this is a `bash` script which needs the `/bin/bash` binary. I remember the magic two characters `#!` as "SHA-BANG" for "hash" for `#` and "bang" for `!`

* `rm -f every2SecsRndWordsInFiles.sh &&` forcefully removes the file `every2SecsRndWordsInFiles.sh` and `&&` executes the command preceeding it before going to the next line

* `echo "blah" >> every2SecsRndWordsInFiles.sh` just spits out the content of the string, i.e., `blah`, in append mode due to `>>` into the file `every2SecsRndWordsInFiles.sh`

The rest of the commands simply create a fresh directory `logsEvery2Secs` and write two randomly chosen animals from the `animals.txt` file into the directory `logsEvery2Secs` with `.log` file names preceeded by minute and second of current time to make a finite number of file names (at most 3600 unique `.log` filenames).

%md

## Step 3

Time to run the script!

%md

The next two cells in `%sh` do the following:

* makes sure the BASH script `every2SecsRndWordsInFiles.sh` is executible 

* run the script in the background without hangup