ScaDaMaLe Course site and book

Project Description and Introduction

Project Members:
Chi Zhang, zchi@chalmers.se;
Shuangshuang Chen, shuche@kth.se;
Magnus Tarle, tarle@kth.se

Project presentation video

The presentation record link is here: https://drive.google.com/drive/folders/13XwlItZqtOeBZ5TJfnCP1hqtQ9imRFq
The final complete 20 minutes presentation is: "Projectcomplete_video.mp4"
Because the cluster seems quite slow when we recording the video and there are too many things to run within 20 minutes, we recorded the video after we finish running all the codes. And some parts of the video are speeded up to meet the 20 minutes requirement. We also put the original raw video records in the folder, from 00 to 08 videos correspond to the script files from 00 to 08, in case you only want to know about a certain part.

Project Introduction - Analysis and Prediction of COVID-19 Data

In this project, we use both scala (data processing part) and python (algorithm part). We deal with scalable data, and use what we learned from the course.

1. Project Plan

In this project, we dealt with COVID-19 data, and do the following tasks:

1. Introduction
2. Struct stream data - to update our database once a day.
3. Preprocessing - clean data.
4. Visualization - visualize new cases on a world map, with different time scope.
5. Statistics analysis - get the distributions, mean and std of different varables.
6. Model 1: K-means - clustering of different countries
7. Model 2: Linear Regression - predict new cases of some contries from other countries.
8. Model 3: Autoregressive integrated moving average (ARIMA) - prediction for new cases and new deaths from past values.
9. Model 4: Gaussian Procecces (GP) - apply Gaussian Procecces (GP) to predict mean and covariance of new cases and new deaths from past values.
10. Ending

2. Tools and methods

For the method part, this particular task is not suitable to use deep learning methods. We choose 4 methods related to Machine leanring for our clustering and prediction task: 1. Clustering model - K-means 2. Time series model - Autoregressive integrated moving average (ARIMA) 3. Gaussian Procecces (GP) 4. Linear Regression (LR)

3. Data resources

We found the following data resources. And finally we chose the 3rd dataset because it has the most features for us to use.

1. WHO: https://covid19.who.int/WHO-COVID-19-global-data.csv

2. COVID Tracking Project API at Cambridge: https://api.covidtracking.com

3. Data on COVID-19 (coronavirus) by Our World in Data: git repo https://github.com/owid/covid-19-data/tree/master/public/data

4. Sweden (proceesed by apify) API: https://api.apify.com/v2/datasets/Nq3XwHX262iDwsFJS/items?format=json&clean=1

Note that the preprocessing, visualization and analysis within these notebooks were made on this 3rd dataset, downloaded December 2020 to Databricks. One dataset from December 2020, "owid-covid-data.csv", can also be found in the same google drive folder as the video presentation.

4. Useful links

1. Description of each column in the "Our World in Data" dataset https://github.com/owid/covid-19-data/blob/master/public/data/owid-covid-codebook.csv
2. If you want to have an intro to GP, you could look at: http://krasserm.github.io/2018/03/19/gaussian-processes/ and https://medium.com/@dan_90739/being-bayesian-and-thinking-deep-time-series-prediction-with-uncertainty-25ff581b056c
3. ARIMA - Autoregressive Integrated Moving Average model. It's widely used in time series analysis. see defination here: https://en.wikipedia.org/wiki/Autoregressiveintegratedmoving_average

5. Meeting Records:

Weekly meetings for group discussion:

1. 2020-12-01 Tuesday:

• Discussion about project topic
• Find data resources

1. 2020-12-04 Friday:
   • Understand dataset 3, and manage download data
   • Set up processing pipeline into dataframe
   • Prepared for statistics analysis
   • Selected the column for analysis and machine learning
   • Finished struct stream for updating data
2. 2020-12-08 Tuesday:
   • Exploited each chosen column, plot statistics
   • Dealt with missing data
   • Post-process for useful features from existed column (for statistics analysis and prediction/regression)
3. 2020-12-16 Wednesday:

• Finished statistics analysis, which correlation would interesting to show
• Progress on visulization
• Progress on GP model
• Progress on LR model
• Progress on ARIMA model

1. 2020-12-18 Friday:

• Progress on K-means model
• Finish visulization
• Finish other models
• Finish Evaluation

1. 2021-01-07 Thursday:

• Finish all models
• Discussing about the final presentation

Project Progress

0. Introduction of the data

The columns we selected to analyze are: - continent - location - date - totalcases - newcases - totaldeaths - newdeaths - reproductionrate - icupatients - hosppatients - weeklyicuadmissions - weeklyhospadmissions - totaltests - newtests - stingencyindex - population - populationdensity - medianage - aged65older - aged70older - gdppercapita - extremepoverty - cardiovascdeathrate - diabetesprevalence - femalesmokers - malesmokers - hospitalbedsperthousand - lifeexpectancy - humandevelopmentindex

1. Streaming

This part has been moved to separate files: "DownloadFilesPeriodicallyScript" and "StreamToFile".

2. Preprocessing

To Rerun Steps 1-5 done in the notebook at: * Workspace -> PATH_TO -> DataPreprocess]

just run the following command as shown in the cell below:

%run "PATH_TO/DataPreprocess"

• Note: If you already evaluated the %run ... command above then:
  • first delete the cell by pressing on x on the top-right corner of the cell and
  • revaluate the run command above.

"./02_DataPreprocess"

3. Explosive Analysis

"./03_ExplosiveAnalysis"

4. Visualization

"./04_DataVisualize"

5. Model 1: Clustering K-means

Clustering of countries based on the dataset features of each country.

This part is in notebook 05Clustering.
The code can be run either in here or in the 05Clustering notebook (after data preprocessing). We run and show the results in the 05_Clustering notebook.

"./05_Clustering"

6. Model 2: Linear Regression (LR) Model

Prediction with constant values, predict new cases of some contries from other countries.

This part is in notebook 06DataPredictionLR.
The code can be run either in here or in the 06DataPredictionLR notebook (after data preprocessing). We run and show the results in the 06DataPredictionLR notebook.

"./06_DataPredicton_LR"

7. Model 3: Autoregressive integrated moving average (ARIMA)

prediction for new cases and new deaths from past values.

This part is in notebook 07DataPredictionARIMA.
The code can be run either in here or in the 07DataPredictionARIMA notebook (after data preprocessing). We run and show the results in the 07DataPredictionARIMA notebook.

"./07_DataPredicton_ARIMA"

8. Model 4: Gaussian Procecces (GP)

apply Gaussian Procecces (GP) to predict mean and covariance of new cases and new deaths from past values.

This part is in notebook 08DataPredictionGP.
The code can be run either in here or in the 08DataPredictionGP notebook (after data preprocessing). We run and show the results in the 08DataPredictionGP notebook.