%md
# [SDS-2.x, Scalable Data Engineering Science](https://lamastex.github.io/scalable-data-science/sds/2/x/)

%md
Archived YouTube video of this live unedited lab-lecture:

[![Archived YouTube video of this live unedited lab-lecture](http://img.youtube.com/vi/rBZih2BQZuw/0.jpg)](https://www.youtube.com/embed/rBZih2BQZuw?start=428&end=3039&autoplay=1)

%md 
#Power Plant ML Pipeline Application
This is an end-to-end example of using a number of different machine learning algorithms to solve a supervised regression problem.

###Table of Contents

- *Step 1: Business Understanding*
- *Step 2: Load Your Data*
- *Step 3: Explore Your Data*
- *Step 4: Visualize Your Data*
- *Step 5: Data Preparation*
- *Step 6: Data Modeling*
- *Step 7: Tuning and Evaluation*
- *Step 8: Deployment*



*We are trying to predict power output given a set of readings from various sensors in a gas-fired power generation plant.  Power generation is a complex process, and understanding and predicting power output is an important element in managing a plant and its connection to the power grid.*

More information about Peaker or Peaking Power Plants can be found on Wikipedia https://en.wikipedia.org/wiki/Peaking_power_plant


Given this business problem, we need to translate it to a Machine Learning task.  The ML task is regression since the label (or target) we are trying to predict is numeric.


The example data is provided by UCI at [UCI Machine Learning Repository Combined Cycle Power Plant Data Set](https://archive.ics.uci.edu/ml/datasets/Combined+Cycle+Power+Plant)

You can read the background on the UCI page, but in summary we have collected a number of readings from sensors at a Gas Fired Power Plant

(also called a Peaker Plant) and now we want to use those sensor readings to predict how much power the plant will generate.


More information about Machine Learning with Spark can be found in the [Spark MLLib Programming Guide](https://spark.apache.org/docs/latest/mllib-guide.html)


*Please note this example only works with Spark version 1.4 or higher*

%md
***
***

%md
To **Rerun Steps 1-4** done in the notebook at:

* [Workspace -> scalable-data-science -> sds-2-2 -> 009_PowerPlantPipeline_01ETLEDA](/#workspace/scalable-data-science/sds-2-2/009_PowerPlantPipeline_01ETLEDA)

just `run` the following command as shown in the cell below: 

  ```%scala
  %run "/scalable-data-science/sds-2-x/009_PowerPlantPipeline_01ETLEDA"
  ```
  
   * *Note:* If you already evaluated the `%run ...` command above then:
   
     * first delete the cell by pressing on `x` on the top-right corner of the cell and 
     * revaluate the `run` command above.

%md
***
***

%md
# Now we will do the following Steps:
## Step 5: Data Preparation, 
## Step 6: Modeling, and 
## Step 7: Tuning and Evaluation 

We will do *Step 8: Deployment* later after we get introduced to SparkStreaming.

%md
##Step 5: Data Preparation

The next step is to prepare the data. Since all of this data is numeric and consistent, this is a simple task for us today.

We will need to convert the predictor features from columns to Feature Vectors using the org.apache.spark.ml.feature.VectorAssembler

The VectorAssembler will be the first step in building our ML pipeline.

%md
##Step 6: Data Modeling
Now let's model our data to predict what the power output will be given a set of sensor readings

Our first model will be based on simple linear regression since we saw some linear patterns in our data based on the scatter plots during the exploration stage.

%md
### Linear Regression Model

* Linear Regression is one of the most useful work-horses of statistical learning
* See Chapter 7 of Kevin Murphy's Machine Learning froma  Probabilistic Perspective for a good mathematical and algorithmic introduction.
* You should have already seen Ameet's treatment of the topic from earlier notebook.

%md
Let's open [http://spark.apache.org/docs/latest/mllib-linear-methods.html#regression](http://spark.apache.org/docs/latest/mllib-linear-methods.html#regression) for some details.

%md
Let's take a few elements of the three DataFrames.

%md
The cell below is based on the Spark ML pipeline API. More information can be found in the Spark ML Programming Guide at https://spark.apache.org/docs/latest/ml-guide.html

%md 
Since Linear Regression is simply a line of best fit over the data that minimizes the square of the error, given multiple input dimensions we can express each predictor as a line function of the form:

$$ y = b_0 + b_1 x_1 + b_2 x_2 + b_3 x_3 + \ldots + b_i x_i + \ldots + b_k x_k $$

where \\(b_0\\) is the intercept and \\(b_i\\)'s are coefficients.

To express the coefficients of that line we can retrieve the Estimator stage from the fitted, linear-regression pipeline model named `lrModel` and express the weights and the intercept for the function.

%md
The model has been fit and the intercept and coefficients displayed above.

Now, let us do some work to make a string of the model that is easy to understand for an applied data scientist or data analyst.

%md
Based on examining the fitted Linear Regression Equation above:

* There is a strong negative correlation between Atmospheric Temperature (AT) and Power Output due to the coefficient being greater than -1.91.
* But our other dimenensions seem to have little to no correlation with Power Output. 

Do you remember **Step 2: Explore Your Data**? When we visualized each predictor against Power Output using a Scatter Plot, only the temperature variable seemed to have a linear correlation with Power Output so our final equation seems logical.


Now let's see what our predictions look like given this model.

%md
Now that we have real predictions we can use an evaluation metric such as Root Mean Squared Error to validate our regression model. The lower the Root Mean Squared Error, the better our model.

%md
Generally a good model will have 68% of predictions within 1 RMSE and 95% within 2 RMSE of the actual value. Let's calculate and see if our RMSE meets this criteria.