%md

# [SDS-2.2, Scalable Data Science](https://lamastex.github.io/scalable-data-science/sds/2/2/)

%md 
#Power Plant ML Pipeline Application
This is an end-to-end example of using a number of different machine learning algorithms to solve a supervised regression problem.

###Table of Contents

- *Step 1: Business Understanding*
- *Step 2: Load Your Data*
- *Step 3: Explore Your Data*
- *Step 4: Visualize Your Data*
- *Step 5: Data Preparation*
- *Step 6: Data Modeling*
- *Step 7: Tuning and Evaluation*
- *Step 8: Deployment*



*We are trying to predict power output given a set of readings from various sensors in a gas-fired power generation plant.  Power generation is a complex process, and understanding and predicting power output is an important element in managing a plant and its connection to the power grid.*

More information about Peaker or Peaking Power Plants can be found on Wikipedia https://en.wikipedia.org/wiki/Peaking_power_plant


Given this business problem, we need to translate it to a Machine Learning task.  The ML task is regression since the label (or target) we are trying to predict is numeric.


The example data is provided by UCI at [UCI Machine Learning Repository Combined Cycle Power Plant Data Set](https://archive.ics.uci.edu/ml/datasets/Combined+Cycle+Power+Plant)

You can read the background on the UCI page, but in summary we have collected a number of readings from sensors at a Gas Fired Power Plant

(also called a Peaker Plant) and now we want to use those sensor readings to predict how much power the plant will generate.


More information about Machine Learning with Spark can be found in the [Spark MLLib Programming Guide](https://spark.apache.org/docs/latest/mllib-guide.html)


*Please note this example only works with Spark version 1.4 or higher*

%md
***
***

%md
To **Rerun Steps 1-4** done in the notebook at:

* [Workspace -> scalable-data-science -> sds-2-2 -> 009_PowerPlantPipeline_01ETLEDA](/#workspace/scalable-data-science/sds-2-2/009_PowerPlantPipeline_01ETLEDA)

just `run` the following command as shown in the cell below: 

  ```%scala
  %run "/scalable-data-science/sds-2-2/009_PowerPlantPipeline_01ETLEDA"
  ```
  
   * *Note:* If you already evaluated the `%run ...` command above then:
   
     * first delete the cell by pressing on `x` on the top-right corner of the cell and 
     * revaluate the `run` command above.

%md
***
***

%md
# Now we will do the following Steps:
## Step 5: Data Preparation, 
## Step 6: Modeling, and 
## Step 7: Tuning and Evaluation 

We will do *Step 8: Deployment* later after we get introduced to SparkStreaming.

%md
##Step 5: Data Preparation

The next step is to prepare the data. Since all of this data is numeric and consistent, this is a simple task for us today.

We will need to convert the predictor features from columns to Feature Vectors using the org.apache.spark.ml.feature.VectorAssembler

The VectorAssembler will be the first step in building our ML pipeline.

%md
##Step 6: Data Modeling
Now let's model our data to predict what the power output will be given a set of sensor readings

Our first model will be based on simple linear regression since we saw some linear patterns in our data based on the scatter plots during the exploration stage.

%md
### Linear Regression Model

* Linear Regression is one of the most useful work-horses of statistical learning
* See Chapter 7 of Kevin Murphy's Machine Learning froma  Probabilistic Perspective for a good mathematical and algorithmic introduction.
* You should have already seen Ameet's treatment of the topic from earlier notebook.

%md
Let's open [http://spark.apache.org/docs/latest/mllib-linear-methods.html#regression](http://spark.apache.org/docs/latest/mllib-linear-methods.html#regression) for some details.

%md
Let's take a few elements of the three DataFrames.