ScaDaMaLe Course site and book

Power Plant ML Pipeline Application

This is an end-to-end example of using a number of different machine learning algorithms to solve a supervised regression problem.

Table of Contents

• Step 1: Business Understanding
• Step 2: Load Your Data
• Step 3: Explore Your Data
• Step 4: Visualize Your Data
• Step 5: Data Preparation
• Step 6: Data Modeling
• Step 7: Tuning and Evaluation
• Step 8: Deployment

We are trying to predict power output given a set of readings from various sensors in a gas-fired power generation plant. Power generation is a complex process, and understanding and predicting power output is an important element in managing a plant and its connection to the power grid.

More information about Peaker or Peaking Power Plants can be found on Wikipedia https://en.wikipedia.org/wiki/Peakingpowerplant

Given this business problem, we need to translate it to a Machine Learning task. The ML task is regression since the label (or target) we are trying to predict is numeric.

The example data is provided by UCI at UCI Machine Learning Repository Combined Cycle Power Plant Data Set

You can read the background on the UCI page, but in summary we have collected a number of readings from sensors at a Gas Fired Power Plant

(also called a Peaker Plant) and now we want to use those sensor readings to predict how much power the plant will generate.

More information about Machine Learning with Spark can be found in the Spark MLLib Programming Guide

Please note this example only works with Spark version 1.4 or higher

---

---

To Rerun Steps 1-4 done in the notebook at:

• Workspace -> PATH_TO -> 009_PowerPlantPipeline_01ETLEDA]

just run the following command as shown in the cell below:

%run "/PATH_TO/009_PowerPlantPipeline_01ETLEDA"

• Note: If you already evaluated the %run ... command above then:

  • first delete the cell by pressing on x on the top-right corner of the cell and
  • revaluate the run command above.

"../000_1-sds-3-x-sql/009_PowerPlantPipeline_01ETLEDA" 

---

---

Now we will do the following Steps:

Step 5: Data Preparation,

Step 6: Modeling, and

Step 7: Tuning and Evaluation

We will do Step 8: Deployment later after we get introduced to SparkStreaming.

Step 5: Data Preparation

The next step is to prepare the data. Since all of this data is numeric and consistent, this is a simple task for us today.

We will need to convert the predictor features from columns to Feature Vectors using the org.apache.spark.ml.feature.VectorAssembler

The VectorAssembler will be the first step in building our ML pipeline.

res2: Int = 321

//Let's quickly recall the schema and make sure our table is here now
table("power_plant_table").printSchema

root
 |-- AT: double (nullable = true)
 |-- V: double (nullable = true)
 |-- AP: double (nullable = true)
 |-- RH: double (nullable = true)
 |-- PE: double (nullable = true)

powerPlantDF // make sure we have the DataFrame too

res21: org.apache.spark.sql.DataFrame = [AT: double, V: double ... 3 more fields]

powerPlantRDD: org.apache.spark.rdd.RDD[String] = /datasets/sds/power-plant/data/Sheet1.tsv MapPartitionsRDD[1] at textFile at command-2971213210277530:1

AT	V	AP	RH	PE
14.96	41.76	1024.07	73.17	463.26
25.18	62.96	1020.04	59.08	444.37
5.11	39.4	1012.16	92.14	488.56
20.86	57.32	1010.24	76.64	446.48

powerPlantDF: org.apache.spark.sql.DataFrame = [AT: double, V: double ... 3 more fields]

import org.apache.spark.ml.feature.VectorAssembler

// make a DataFrame called dataset from the table
val dataset = sqlContext.table("power_plant_table") 

val vectorizer =  new VectorAssembler()
                      .setInputCols(Array("AT", "V", "AP", "RH"))
                      .setOutputCol("features")

import org.apache.spark.ml.feature.VectorAssembler
dataset: org.apache.spark.sql.DataFrame = [AT: double, V: double ... 3 more fields]
vectorizer: org.apache.spark.ml.feature.VectorAssembler = VectorAssembler: uid=vecAssembler_38ac3d0b0ea7, handleInvalid=error, numInputCols=4

root
 |-- AT: double (nullable = true)
 |-- V: double (nullable = true)
 |-- AP: double (nullable = true)
 |-- RH: double (nullable = true)
 |-- PE: double (nullable = true)

res9: Long = 9568

+-----+-----+-------+-----+------+
|   AT|    V|     AP|   RH|    PE|
+-----+-----+-------+-----+------+
|14.96|41.76|1024.07|73.17|463.26|
|25.18|62.96|1020.04|59.08|444.37|
| 5.11| 39.4|1012.16|92.14|488.56|
|20.86|57.32|1010.24|76.64|446.48|
|10.82| 37.5|1009.23|96.62| 473.9|
|26.27|59.44|1012.23|58.77|443.67|
|15.89|43.96|1014.02|75.24|467.35|
| 9.48|44.71|1019.12|66.43|478.42|
|14.64| 45.0|1021.78|41.25|475.98|
|11.74|43.56|1015.14|70.72| 477.5|
+-----+-----+-------+-----+------+
only showing top 10 rows

res12: Long = 9568

+-----------------+--------+-----------+---------+-----------+
|             name|database|description|tableType|isTemporary|
+-----------------+--------+-----------+---------+-----------+
|power_plant_table|    null|       null|TEMPORARY|       true|
+-----------------+--------+-----------+---------+-----------+

Step 6: Data Modeling

Now let's model our data to predict what the power output will be given a set of sensor readings

Our first model will be based on simple linear regression since we saw some linear patterns in our data based on the scatter plots during the exploration stage.