// Databricks notebook source exported at Sun, 19 Jun 2016 03:06:55 UTC

# Scalable Data Science

### prepared by Raazesh Sainudiin and Sivanand Sivaram

This is an elaboration of the Apache Spark 1.6 mllib-progamming-guide on mllib-data-types.

# Overview

## Data Types - MLlib Programming Guide

- Local vector and URL
- Labeled point and URL
- Local matrix and URL
- Distributed matrix and URL
- RowMatrix and URL
- IndexedRowMatrix and URL
- CoordinateMatrix and URL
- BlockMatrix and URL

MLlib supports local vectors and matrices stored on a single machine, as well as distributed matrices backed by one or more RDDs. Local vectors and local matrices are simple data models that serve as public interfaces. The underlying linear algebra operations are provided by Breeze and jblas. A training example used in supervised learning is called a “labeled point” in MLlib.

### BlockMatrix in Scala

A `BlockMatrix`

is a distributed matrix backed by an RDD of
`MatrixBlock`

s, where a `MatrixBlock`

is a tuple of
`((Int, Int), Matrix)`

, where the `(Int, Int)`

is the index of the
block, and `Matrix`

is the sub-matrix at the given index with size
`rowsPerBlock`

x `colsPerBlock`

. `BlockMatrix`

supports methods such as
`add`

and `multiply`

with another `BlockMatrix`

. `BlockMatrix`

also has
a helper function `validate`

which can be used to check whether the
`BlockMatrix`

is set up properly.

A `BlockMatrix`

can be most easily created from an `IndexedRowMatrix`

or
`CoordinateMatrix`

by calling `toBlockMatrix`

. `toBlockMatrix`

creates
blocks of size 1024 x 1024 by default. Users may change the block size
by supplying the values through
`toBlockMatrix(rowsPerBlock, colsPerBlock)`

.

Refer to the `BlockMatrix`

Scala docs
for details on the API.

```
//import org.apache.spark.mllib.linalg.{Matrix, Matrices}
import org.apache.spark.mllib.linalg.distributed.{BlockMatrix, CoordinateMatrix, MatrixEntry}
```

```
val entries: RDD[MatrixEntry] = sc.parallelize(Array(MatrixEntry(0, 0, 1.2), MatrixEntry(1, 0, 2.1), MatrixEntry(6, 1, 3.7))) // an RDD of matrix entries
```

```
// Create a CoordinateMatrix from an RDD[MatrixEntry].
val coordMat: CoordinateMatrix = new CoordinateMatrix(entries)
```

```
// Transform the CoordinateMatrix to a BlockMatrix
val matA: BlockMatrix = coordMat.toBlockMatrix().cache()
```

```
// Validate whether the BlockMatrix is set up properly. Throws an Exception when it is not valid.
// Nothing happens if it is valid.
matA.validate()
```

```
// Calculate A^T A.
val ata = matA.transpose.multiply(matA)
```

```
ata.blocks.collect()
```

```
ata.toLocalMatrix()
```

### BlockMatrix in Scala

A `BlockMatrix`

can be created from an `RDD`

of sub-matrix blocks, where a sub-matrix
block is a `((blockRowIndex, blockColIndex), sub-matrix)`

tuple.

Refer to the `BlockMatrix`

Python docs
for more details on the API.

```
%py
from pyspark.mllib.linalg import Matrices
from pyspark.mllib.linalg.distributed import BlockMatrix
# Create an RDD of sub-matrix blocks.
blocks = sc.parallelize([((0, 0), Matrices.dense(3, 2, [1, 2, 3, 4, 5, 6])),
((1, 0), Matrices.dense(3, 2, [7, 8, 9, 10, 11, 12]))])
# Create a BlockMatrix from an RDD of sub-matrix blocks.
mat = BlockMatrix(blocks, 3, 2)
# Get its size.
m = mat.numRows() # 6
n = mat.numCols() # 2
print (m,n)
# Get the blocks as an RDD of sub-matrix blocks.
blocksRDD = mat.blocks
# Convert to a LocalMatrix.
localMat = mat.toLocalMatrix()
# Convert to an IndexedRowMatrix.
indexedRowMat = mat.toIndexedRowMatrix()
# Convert to a CoordinateMatrix.
coordinateMat = mat.toCoordinateMatrix()
```