%md
# [SDS-2.x, Scalable Data Engineering Science](https://lamastex.github.io/scalable-data-science/sds/2/x/)

%md
#Topic Modeling with Latent Dirichlet Allocation

This is an augmentation of a notebook from Databricks Guide.   
This notebook will provide a brief algorithm summary, links for further reading, and an example of how to use LDA for Topic Modeling.

%md
##Algorithm Summary

- **Task**: Identify topics from a collection of text documents
- **Input**: Vectors of word counts
- **Optimizers**: 
    - EMLDAOptimizer using [Expectation Maximization](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm)
    - OnlineLDAOptimizer using Iterative Mini-Batch Sampling for [Online Variational Bayes](https://www.cs.princeton.edu/~blei/papers/HoffmanBleiBach2010b.pdf)

%md
## Intro to LDA by David Blei
Watch at least the first 25 or so minutes of this video by David Blei on a crash introduction to topic modeling via Latent Dirichlet Allocation (LDA).

[![AJ's What is Collaborative Filtering](http://img.youtube.com/vi/FkckgwMHP2s/0.jpg)](https://www.youtube.com/watch?v=FkckgwMHP2s)

%md
## Links

- Spark API docs
  - Scala: [LDA](https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.mllib.clustering.LDA)
  - Python: [LDA](https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.clustering.LDA)
- [MLlib Programming Guide](http://spark.apache.org/docs/latest/mllib-clustering.html#latent-dirichlet-allocation-lda)
- [ML Feature Extractors & Transformers](http://spark.apache.org/docs/latest/ml-features.html)
- [Wikipedia: Latent Dirichlet Allocation](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation)

%md
## Readings for LDA

* A high-level introduction to the topic from Communications of the ACM
    * [http://www.cs.columbia.edu/~blei/papers/Blei2012.pdf](http://www.cs.columbia.edu/~blei/papers/Blei2012.pdf)
* A very good high-level humanities introduction to the topic (recommended by Chris Thomson in English Department at UC, Ilam): 
    * [http://journalofdigitalhumanities.org/2-1/topic-modeling-and-digital-humanities-by-david-m-blei/](http://journalofdigitalhumanities.org/2-1/topic-modeling-and-digital-humanities-by-david-m-blei/)

Also read the methodological and more formal papers cited in the above links if you want to know more.

%md

Let's get a bird's eye view of LDA from [http://www.cs.columbia.edu/~blei/papers/Blei2012.pdf](http://www.cs.columbia.edu/~blei/papers/Blei2012.pdf) next.

* See pictures (hopefully you read the paper last night!)
* Algorithm of the generative model (this is unsupervised clustering)
* For a careful introduction to the topic see Section 27.3 and 27.4 (pages 950-970) pf Murphy's *Machine Learning: A Probabilistic Perspective, MIT Press, 2012*. 
* We will be quite application focussed or applied here!
* Understand Expectation Maximization Algorithm read *Section 8.5 The EM Algorithm* in *The Elements of Statistical Learning* by Hastie, Tibshirani and Freidman (2001, Springer Series in Statistics). Read from free 21MB PDF of the book available from here [https://web.stanford.edu/~hastie/Papers/ESLII.pdf](https://web.stanford.edu/~hastie/Papers/ESLII.pdf) or from its backup here [http://lamastex.org/research_events/Readings/StatLearn/ESLII.pdf](http://lamastex.org/research_events/Readings/StatLearn/ESLII.pdf).

//This allows easy embedding of publicly available information into any other notebook
//when viewing in git-book just ignore this block - you may have to manually chase the URL in frameIt("URL").
//Example usage:
// displayHTML(frameIt("https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation#Topics_in_LDA",250))
def frameIt( u:String, h:Int ) : String = {
      """<iframe 
 src=""""+ u+""""
 width="95%" height="""" + h + """"
 sandbox>
  <p>
    <a href="http://spark.apache.org/docs/latest/index.html">
      Fallback link for browsers that, unlikely, don't support frames
    </a>
  </p>
</iframe>"""
   }
displayHTML(frameIt("https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation#Topics_in_LDA",200))

displayHTML(frameIt("https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation#Model",600))

displayHTML(frameIt("https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation#Mathematical_definition",910))

%md 
## Probabilistic Topic Modeling Example

This is an outline of our Topic Modeling workflow. Feel free to jump to any subtopic to find out more.

- Step 0. Dataset Review
- Step 1. Downloading and Loading Data into DBFS 
    - (Step 1. only needs to be done once per shard - see details at the end of the notebook for Step 1.)
- Step 2. Loading the Data and Data Cleaning 
- Step 3. Text Tokenization
- Step 4. Remove Stopwords
- Step 5. Vector of Token Counts
- Step 6. Create LDA model with Online Variational Bayes
- Step 7. Review Topics
- Step 8. Model Tuning - Refilter Stopwords
- Step 9. Create LDA model with Expectation Maximization
- Step 10. Visualize Results

%md 

## Step 0. Dataset Review

In this example, we will use the mini [20 Newsgroups dataset](http://kdd.ics.uci.edu/databases/20newsgroups/20newsgroups.html), which is a random subset of the original 20 Newsgroups dataset. Each newsgroup is stored in a subdirectory, with each article stored as a separate file.

***
***

The following is the markdown file `20newsgroups.data.md` of the original details on the dataset, obtained as follows:

```%sh
$ wget -k http://kdd.ics.uci.edu/databases/20newsgroups/20newsgroups.data.html
--2016-04-07 10:31:51--  http://kdd.ics.uci.edu/databases/20newsgroups/20newsgroups.data.html
Resolving kdd.ics.uci.edu (kdd.ics.uci.edu)... 128.195.1.95
Connecting to kdd.ics.uci.edu (kdd.ics.uci.edu)|128.195.1.95|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4371 (4.3K) [text/html]
Saving to: '20newsgroups.data.html’

100%[======================================>] 4,371       --.-K/s   in 0s      

2016-04-07 10:31:51 (195 MB/s) - '20newsgroups.data.html’ saved [4371/4371]

Converting 20newsgroups.data.html... nothing to do.
Converted 1 files in 0 seconds.

$ pandoc -f html -t markdown 20newsgroups.data.html > 20newsgroups.data.md
```
### 20 Newsgroups

#### Data Type

text

#### Abstract

This data set consists of 20000 messages taken from 20 newsgroups.

#### Sources

##### Original Owner and Donor

    Tom Mitchell
    School of Computer Science
    Carnegie Mellon University
    tom.mitchell@cmu.edu

**Date Donated:** September 9, 1999

#### Data Characteristics

One thousand Usenet articles were taken from each of the following 20
newsgroups.

        alt.atheism
        comp.graphics
        comp.os.ms-windows.misc
        comp.sys.ibm.pc.hardware
        comp.sys.mac.hardware
        comp.windows.x
        misc.forsale
        rec.autos
        rec.motorcycles
        rec.sport.baseball
        rec.sport.hockey
        sci.crypt
        sci.electronics
        sci.med
        sci.space
        soc.religion.christian
        talk.politics.guns
        talk.politics.mideast
        talk.politics.misc
        talk.religion.misc

Approximately 4% of the articles are crossposted. The articles are
typical postings and thus have headers including subject lines,
signature files, and quoted portions of other articles.

#### Data Format

Each newsgroup is stored in a subdirectory, with each article stored as
a separate file.

#### Past Usage

T. Mitchell. Machine Learning, McGraw Hill, 1997.

T. Joachims (1996). [A probabilistic analysis of the Rocchio algorithm
with TFIDF for text
categorization](http://reports-archive.adm.cs.cmu.edu/anon/1996/CMU-CS-96-118.ps),
Computer Science Technical Report CMU-CS-96-118. Carnegie Mellon
University.

#### Acknowledgements, Copyright Information, and Availability

You may use this material free of charge for any educational purpose,
provided attribution is given in any lectures or publications that make
use of this material.

#### References and Further Information

Naive Bayes code for text classification is available from:
[http://www.cs.cmu.edu/afs/cs/project/theo-11/www/naive-bayes.html](http://www.cs.cmu.edu/afs/cs/project/theo-11/www/naive-bayes.html)

* * * * *

[The UCI KDD Archive](http://kdd.ics.uci.edu/) \
 [Information and Computer Science](http://www.ics.uci.edu/) \
 [University of California, Irvine](http://www.uci.edu/) \
 Irvine, CA 92697-3425 \

Last modified: September 9, 1999

***
***

%md
**NOTE:** The mini dataset consists of 100 articles from the following 20 Usenet newsgroups:

    alt.atheism
    comp.graphics
    comp.os.ms-windows.misc
    comp.sys.ibm.pc.hardware
    comp.sys.mac.hardware
    comp.windows.x
    misc.forsale
    rec.autos
    rec.motorcycles
    rec.sport.baseball
    rec.sport.hockey
    sci.crypt
    sci.electronics
    sci.med
    sci.space
    soc.religion.christian
    talk.politics.guns
    talk.politics.mideast
    talk.politics.misc
    talk.religion.misc

Some of the newsgroups seem pretty similar on first glance, such as *comp.sys.ibm.pc.hardware* and *comp.sys.mac.hardware*, which may affect our results.

%md

**NOTE:** A simpler and slicker version of the analysis is available in this notebook:

* [https://docs.cloud.databricks.com/docs/latest/sample_applications/07%20Sample%20ML/MLPipeline%20Newsgroup%20Dataset.html](https://docs.cloud.databricks.com/docs/latest/sample_applications/07%20Sample%20ML/MLPipeline%20Newsgroup%20Dataset.html)
   
   But, let's do it the hard way here so that we can do it on other arbitrary datasets.

%md 
## Step 2. Loading the Data and Data Cleaning

We have already used the wget command to download the file, and put it in our distributed file system (this process takes about 10 minutes). To repeat these steps or to download data from another source follow the steps at the bottom of this worksheet on **Step 1. Downloading and Loading Data into DBFS**.

Let's make sure these files are in dbfs now:

%md
Now let us read in the data using `wholeTextFiles()`.

Recall that the `wholeTextFiles()` command will read in the entire directory of text files, and return a key-value pair of (filePath, fileContent).

As we do not need the file paths in this example, we will apply a map function to extract the file contents, and then convert everything to lowercase.

%md
Review first 5 documents to get a sense for the data format.

%md
To review a random document in the corpus uncomment and evaluate the following cell.

%md
Note that the document begins with a header containing some metadata that we don't need, and we are only interested in the body of the document. We can do a bit of simple data cleaning here by removing the metadata of each document, which reduces the noise in our dataset. This is an important step as the accuracy of our models depend greatly on the quality of data used.

%md
Let's review first 5 documents with metadata removed.

%md 
## Feature extraction and transformation APIs

%md
See [http://spark.apache.org/docs/latest/ml-features.html](http://spark.apache.org/docs/latest/ml-features.html)

%md 
To use the convenient [Feature extraction and transformation APIs](http://spark.apache.org/docs/latest/ml-features.html), we will convert our RDD into a DataFrame.

We will also create an ID for every document using `zipWithIndex` 

  * for sytax and details search for `zipWithIndex` in [https://spark.apache.org/docs/latest/api/scala/org/apache/spark/rdd/RDD.html](https://spark.apache.org/docs/latest/api/scala/org/apache/spark/rdd/RDD.html)

%md 
## Step 3. Text Tokenization

We will use the RegexTokenizer to split each document into tokens. We can setMinTokenLength() here to indicate a minimum token length, and filter away all tokens that fall below the minimum.

%md
See [http://spark.apache.org/docs/latest/ml-features.html#tokenizer](http://spark.apache.org/docs/latest/ml-features.html#tokenizer).

%md 
## Step 4. Remove Stopwords

We can easily remove stopwords using the StopWordsRemover(). 

%md
See [http://spark.apache.org/docs/latest/ml-features.html#stopwordsremover](http://spark.apache.org/docs/latest/ml-features.html#stopwordsremover).

%md
If a list of stopwords is not provided, the StopWordsRemover() will use [this list of stopwords](http://ir.dcs.gla.ac.uk/resources/linguistic_utils/stop_words), also shown below, by default. 

``` a,about,above,across,after,afterwards,again,against,all,almost,alone,along,already,also,although,always,am,among,amongst,amoungst,amount,an,and,another,any,anyhow,anyone,anything,anyway,anywhere,
are,around,as,at,back,be,became,because,become,becomes,becoming,been,before,beforehand,behind,being,below,beside,besides,between,beyond,bill,both,bottom,but,by,call,can,cannot,cant,co,computer,con,could,
couldnt,cry,de,describe,detail,do,done,down,due,during,each,eg,eight,either,eleven,else,elsewhere,empty,enough,etc,even,ever,every,everyone,everything,everywhere,except,few,fifteen,fify,fill,find,fire,first,
five,for,former,formerly,forty,found,four,from,front,full,further,get,give,go,had,has,hasnt,have,he,hence,her,here,hereafter,hereby,herein,hereupon,hers,herself,him,himself,his,how,however,hundred,i,ie,if,
in,inc,indeed,interest,into,is,it,its,itself,keep,last,latter,latterly,least,less,ltd,made,many,may,me,meanwhile,might,mill,mine,more,moreover,most,mostly,move,much,must,my,myself,name,namely,neither,never,
nevertheless,next,nine,no,nobody,none,noone,nor,not,nothing,now,nowhere,of,off,often,on,once,one,only,onto,or,other,others,otherwise,our,ours,ourselves,out,over,own,part,per,perhaps,please,put,rather,re,same,
see,seem,seemed,seeming,seems,serious,several,she,should,show,side,since,sincere,six,sixty,so,some,somehow,someone,something,sometime,sometimes,somewhere,still,such,system,take,ten,than,that,the,their,them,
themselves,then,thence,there,thereafter,thereby,therefore,therein,thereupon,these,they,thick,thin,third,this,those,though,three,through,throughout,thru,thus,to,together,too,top,toward,towards,twelve,twenty,two,
un,under,until,up,upon,us,very,via,was,we,well,were,what,whatever,when,whence,whenever,where,whereafter,whereas,whereby,wherein,whereupon,wherever,whether,which,while,whither,who,whoever,whole,whom,whose,why,will,
with,within,without,would,yet,you,your,yours,yourself,yourselves
```

You can use `getStopWords()` to see the list of stopwords that will be used.

In this example, we will specify a list of stopwords for the StopWordsRemover() to use. We do this so that we can add on to the list later on.

%md
If the file `dbfs:/tmp/stopwords` already exists then skip the next two cells, otherwise download and load it into DBFS by uncommenting and evaluating the next two cells.

%md
Finally, we can just remove the stopwords using the `StopWordsRemover` as follows:

%md 
## Step 5. Vector of Token Counts

LDA takes in a vector of token counts as input. We can use the `CountVectorizer()` to easily convert our text documents into vectors of token counts.

The `CountVectorizer` will return `(VocabSize, Array(Indexed Tokens), Array(Token Frequency))`.

Two handy parameters to note:

  - `setMinDF`: Specifies the minimum number of different documents a term must appear in to be included in the vocabulary.
  - `setMinTF`: Specifies the minimum number of times a term has to appear in a document to be included in the vocabulary.

%md
See [http://spark.apache.org/docs/latest/ml-features.html#countvectorizer](http://spark.apache.org/docs/latest/ml-features.html#countvectorizer).

%md 
To use the LDA algorithm in the MLlib library, we have to convert the DataFrame back into an RDD.

%md
## Let's get an overview of LDA in Spark's MLLIB

%md
See [http://spark.apache.org/docs/latest/mllib-clustering.html#latent-dirichlet-allocation-lda](http://spark.apache.org/docs/latest/mllib-clustering.html#latent-dirichlet-allocation-lda)

%md 
## Create LDA model with Online Variational Bayes

We will now set the parameters for LDA. We will use the OnlineLDAOptimizer() here, which implements Online Variational Bayes.

Choosing the number of topics for your LDA model requires a bit of domain knowledge. As we know that there are 20 unique newsgroups in our dataset, we will set numTopics to be 20.

%md
We will set the parameters needed to build our LDA model. We can also setMiniBatchFraction for the OnlineLDAOptimizer, which sets the fraction of corpus sampled and used at each iteration. In this example, we will set this to 0.8.

%md 
Create the LDA model with Online Variational Bayes.

%md
Watch **Online Learning for Latent Dirichlet Allocation** in NIPS2010 by Matt Hoffman (right click and open in new tab)

[![Matt Hoffman's NIPS 2010 Talk Online LDA]](http://videolectures.net/nips2010_hoffman_oll/thumb.jpg)](http://videolectures.net/nips2010_hoffman_oll/)
  
Also see the paper on *Online varioational Bayes* by Matt linked for more details (from the above URL): [http://videolectures.net/site/normal_dl/tag=83534/nips2010_1291.pdf](http://videolectures.net/site/normal_dl/tag=83534/nips2010_1291.pdf)

%md 
Note that using the OnlineLDAOptimizer returns us a [LocalLDAModel](http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.mllib.clustering.LocalLDAModel), which stores the inferred topics of your corpus.

%md 
## Review Topics

We can now review the results of our LDA model. We will print out all 20 topics with their corresponding term probabilities.

Note that you will get slightly different results every time you run an LDA model since LDA includes some randomization.

Let us review results of LDA model with Online Variational Bayes, step by step.

%md
Feel free to take things apart to understand!

%md
Review Results of LDA model with Online Variational Bayes - Doing all four steps earlier at once.

%md
Going through the results, you may notice that some of the topic words returned are actually stopwords that are specific to our dataset (for eg: "writes", "article"...). Let's try improving our model.

%md 
## Step 8. Model Tuning - Refilter Stopwords

We will try to improve the results of our model by identifying some stopwords that are specific to our dataset. We will filter these stopwords out and rerun our LDA model to see if we get better results.

%md
We will also increase MaxIterations to 10 to see if we get better results.

%md
#### How to find what the default values are?

Dive into the source!!!

1. Let's find the default value for `docConcentration` now.
* Got to Apache Spark package Root: [https://spark.apache.org/docs/latest/api/scala/#package](https://spark.apache.org/docs/latest/api/scala/#package)
* search for 'ml' in the search box on the top left (ml is for ml library)
* Then find the `LDA` by scrolling below on the left to mllib's `clustering` methods and click on `LDA`
* Then click on the source code link which should take you here:
  * [https://github.com/apache/spark/blob/v2.2.0/mllib/src/main/scala/org/apache/spark/ml/clustering/LDA.scala](https://github.com/apache/spark/blob/v2.2.0/mllib/src/main/scala/org/apache/spark/ml/clustering/LDA.scala)
  * Now, simply go to the right function and see the following comment block:
  
  ```
  /**
   * Concentration parameter (commonly named "alpha") for the prior placed on documents'
   * distributions over topics ("theta").
   *
   * This is the parameter to a Dirichlet distribution, where larger values mean more smoothing
   * (more regularization).
   *
   * If not set by the user, then docConcentration is set automatically. If set to
   * singleton vector [alpha], then alpha is replicated to a vector of length k in fitting.
   * Otherwise, the [[docConcentration]] vector must be length k.
   * (default = automatic)
   *
   * Optimizer-specific parameter settings:
   *  - EM
   *     - Currently only supports symmetric distributions, so all values in the vector should be
   *       the same.
   *     - Values should be > 1.0
   *     - default = uniformly (50 / k) + 1, where 50/k is common in LDA libraries and +1 follows
   *       from Asuncion et al. (2009), who recommend a +1 adjustment for EM.
   *  - Online
   *     - Values should be >= 0
   *     - default = uniformly (1.0 / k), following the implementation from
   *       [[https://github.com/Blei-Lab/onlineldavb]].
   * @group param
   */
  ```

%md

**HOMEWORK:** Try to find the default value for `TopicConcentration`.

%md 
We managed to get better results here. We can easily infer that topic 3 is about space, topic 7 is about religion, etc.

```
==========
TOPIC 3
station 0.0022184815200582244
launch  0.0020621309179376145
shuttle 0.0019305627762549198
space   0.0017600147075534092
redesign    0.0014972130065346592
==========
TOPIC 7
people  0.0038165245379908675
church  0.0036902650900400543
jesus   0.0029942866750178893
paul    0.0026144777524277044
bible   0.0020476251853453016
==========
```

%md 
## Step 9. Create LDA model with Expectation Maximization

Let's try creating an LDA model with Expectation Maximization on the data that has been refiltered for additional stopwords. We will also increase MaxIterations here to 100 to see if that improves results.

%md
See [http://spark.apache.org/docs/latest/mllib-clustering.html#latent-dirichlet-allocation-lda](http://spark.apache.org/docs/latest/mllib-clustering.html#latent-dirichlet-allocation-lda)

%md
Note that the EMLDAOptimizer produces a DistributedLDAModel, which stores not only the inferred topics but also the full training corpus and topic distributions for each document in the training corpus.

%md
This is just doing it all at once.

%md 
We've managed to get some good results here. For example, we can easily infer that Topic 0 is about computers, Topic 8 is about space, etc. 


We still get some ambiguous results like Topic 17.

%md

To improve our results further, we could employ some of the below methods:

- Refilter data for additional data-specific stopwords
- Use Stemming or Lemmatization to preprocess data
- Experiment with a smaller number of topics, since some of these topics in the 20 Newsgroups are pretty similar
- Increase model's MaxIterations

%md 
## Visualize Results

We will try visualizing the results obtained from the EM LDA model with a d3 bubble chart.

%md 
We will convert the DataFrame into a JSON format, which will be passed into d3.

%md 
We are now ready to use D3 on the rawJson data.

displayHTML(s"""
<!DOCTYPE html>
<meta charset="utf-8">
<style>

circle {
  fill: rgb(31, 119, 180);
  fill-opacity: 0.5;
  stroke: rgb(31, 119, 180);
  stroke-width: 1px;
}

.leaf circle {
  fill: #ff7f0e;
  fill-opacity: 1;
}

text {
  font: 14px sans-serif;
}

</style>
<body>
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js"></script>
<script>

var json = {
 "name": "data",
 "children": [
  {
     "name": "topics",
     "children": [
      ${rawJson}
     ]
    }
   ]
};

var r = 1000,
    format = d3.format(",d"),
    fill = d3.scale.category20c();

var bubble = d3.layout.pack()
    .sort(null)
    .size([r, r])
    .padding(1.5);

var vis = d3.select("body").append("svg")
    .attr("width", r)
    .attr("height", r)
    .attr("class", "bubble");

  
var node = vis.selectAll("g.node")
    .data(bubble.nodes(classes(json))
    .filter(function(d) { return !d.children; }))
    .enter().append("g")
    .attr("class", "node")
    .attr("transform", function(d) { return "translate(" + d.x + "," + d.y + ")"; })
    color = d3.scale.category20();
  
  node.append("title")
      .text(function(d) { return d.className + ": " + format(d.value); });

  node.append("circle")
      .attr("r", function(d) { return d.r; })
      .style("fill", function(d) {return color(d.topicName);});

var text = node.append("text")
    .attr("text-anchor", "middle")
    .attr("dy", ".3em")
    .text(function(d) { return d.className.substring(0, d.r / 3)});
  
  text.append("tspan")
      .attr("dy", "1.2em")
      .attr("x", 0)
      .text(function(d) {return Math.ceil(d.value * 10000) /10000; });

// Returns a flattened hierarchy containing all leaf nodes under the root.
function classes(root) {
  var classes = [];

  function recurse(term, node) {
    if (node.children) node.children.forEach(function(child) { recurse(node.term, child); });
    else classes.push({topicName: node.topicId, className: node.term, value: node.probability});
  }

  recurse(null, root);
  return {children: classes};
}
</script>
""")

%md
### You try!

**NOW or Later as HOMEWORK** 

1. Try to do the same process for the State of the Union Addresses dataset from Week1. As a first step, first locate where that data is... Go to week1 and try to see if each SoU can be treated as a document for topic modeling and whether there is temporal clustering of SoU's within the same topic.

2. Try to improve the tuning by elaborating the pipeline with stemming, lemmatization, etc in this news-group dataset (if you want to do a project based on this, perhaps). You can also parse the input to bring in the newsgroup id's from the directories (consider exploiting the file names in the `wholeTextFiles` method) as this will let you explore how well your unsupervised algorithm is doing relative to the known newsgroups each document falls in (note you generally won't have the luxury of knowing the topic labels for typical datasets in the unsupervised topic modeling domain).

3. Try to parse the data closer to the clean dataset available in `/databricks-datasets/news20.binary/*` and walk through the following notebook (*but in Scala!*):
    * [https://docs.cloud.databricks.com/docs/latest/sample_applications/07%20Sample%20ML/MLPipeline%20Newsgroup%20Dataset.html](https://docs.cloud.databricks.com/docs/latest/sample_applications/07%20Sample%20ML/MLPipeline%20Newsgroup%20Dataset.html)
   

%md 
## Step 1. Downloading and Loading Data into DBFS

**you don't have to do the download in databricks if above cell has contents in `/databricks-datasets/news20.binary/data-001`**

Here are the steps taken for downloading and saving data to the distributed file system.  Uncomment them for repeating this process on your databricks cluster or for downloading a new source of data.