%md
# [SDS-2.x, Scalable Data Engineering Science](https://lamastex.github.io/scalable-data-science/sds/2/x/)

%md

# Twitter Hashtag Count

Using Twitter Streaming is a great way to learn Spark Streaming if you don't have your streaming datasource and want a great rich input dataset to try Spark Streaming transformations on.

In this example, we show how to calculate the top hashtags seen in the last X window of time every Y time unit.

Extracting knowledge from tweets is "easy" using techniques shown here, but one has to take responsibility for the use of this knowledge and conform to the rules and policies linked below.

Remeber that the use of twitter itself comes with various strings attached. Read:

- [Twitter Rules](https://twitter.com/rules)


Crucially, the use of the content from twitter by you (as done in this worksheet) comes with some strings.  Read:
- [Developer Agreement & Policy Twitter Developer Agreement](https://dev.twitter.com/overview/terms/agreement-and-policy)

%md

### Step 1: Enter your Twitter API Credentials.
* Go to https://apps.twitter.com and look up your Twitter API Credentials, or create an app to create them.
* Run the code in a cell to Enter your own credentials.

```%scala
// put your own twitter developer credentials below instead of xxx
// instead of the '%run "scalable-data-science/secrets/026_secret_MyTwitterOAuthCredentials"' above
// this notebook we just ran contains the following commented code block

/*
import twitter4j.auth.OAuthAuthorization
import twitter4j.conf.ConfigurationBuilder

def MyconsumerKey       = "xxx"
def MyconsumerSecret    = "xxx"
def Mytoken             = "xxx"
def MytokenSecret       = "xxx"

System.setProperty("twitter4j.oauth.consumerKey", MyconsumerKey)
System.setProperty("twitter4j.oauth.consumerSecret", MyconsumerSecret)
System.setProperty("twitter4j.oauth.accessToken", Mytoken)
System.setProperty("twitter4j.oauth.accessTokenSecret", MytokenSecret)
*/
```

The cell-below is hidden to not expose my Twitter API Credentials: `consumerKey`, `consumerSecret`, `accessToken` and `accessTokenSecret`. Use the code above to enter your own credentials!

%md
If you see warnings then ignore for now:
[https://forums.databricks.com/questions/6941/change-in-getargument-for-notebook-input.html](https://forums.databricks.com/questions/6941/change-in-getargument-for-notebook-input.html).

%md

### Step 2: Configure where to output the top hashtags and how often to compute them.

* Run this cell for the input cells to appear.
* Enter your credentials.
* Run the cell again to pick up your defaults.

%md

### Step 3: Run the Twitter Streaming job.

%md
Clean up any old files.

%md 
Let us write the function that creates the Streaming Context and sets up the streaming job.

%md 
Create the StreamingContext using getActiveOrCreate, as required when starting a streaming job in Databricks.

%md

Start the Spark Streaming Context and return when the Streaming job exits or return with the specified timeout.  

%md
Check out the Clusters `Streaming` UI as the job is running.

%md

It should automatically stop the streaming job after `timeoutJobLength`.

If not, then stop any active Streaming Contexts, but don't stop the spark contexts they are attached to using the following command.

%md

### Step 4: View the Results.

%md
There should be 100 intervals for each second and the top hashtags for each of them should be in the file `top_hashtags_N` for `N` in 0,1,2,...,99 and the top hashtags should be based on the past 5 seconds window.