Databricks notebook source exported at Tue, 28 Jun 2016 10:38:24 UTC

Scalable Data Science

Student Project Presentation by Shanshan Zhou

supported by and

The html source url of this databricks notebook and its recorded Uji Image of Uji, Dogen's Time-Being:

sds/uji/studentProjects/05_ShanshanZhou/051_EEG_Explore

Identify hand motions from EEG recordings

by Shanshan Zhou

**Patients who have lost hand function due to amputation or neurological disabilities wake up to this reality everyday. **

  • Restoring a patient’s ability to perform these basic activities of daily life with a brain-computer interface (BCI) prosthetic device would greatly increase their independence and quality of life.
  • Currently, there are no realistic, affordable, or low-risk options for neurologically disabled patients to directly control external prosthetics with their brain activity.

A possible solution …

  • Recorded from the human scalp, EEG signals are evoked by brain activity.
  • The relationship between brain activity and EEG signals is complex and poorly understood outside of specific laboratory tests.
  • Providing affordable, low-risk, non-invasive BCI devices is dependent on further advancements in interpreting EEG signals.

A tutorial on how to process EEG data

by Alexandre Barachant

http://blog.kaggle.com/2015/10/12/grasp-and-lift-eeg-winners-interview-1st-place-cat-dog/


%scala
//This allows easy embedding of publicly available information into any other notebook
//when viewing in git-book just ignore this block - you may have to manually chase the URL in frameIt("URL").
//Example usage:
// displayHTML(frameIt("https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation#Topics_in_LDA",250))
def frameIt( u:String, h:Int ) : String = {
      """<iframe 
 src=""""+ u+""""
 width="95%" height="""" + h + """"
 sandbox>
  <p>
    <a href="http://spark.apache.org/docs/latest/index.html">
      Fallback link for browsers that, unlikely, don't support frames
    </a>
  </p>
</iframe>"""
   }
displayHTML(frameIt("http://blog.kaggle.com/2015/10/12/grasp-and-lift-eeg-winners-interview-1st-place-cat-dog/",600))



display(dbutils.fs.ls("dbfs:/datasets/eeg/")) #data already in dbfs - see below for details



testRdd = sc.textFile('dbfs:/datasets/eeg/test.zip')
trainRdd = sc.textFile('dbfs:/datasets/eeg/train.zip')



%fs ls "dbfs:/home/ubuntu/databricks/EEG/train"



subj3_series3_events_Path = "dbfs:/home/ubuntu/databricks/EEG/train/subj3_series3_events.csv"
subj3_series4_events_Path = "dbfs:/home/ubuntu/databricks/EEG/train/subj3_series4_events.csv" 



subj3_series3_data_Path = "dbfs:/home/ubuntu/databricks/EEG/train/subj3_series3_data.csv"
subj3_series4_data_Path = "dbfs:/home/ubuntu/databricks/EEG/train/subj3_series4_data.csv"

generate RDD


subj3_series3_events = sc.textFile(subj3_series3_events_Path)
subj3_series4_events = sc.textFile(subj3_series4_events_Path)
subj3_series34_events = subj3_series3_events.union(subj3_series4_events)



subj3_series3_data = sc.textFile(subj3_series3_data_Path)
subj3_series4_data = sc.textFile(subj3_series4_data_Path)
subj3_series34 = subj3_series3_data.union(subj3_series4_data)

generate DataFrame from csv file


subj3_series3_Raw_DF = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferSchema='true').load(subj3_series3_data_Path)



subj3_series3_DF = subj3_series3_Raw_DF.drop('id')



display(subj3_series3_DF)

create DF from RDD


subj3_series4_Raw_DF = subj3_series4_data.map(lambda x: (x, )).toDF()



subj3_series3_events_Raw_DF = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferSchema='true').load(subj3_series3_events_Path)
subj3_series4_events_Raw_DF = subj3_series4_events.map(lambda x: (x, )).toDF()




display(subj3_series3_events_DF)

#neural oscillation

  • neural oscillationis characterized by change in signal power in specific frequency bands. These oscillations appear naturally in ongoing EEG activity, can be induced by a specific task, for example a hand movement, or mental calculus.
  • For each subject, we should see a spot over the electrode C3 (Left motor cortex,corresponding to a right hand movement), and a decrease of the signal power in 10 and 20 Hz during the movement (by reference to after the movement).

subj3_series3_events_DF.filter("HandStart = 1").count()



subj3_series34.map(lambda x: (x, )).toDF().filter("HandStart = 1").count()



raw = creat_mne_raw_object(subj3_series3_DF)



# get chanel names
ch_names = list(subj3_series3_DF)
    
ch_names

To get data to dbfs let’s download and save.


%sh
pwd



%sh 
df -h /databricks/driver

This data in http://www.math.canterbury.ac.nz/~r.sainudiin/tmp/ may be deleted in the future.


%sh
wget http://www.math.canterbury.ac.nz/~r.sainudiin/tmp/test.zip



%sh
wget http://www.math.canterbury.ac.nz/~r.sainudiin/tmp/train.zip



dbutils.fs.mkdirs("dbfs:/datasets/eeg")



dbutils.fs.cp("file:/databricks/driver/train.zip","dbfs:/datasets/eeg/")



display(dbutils.fs.ls("dbfs:/datasets/eeg/"))



testRdd = sc.textFile('dbfs:/datasets/eeg/test.zip')
trainRdd = sc.textFile('dbfs:/datasets/eeg/train.zip')



testRdd.take(5)



trainRdd.take(5)



%sh
rm train.zip test.zip







dbutils.fs.help()

Scalable Data Science

Student Project Presentation by Shanshan Zhou

supported by and

Updated: