{ "cells": [ { "cell_type": "markdown", "metadata": { "deletable": false }, "source": [ "# [Applied Statistics](https://lamastex.github.io/scalable-data-science/as/2019/)\n", "## 1MS926, Spring 2019, Uppsala University \n", "©2019 Raazesh Sainudiin. [Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 00. Introduction \n", "\n", "1. Introduction\n", "2. What is SageMath and why are we using it?\n", "* Interaction - learning/teaching style\n", "* What can you expect to get out of this course? \n", "\n", "\n", "## Introduction\n", "\n", "See [Applied Statistics](https://lamastex.github.io/scalable-data-science/as/2019/) for learning outcomes, content, assessment, etc.\n", "\n", "**Assessment:**\n", "\n", "- Three Computer Lab Assessments (attendance will be taken in the Labs)\n", "- Final Exam will be in Computer Lab\n", "\n", "## What is SageMath and why are we using it?\n", "\n", "We will be using Sage or [SageMath](http://www.sagemath.org/) for our *hands-on* work in this course. Sage is a free open-source mathematics software system licensed under the GPL. Sage can be used to study mathematics and statistics, including algebra, calculus, elementary to very advanced number theory, cryptography, commutative algebra, group theory, combinatorics, graph theory, exact linear algebra, optimization, interactive data visualization, randomized or Monte Carlo algorithms, scientific and statistical computing and much more. It combines various software packages into an integrative learning, teaching and research experience that is well suited for novice as well as professional researchers.\n", " \n", "Sage is a set of software libraries built on top of [Python](http://www.python.org/), a widely used general purpose programming language. Sage greatly enhance Python's already mathematically friendly nature. It is one of the languages used at Google, US National Aeronautic and Space Administration (NASA), US Jet Propulsion Laboratory (JPL), Industrial Light and Magic, YouTube, and other leading entities in industry and public sectors. Scientists, engineers, and mathematicians often find it well suited for their work. Obtain a more thorough rationale for Sage from Why Sage? and Success Stories, Testimonials and News Articles. Jump start your motivation by taking a Sage Feature Tour right now!\n", "\n", "## Interaction - learning/teaching style\n", "\n", "This is an interactive jupyter notebook with SageMath interpreter and interactive means...\n", "\n", "#### Videos\n", "We will embed relevant videos in the notebook, such as those from [The Khan Academy](http://www.khanacademy.org/) or open MOOCs from google, facebook, academia, etc.\n", "\n", "* [watch Google's Hal Varian's 'statistics is the dream' job speech](https://www.youtube.com/embed/D4FQsYTbLoI)\n", "* [watch UC Berkeley Professor Michael Jordan's speech on 'The Data Science Revolution'](https://youtu.be/ggq7HiDO0OU)\n", "\n", "#### Latex\n", "We will *formally present mathematical and statistical concepts* in the Notebook using Latex as follows:\n", "\n", "$$\\sum_{i=1}^5 i = 1+2+3+4+5=15, \\qquad \\prod_{i=3}^6 i = 3 \\times 4 \\times 5 \\times 6 = 360$$\n", "\n", "$$\\binom{n}{k}:= \\frac{n!}{k!(n-k)!}, \\qquad \\lim_{x \\to \\infty}\\exp{(-x)} = 0$$\n", "\n", "$$\\{\\alpha, \\beta, \\gamma, \\delta, \\epsilon, \\zeta, \\mu,\\theta, \\vartheta, \\phi, \\varphi, \\omega, \\sigma, \\varsigma,\\Gamma, \\Delta, \\Theta, \\Phi, \\Omega\\}, \\qquad \\forall x \\in X, \\quad \\exists y \\leq \\epsilon, \\ldots$$\n", "\n", "#### Interactive Visualizations\n", "We will use interactive visualisations to convey concepts when possible. See the Taylor approximation below for a given order." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "0de36b8ed60a462baa7d8d02a23609e3", "version_major": 2, "version_minor": 0 }, "text/plain": [ "SW50ZXJhY3RpdmUgZnVuY3Rpb24gPGZ1bmN0aW9uIF8gYXQgMHg3ZjdmMWI5YmVjODA+IHdpdGggMSB3aWRnZXQKICBvcmRlcjogRHJvcGRvd24oZGVzY3JpcHRpb249dSdvcmRlcicsIG9wdGnigKY=\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "var('x')\n", "x0 = 0\n", "f = sin(x)*e^(-x)\n", "p = plot(f,-1,5, thickness=2)\n", "dot = point((x0,f(x=x0)),pointsize=80,rgbcolor=(1,0,0))\n", "@interact\n", "def _(order=[1..12]):\n", " ft = f.taylor(x,x0,order)\n", " pt = plot(ft,-1, 5, color='green', thickness=2)\n", " pretty_print(html('$f(x)\\;=\\;%s$'%latex(f)))\n", " pretty_print(html('$\\hat{f}(x;%s)\\;=\\;%s+\\mathcal{O}\\\n", " (x^{%s})$'%(x0,latex(ft),order+1)))\n", " show(dot + p + pt, ymin = -.5, ymax = 1, figsize=[6,3])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Lab-Lecture Style of Teaching-Learning\n", "\n", "We will *write computer programs* within code cells in the Notebook right after we learn the mathematical and statistical concepts. \n", "\n", "Thus, there is a significant overlap between traditional lectures and labs in this course -- in fact these interactions are *lab-lectures*.\n", "\n", "#### Live Data Explorations and Modeling\n", "Let us visualize the CO2 data, fetched from US NOAA, and do a simple linear regression." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "80e6cb2e379945d7944e77cc56456f30", "version_major": 2, "version_minor": 0 }, "text/plain": [ "SW50ZXJhY3RpdmUgZnVuY3Rpb24gPGZ1bmN0aW9uIG1hdW5hX2xvYV9jbzIgYXQgMHg3Zjc0N2M0YzlmNTA+IHdpdGggMiB3aWRnZXRzCiAgc3RhcnRfZGF0ZTogVHJhbnNmb3JtSW50U2xpZGXigKY=\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Author: Marshall Hampton \n", "import urllib2 as U\n", "import scipy.stats as Stat\n", "from IPython.display import HTML\n", "co2data = U.urlopen(\\\n", " 'ftp://ftp.cmdl.noaa.gov/ccg/co2/trends/co2_mm_mlo.txt'\\\n", " ).readlines()\n", "datalines = []\n", "for a_line in co2data:\n", " if a_line.find('Creation:') != -1:\n", " cdate = a_line\n", " if a_line != '#':\n", " temp = a_line.replace('\\n','').split(' ')\n", " temp = [float(q) for q in temp if q != '']\n", " datalines.append(temp)\n", "trdf = RealField(16)\n", "@interact\n", "def mauna_loa_co2(start_date = slider(1958,2018,1,1958), \\\n", " end_date = slider(1958, 2018,1,2018)):\n", " htmls1 = '

### CO2 monthly averages at Mauna Loa (interpolated),\\\n", " from NOAA/ESRL data

'\n", " htmls2 = '

#### '+cdate+'

'\n", " sel_data = [[q,q] for q in datalines if start_date < \\\n", " q < end_date]\n", " c_max = max([q for q in sel_data])\n", " c_min = min([q for q in sel_data])\n", " slope, intercept, r, ttprob, stderr = Stat.linregress(sel_data)\n", " pretty_print(html(htmls1+htmls2+'

#### Linear regression slope: '\\\n", " + str(trdf(slope))+ \\\n", " ' ppm/year; correlation coefficient: ' +\\\n", " str(trdf(r)) + '

'))\n", " var('x,y')\n", " show(list_plot(sel_data, plotjoined=True, rgbcolor=(1,0,0)) \n", " + plot(slope*x+intercept,start_date,end_date), \n", " xmin = start_date, ymin = c_min-2, axes = True, \\\n", " xmax = end_date, ymax = c_max+3, \\\n", " frame = False, figsize=[8,3])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### We will use publicly available resources generously!\n", "\n", "Here is an image of number systems from Wikipedia.\n", "\n", " \n", "\n", "We will also sometimes embed whole wikipedia pages. Expect cached wikipedia pages in your final exam. The curse will prepare you to think from facts in publicly available information." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "def showURL(url, ht=500):\n", " \"\"\"Return an IFrame of the url to show in notebook \\\n", " with height ht\"\"\"\n", " from IPython.display import IFrame\n", " return IFrame(url, width='95%', height=ht) \n", "showURL('https://en.wikipedia.org/wiki/Number',400)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What can you expect to get out of this course?\n", "\n", "Strengthen your foundations in:\n", "\n", "- probability and statistics,\n", "- computing and data analysis\n", "\n", "in order to understand the probabilistic models and statistical inference procedures as well implement computer programs for processing raw data - a crucial distinguishing skillset of a *modern applied statistician*, i.e., a *data scientist* who knows her/his probabilistic and statistical foundations.\n", " \n", "#### What is Data Science?\n", "\n", "We will steer clear of academic/philosophical discussions on \"what is data science?\" and focus instead on the core skillset in mathematics, statistics and computing that is expected in a typical data science job today." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "showURL(\"https://en.wikipedia.org/wiki/Data_science\")" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "# Course Structure\n", "\n", "There will be 12 \"lab-lectures\" where I will be using .ipynb or IPython notebooks like this and 3 dedicated computer laboratories.\n", "\n", "We will start with basics of programming in BASH and a review of Python before recollecting concepts in probability and setting the stage for applied statistics, including, hypothesis testing and parameter estimation.\n", "\n", "**Connection to Statistical Machine Learning and AI:** An immediate application of paramter estimation in *statistical machine learning (SML)* and *artifical intelligence (AI)*, where you apply more sophisticated methods than what you will learn here, is *prediction*. This course will prepare you for SML and AI.\n", "\n", "**STS also stands for Society**\n", "\n", "As STS students, you have the natural responsibility to ask questions such as: *what is the cost of such sophisticated prediction algorithms on our society and planet?*\n", "\n", "Here your first assigned reading is from the following work:\n", "\n", "- [https://anatomyof.ai/](https://anatomyof.ai/), \"Anatomy of an AI System\" By Kate Crawford and Vladan Joler (2018)\n", "\n", "The Amazon Echo as an anatomical map of human labor, data and planetary resources. Download the detailed [ai-anatomy-map.pdf](https://anatomyof.ai/img/ai-anatomy-map.pdf).\n", "\n", " " ] }, { "cell_type": "markdown", "metadata": { "lx_assignment_number": "1", "lx_problem_cell_type": "PROBLEM" }, "source": [ "---\n", "## Assignment 1, PROBLEM 0\n", "Maximum Points = 3" ] }, { "cell_type": "markdown", "metadata": { "lx_assignment_number": "1", "lx_problem_cell_type": "PROBLEM", "lx_problem_number": "0", "lx_problem_points": "3" }, "source": [ "\n", "Given that you are in the civil engineering programme in *systems in technology and society*, spend some time reading the following:\n", "- [https://anatomyof.ai/](https://anatomyof.ai/), \"Anatomy of an AI System\" By Kate Crawford and Vladan Joler (2018)\n", "\n", "Answer whether each of the following statements is True or False *according to the authors* by appropriately replacing Xxxxx coresponding to TruthValueOfStatement0a, TruthValueOfStatement0b and TruthValueOfStatement0c, respectively, in the next cell to demonstrate your reading comprehension.\n", "\n", "1. Statement0a = *Each small moment of convenience (provided by Amazon's Echo) – be it answering a question, turning on a light, or playing a song – requires a vast planetary network, fueled by the extraction of non-renewable materials, labor, and data.*\n", "- Statement0b = *The Echo user is simultaneously a consumer, a resource, a worker, and a product* \n", "- Statement0c = *Many of the assumptions about human life made by machine learning systems are narrow, normative and laden with error. Yet they are inscribing and building those assumptions into a new world, and will increasingly play a role in how opportunities, wealth, and knowledge are distributed.*" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "lx_assignment_number": "1", "lx_problem_cell_type": "PROBLEM", "lx_problem_number": "0", "lx_problem_points": "3" }, "outputs": [], "source": [ "# Replace Xxxxx with True or False; Don't modify anything else in this cell!\n", "\n", "TruthValueOfStatement0a = Xxxxx\n", "\n", "TruthValueOfStatement0b = Xxxxx\n", "\n", "TruthValueOfStatement0c = Xxxxx" ] }, { "cell_type": "markdown", "metadata": { "lx_assignment_number": "1", "lx_problem_cell_type": "Test" }, "source": [ "---\n", "#### Local Test for Assignment 1, PROBLEM 0\n", "Evaluate cell below to make sure your answer is valid. You **should not** modify anything in the cell below when evaluating it to do a local test of your solution." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "lx_assignment_number": "1", "lx_problem_cell_type": "Test", "lx_problem_number": "0", "lx_problem_points": "3" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Good, you have answered either True or False. Hopefully they are the correct answers!\n" ] } ], "source": [ "# Test locally to ensure an acceptable answer, True or False\n", "try:\n", " assert(isinstance(TruthValueOfStatement0a, bool)) \n", " assert(isinstance(TruthValueOfStatement0b, bool)) \n", " assert(isinstance(TruthValueOfStatement0c, bool))\n", " print(\"Good, you have answered either True or False. Hopefully they are the correct answers!\")\n", "except AssertionError:\n", " print(\"Try again. You are not writing True or False for your answers.\")" ] }, { "cell_type": "markdown", "metadata": { "lx_assignment_number": "1", "lx_problem_cell_type": "PROBLEM" }, "source": [ "---\n", "## Assignment 1, PROBLEM 1\n", "Maximum Points = 1" ] }, { "cell_type": "markdown", "metadata": { "lx_assignment_number": "1", "lx_problem_cell_type": "PROBLEM", "lx_problem_number": "1", "lx_problem_points": "1" }, "source": [ "\n", "#### NonAutoGraded\n", "#### Summarise in 50-100 words your thoughts after reading [https://anatomyof.ai/](https://anatomyof.ai/), \"Anatomy of an AI System\" By Kate Crawford and Vladan Joler (2018)\n", "\n", "\n", "You can double-click this cell and start writing your summary below between the two --- lines in English. When you are done just CTRL-Enter (press down the ctrl key and hit the Enter key) to see how it looks in display mode.\n", "\n", "---\n", "\n", "\n", "\n", "---" ] } ], "metadata": { "kernelspec": { "display_name": "SageMath 8.4", "language": "", "name": "sagemath" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.15" }, "lx_course_instance": "Spring 2019, Uppsala University", "lx_course_name": "Applied Statistics", "lx_course_number": "1MS926" }, "nbformat": 4, "nbformat_minor": 2 }