![](\"images/unknownProcessTimesAnim.gif\")
\n",
"\n",
"Consider the following non-parametric product experiment:\n",
"\n",
"$$X_1, X_2, \\ldots, X_n\\ \\overset{IID}{\\sim} F^* \\in \\{\\text{all DFs}\\}$$\n",
"\n",
"We want to produce a point estimate for $F^*$, which is a allowed to be any DF (\"lives in the set of all DFs\"), i.e., $F^* \\in \\{\\text{all DFs}\\}$\n",
"\n",
"Crucially, $\\{\\text{all DFs}\\}$, i.e., the set of all distribution functions over $\\mathbb{R}$ is infinite dimensional.\n",
"\n",
"![](\"images/TwoDFs.png\")
\n",
"\n",
"We have already seen an estimate, made using the data, of a distribution function: the empirical or data-based distribution function (or empirical cumulative distribution function). This can be formalized as the following process of adding indicator functions of the half-lines beginning at the data points $[X_1,+\\infty),[X_2,+\\infty),\\ldots,[X_n,+\\infty)$:\n",
"\n",
"$$\\widehat{F}_n (x) = \\frac{1}{n} \\sum_{i=1}^n \\mathbf{1}_{[X_i,+\\infty)}(x)$$\n",
"\n",
"\n",
"where,\n",
"\n",
"$$\\mathbf{1}_{[X_i,+\\infty)}(x) := \\begin{cases} & 1 \\quad \\text{ if } X_i \\leq x \\\\ & 0 \\quad \\text{ if }X_i > x \\end{cases}$$\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First let us evaluate a set of functions that will help us conceptualize faster:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"def makeEMFHidden(myDataList):\n",
" '''Make an empirical mass function from a data list.\n",
" \n",
" Param myDataList, list of data to make emf from.\n",
" Return list of tuples comprising (data value, relative frequency) ordered by data value.'''\n",
" \n",
" sortedUniqueValues = sorted(list(set(myDataList)))\n",
" freqs = [myDataList.count(i) for i in sortedUniqueValues]\n",
" relFreqs = [ZZ(fr)/len(myDataList) for fr in freqs] # use a list comprehension\n",
" \n",
" return zip(sortedUniqueValues, relFreqs)\n",
" \n",
"\n",
"from pylab import array\n",
"\n",
"def makeEDFHidden(myDataList, offset=0):\n",
" '''Make an empirical distribution function from a data list.\n",
" \n",
" Param myDataList, list of data to make ecdf from.\n",
" Param offset is an offset to adjust the edf by, used for doing confidence bands.\n",
" Return list of tuples comprising (data value, cumulative relative frequency) ordered by data value.'''\n",
" \n",
" sortedUniqueValues = sorted(list(set(myDataList)))\n",
" freqs = [myDataList.count(i) for i in sortedUniqueValues]\n",
" from pylab import cumsum\n",
" cumFreqs = list(cumsum(freqs)) #\n",
" cumRelFreqs = [ZZ(i)/len(myDataList) for i in cumFreqs] # get cumulative relative frequencies as rationals\n",
" if offset > 0: # an upper band\n",
" cumRelFreqs = [min(i ,1) for i in cumRelFreqs] # use a list comprehension\n",
" if offset < 0: # a lower band\n",
" cumRelFreqs = [max(i, 0) for i in cumFreqs] # use a list comprehension\n",
" return zip(sortedUniqueValues, cumRelFreqs)\n",
" \n",
"# EPMF plot\n",
"def epmfPlot(samples):\n",
" '''Returns an empirical probability mass function plot from samples data.'''\n",
" \n",
" epmf_pairs = makeEMFHidden(samples)\n",
" epmf = point(epmf_pairs, rgbcolor = \"blue\", pointsize=\"20\")\n",
" for k in epmf_pairs: # for each tuple in the list\n",
" kkey, kheight = k # unpack tuple\n",
" epmf += line([(kkey, 0),(kkey, kheight)], rgbcolor=\"blue\", linestyle=\":\")\n",
" # padding\n",
" epmf += point((0,1), rgbcolor=\"black\", pointsize=\"0\")\n",
" return epmf\n",
" \n",
"\n",
"# ECDF plot\n",
"def ecdfPlot(samples):\n",
" '''Returns an empirical probability mass function plot from samples data.'''\n",
" ecdf_pairs = makeEDFHidden(samples)\n",
" ecdf = point(ecdf_pairs, rgbcolor = \"red\", faceted = false, pointsize=\"20\")\n",
" for k in range(len(ecdf_pairs)):\n",
" x, kheight = ecdf_pairs[k] # unpack tuple\n",
" previous_x = 0\n",
" previous_height = 0\n",
" if k > 0:\n",
" previous_x, previous_height = ecdf_pairs[k-1] # unpack previous tuple\n",
" ecdf += line([(previous_x, previous_height),(x, previous_height)], rgbcolor=\"grey\")\n",
" ecdf += points((x, previous_height),rgbcolor = \"white\", faceted = true, pointsize=\"20\")\n",
" ecdf += line([(x, previous_height),(x, kheight)], rgbcolor=\"grey\", linestyle=\":\")\n",
" # padding\n",
" ecdf += line([(ecdf_pairs[0][0]-0.2, 0),(ecdf_pairs[0][0], 0)], rgbcolor=\"grey\")\n",
" max_index = len(ecdf_pairs)-1\n",
" ecdf += line([(ecdf_pairs[max_index][0], ecdf_pairs[max_index][1]),(ecdf_pairs[max_index][0]+0.2, ecdf_pairs[max_index][1])],rgbcolor=\"grey\")\n",
" return ecdf\n",
" \n",
"def calcEpsilon(alphaE, nE):\n",
" '''Return confidence band epsilon calculated from parameters alphaE > 0 and nE > 0.'''\n",
" \n",
" return sqrt(1/(2*nE)*log(2/alphaE))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Let us continue with the concepts\n",
"\n",
"We can remind ourselves of this for a small sample of $de\\,Moivre(k=5)$ RVs:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[2, 1, 4, 1, 5, 2, 1, 4, 5, 2, 2, 3, 2, 4, 5, 5, 2, 1, 5, 4]"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"deMs=[randint(1,5) for i in range(20)] # randint can be used to uniformly sample integers in a specified range\n",
"deMs"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[(1, 1/5), (2, 1/2), (3, 11/20), (4, 3/4), (5, 1)]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sortedUniqueValues = sorted(list(set(deMs)))\n",
"freqs = [deMs.count(i) for i in sortedUniqueValues]\n",
"from pylab import cumsum\n",
"cumFreqs = list(cumsum(freqs)) #\n",
"cumRelFreqs = [ZZ(i)/len(deMs) for i in cumFreqs] # get cumulative relative frequencies as rationals\n",
"zip(sortedUniqueValues, cumRelFreqs)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAksAAAEgCAYAAACkZjcGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3X9wVOW9x/HP5gcbkpAlmJBGDJjqoJAIjIFBEDql1cykXoS5OoOMBVv1og5SgTojFKzyy/ijdezUQA2ioqOBGShKW/Cador4o1wQjTqRGijGoN0Qks3ukl+bZHfvH5RoDCzZkOyzZ/N+zew4e9hz9hOPwpfv85znsQWDwaAAAABwTnGmAwAAAEQziiUAAIAQKJYAAABCoFgCAAAIgWIJAAAgBIolAACAECiWAAAAQqBYAgAACIFiCQAAIASKJQAAgBAolgAAAEKwRLEUDAbl9XrFNnYAACDSLFEsnT59Wg6HQ6dPnzYdBQAADDIRKZb279+v2bNn69JLL5XNZtPrr78eia8FAADRprJS+r//k/x+00l6LSLFUnNzsyZOnKhnn302El8HAACiTX29NHOmlJ8vXXeddMUV0gcfmE7VKwmR+JKioiIVFRVF4qsAAEA0Wr5cevfdb95/+aU0b5509KgUF92zgiJSLIXL5/PJ5/N1vfd6vQbTAACAi7ZrlySpIzFR9RkZZ461tkr790tXXXXRl8/IyFBiYuJFX+dcorJYKi4u1po1a0zHAAAA/SUtTWpq0smRI7Xlf/7nm+Nvv33mdZEWLVqk7Ozsi77OudiCEX4e32azadeuXZo7d+55P3OuzlJOTo48Ho/S0tIiERMAAPSnxx6TVq3q3lmaPl36/e/75fKDrrNkt9tlt9tNxwAAAP1l5UopPl5xzz2nVL9fyT/5ieIff1xyOEwnu6DonlEFAABig80mPfSQ6t57T0/feafqfv1rSxRKUoQ6S01NTTp27FjX+y+++EIVFRUaMWKERo8eHYkIAAAgCqSnp2v+/PlKT083HaXXIjJnad++fZo1a1aP43fccYdeeumlC57v9XrlcDiYswQAACIu4hO8+4JiCQCA2NDc3KzKykrl5eUpJSXFdJxeYc4SAACIGK/Xq7feestSayjSWQIAAAiBzhIAAEAIFEsAACBiGhoa9PLLL6uhocF0lF6jWAIAABETFxenlJQUxUX55rnfFtVzlkpKSlRSUiK/36+qqirmLAEAgIiL6mLpLCZ4AwAQGwKBgDo6OpSYmGiZ7pI1UgIAgJhw8uRJPf744zp58qTpKL1GsQQAACJm+PDhuvXWWzV8+HDTUXotInvDAQAASNLQoUOVl5dnOkZY6CwBAICIaWlpUUVFhVpaWkxH6TWKJQAAEDEej0dvvPGGPB6P6Si9xtNwAAAgYoLBoILBoGw2m2w2m+k4vcKcJQAAEDFWKpLOYhgOAABEjMvlUllZmVwul+kovUaxBAAAEEJUz1liuxMAAGBaVBdLZzHBGwCA2GDFCd4MwwEAgIipra3VunXrVFtbazpKr1EsAQCAiHE4HJozZ44cDofpKL3G0gEAACBikpOTNWnSJNMxwkJnCQAARExra6sqKyvV2tpqOkqvUSwBAICIcbvd2rFjh9xut+kovcbTcAAAIGICgYA6OjqUmJiouDhr9GyYswQAACImLi5OdrvddIywWKOkAwAAMaGxsVE7d+5UY2Oj6Si9RrEEAAAiJhAIqLm5WYFAwHSUXmPOEgAAQAhR3VkqKSnR+PHjNWXKFNNRAADAIEVnCQAARIzT6dSWLVt01113KTs723ScXonqzhIAAIgtaWlpKiwstFTzg84SAABACHSWAABAxLS1tamqqkptbW2mo/QaxRIAAIiYxsZGlZWVWWqdJYbhAABAxPj9frW0tCg5OVnx8fGm4/QK250AAICIiY+P17Bhw0zHCEufhuE2btyo3NxcJSUlqaCgQO+8807Izz/zzDO66qqrNHToUOXk5GjZsmWWGqsEAAD9w+12a/fu3XK73aaj9FrYxdL27du1dOlSrVq1Sh999JFmzpypoqIi1dTUnPPzr776qlasWKFHHnlER44c0ZYtW7R9+3atXLnyosMDAABr6ezs1KlTp9TZ2Wk6Sq+FPWdp6tSpuvbaa7Vp06auY+PGjdPcuXNVXFzc4/P333+/jhw5or/97W9dx375y1/q4MGDF+xIncWcJQAAYEpYnaX29nYdPnxYhYWF3Y4XFhbq/fffP+c5M2bM0OHDh3Xw4EFJ0vHjx7Vnzx7ddNNN5/0en88nr9fb7QUAAGBCWMVSfX29/H6/srKyuh3PyspSbW3tOc+57bbbtG7dOs2YMUOJiYm64oorNGvWLK1YseK831NcXCyHw9H1ysnJCScmAACIUrW1tXriiSfOWzdEoz5N8LbZbN3eB4PBHsfO2rdvnzZs2KCNGzfqww8/1B//+Ef9+c9/1rp16857/ZUrV8rj8XS9Tpw40ZeYAAAgyqSmpmrGjBlKTU01HaXXwlo6ICMjQ/Hx8T2qwbq6uh7dprMefvhhLViwQHfffbck6ZprrlFzc7MWLVqkVatWKS6uZ71mt9tlt9vDiQYAACwgNTVV119/vekYYQmrszRkyBAVFBSovLy82/Hy8nJNnz79nOe0tLT0KIji4+MVDAZlgfUwAQBAP/L5fKqurpbP5zMdpdfCHoZbvny5nn/+eb3wwgs6cuSIli1bppqaGt17772SpIULF3ZbFmD27NnatGmTtm3bpi+++ELl5eV6+OGHdfPNN1tm5U4AANA/XC6Xtm7dKpfLZTpKr4W9gve8efPU0NCgtWvXyul0Kj8/X3v27NGYMWMkSTU1Nd06SatXr5bNZtPq1av19ddfKzMzU7Nnz9aGDRv676cAAACWkJmZqSVLllhqKSD2hgMAAAihT0/DAQAA9IXH49HevXvl8XhMR+k1iiUAQGw7cUJqbDSdAv/R3t6u6upqtbe3m47SaxRLAIDY9NlnUkGBNHq0NHKktHChxCbuxmVmZuq+++5TZmam6Si9RrEEAIg9gYA0Z4704Ydn3nd2Sq+8Iv3612ZzwZLCfhoOAICod+iQdOyYJKkjMVH1GRlnjv/v/0rLlhkMBpfLpb1792r+/PkaNWqU6Ti9EtXFUklJiUpKSuT3+01HAQBYScI3f7zVZ2So9J57vvm10lIDgfBdbRYaEmXpAABAbJo4UfrkE/kSE1Wbna2Ejg7F3XOPtGiR6WTQmS3UEhMTTcfolajuLAEA0Gd/+pO0aJFcn3yil+68U4taWpS9alW3rhPQG3SWAAAxrbWxUce/+krfv/JKDR061HQcWBDlNQAgpg1NT1deerrpGLAwlg4AAMS0lpYWVVRUqKWlxXQUWBTFEgAgpnk8Hr3xxhuW2l4D0YU5SwCAmBYMBhUMBmWz2WSz2UzHgQUxZwkAENMoknCxGIYDAMQ0l8ulsrIyuVwu01FgURRLAAAAITBnCQAAIISo7iyVlJRo/PjxmjJliukoAACLCgaDCgQCskBvAFGKzhIAIKY5nU6VlpZq0aJFys7ONh0HFhTVnSUAAC6Ww+HQnDlz5HA4TEeBRbF0AAAgpiUnJ2vSpEmmY8DC6CwBAGJaa2urKisr1draajoKLIpiCQAQ09xut3bs2CG32206CiyKCd4AgJgWCATU0dGhxMRExcXRI0D4mLMEAIhpcXFxstvtpmPAwiixAQAxrbGxUTt37lRjY6PpKLAoiiUAQEwLBAJqbm5WIBAwHQUWxZwlAACAEOgsAQAAhBDVxRJ7wwEALpbT6dT69evldDpNR4FFMQwHAIhpzc3NqqysVF5enlJSUkzHgQVRLAEAAIQQ1cNwAABcrLa2NlVVVamtrc10FFgUxRIAIKY1NjaqrKyMdZbQZwzDAQBimt/vV0tLi5KTkxUfH286DiyI7U4AADEtPj5ew4YNMx0DFtanYbiNGzcqNzdXSUlJKigo0DvvvBPy8263W4sXL1Z2draSkpI0btw47dmzp0+BAQAIh9vt1u7du+V2u01HgUWF3Vnavn27li5dqo0bN+r666/Xc889p6KiIn322WcaPXp0j8+3t7frxhtv1MiRI7Vjxw5ddtllOnHiBFU+ACAiOjs7derUKXV2dpqOAosKe87S1KlTde2112rTpk1dx8aNG6e5c+equLi4x+f/8Ic/6KmnntI///lPJSYm9ikkc5YAAIApYQ3Dtbe36/DhwyosLOx2vLCwUO+///45z9m9e7emTZumxYsXKysrS/n5+Xrsscfk9/vP+z0+n09er7fbCwAAwISwiqX6+nr5/X5lZWV1O56VlaXa2tpznnP8+HHt2LFDfr9fe/bs0erVq/Xb3/5WGzZsOO/3FBcXy+FwdL1ycnLCiQkAQJfa2lo98cQT5/1zCriQPk3wttls3d4Hg8Eex84KBAIaOXKkSktLVVBQoNtuu02rVq3qNoz3XStXrpTH4+l6nThxoi8xAQBQamqqZsyYodTUVNNRYFFhTfDOyMhQfHx8j+q8rq6uR7fprOzsbCUmJnZb22LcuHGqra1Ve3u7hgwZ0uMcu90uu90eTjQAAM4pNTVV119/vekYsLCwOktDhgxRQUGBysvLux0vLy/X9OnTz3nO9ddfr2PHjikQCHQdq6qqUnZ29jkLJQAA+pPP51N1dbV8Pp/pKLCosIfhli9frueff14vvPCCjhw5omXLlqmmpkb33nuvJGnhwoVauXJl1+fvu+8+NTQ06IEHHlBVVZX+8pe/6LHHHtPixYv776cAAOA8XC6Xtm7dKpfLZToKLCrsdZbmzZunhoYGrV27Vk6nU/n5+dqzZ4/GjBkjSaqpqVFc3Dc1WE5Ojt566y0tW7ZMEyZM0KhRo/TAAw/ooYce6r+fAgCA88jMzNSSJUtYegZ9xt5wAAAAIfTpaTgAAKzC4/Fo79698ng8pqPAoiiWAAAxrb29XdXV1WpvbzcdBRbFMBwA9Jdjx6Q1a6RDh6Tx46XVq6VrrzWdCsBFolgCgP7gckl5edK316FLSZEqKqQrrzSXC8BFC/tpOADAObz6aleh1JGYqPqMjDPHS0ulZcsMBoPL5dLevXs1f/58jRo1ynQcWFBUF0slJSUqKSkJuekuAESFb3WUvszJ0asLF37za6WlBgLhu9ra2kxHgEUxDAcA/eHdd6WZMyVJjQ6HDlx3ncZWVSl51SrpJz8xHA7SmS27EhMTTceABVEsAUB/WbFCeuop6ez2TgsWSC+9JMXx4DFgZRRLANCfvvhCvoMH5Rw5UtnTp7MpOBAD+OsOAPSn3Fy5fvADbd2/n73IgBhBZwkA+llnZ6e8Xq/S0tKUkBDVz9EA6AX+LwaAfpaQkKARI0aYjgGgnzAMBwD9jL3IgNhCsQTEgo4O0wnwLexFBsQWiiXAyp5+Wvre96QhQ6Qf/lCqrDSdCJIyMzN13333KTMz03QUAP2ACd6AVb3yivTtVaIl6dJLpePHJR5XB4B+wwRvwKq2bJH0nX3IgkHpj38802WCMexFBsSWqC6W2BsOCMHnk3SOfciqqs68YBx7kQGxgWE4wKp+9ztp6VLVjxihfbNmaeLHHyvVZpP27JFSUkyng9iLDIgVFEuAVfn90tKlZ3a0b2+XcnOlF15gCA4A+hnFEmBxnfX18tbUKC0/XwlDhpiOAwAxh6UDAIs71dGh3//pTzrV0GA6CgDEJDpLgMX5fD45nU5lZ2ezwz0ADICofhoOwIXZ7XZdfvnlpmMAQMxiGA6wuKamJr333ntqamoyHQUAYhLFEmBxTU1NevfddymWAGCAMGcJAAAgBDpLAAAAIVAsARZXX1+vLVu2qL6+3nQUAIhJUV0slZSUaPz48ZoyZYrpKEDUSkhIUGZmphISeLgVAAYCc5YAAABCiOrOEoAL8/v9On36tPx+v+koABCTKJYAi6urq9PTTz+turo601EAICZRLAEWl56ervnz5ys9Pd10FACIScwIBSwuKSlJY8eONR0DAGIWnSXA4pqbm3Xw4EE1NzebjgIAMYliCbA4r9ert956S16v13QUAIhJfSqWNm7cqNzcXCUlJamgoEDvvPNOr87btm2bbDab5s6d25evBXAO2dnZWr16tbKzs01HAYCYFHaxtH37di1dulSrVq3SRx99pJkzZ6qoqEg1NTUhz/vyyy/14IMPaubMmX0OCwAAEGlhF0tPP/207rrrLt19990aN26cnnnmGeXk5GjTpk3nPcfv9+v222/XmjVr9P3vf/+iAgPorqGhQS+//LIaGhpMRwGAmBRWsdTe3q7Dhw+rsLCw2/HCwkK9//775z1v7dq1yszM1F133dWr7/H5fPJ6vd1eAM4tLi5OKSkpiotjCiIADISwlg6or6+X3+9XVlZWt+NZWVmqra095znvvfeetmzZooqKil5/T3FxsdasWRNONGDQSk9P1y233GI6BgDErD79VdRms3V7HwwGexyTpNOnT+unP/2pNm/erIyMjF5ff+XKlfJ4PF2vEydO9CUmMCgEAgH5fD4FAgHTUQAgJoXVWcrIyFB8fHyPLlJdXV2PbpMk/etf/1J1dbVmz57ddezsb+gJCQn6/PPPdcUVV/Q4z263y263hxMNGLROnjyp0tJSLVq0iCfiAGAAhNVZGjJkiAoKClReXt7teHl5uaZPn97j81dffbU+/fRTVVRUdL1uvvlmzZo1SxUVFcrJybm49AA0fPhw3XrrrRo+fLjpKAAQk8Le7mT58uVasGCBJk+erGnTpqm0tFQ1NTW69957JUkLFy7UqFGjVFxcrKSkJOXn53c7/+xv6N89DqBvhg4dqry8PNMxACBmhV0szZs3Tw0NDVq7dq2cTqfy8/O1Z88ejRkzRpJUU1PDUzlABLW0tKiqqkpjx45VcnKy6TgAEHNswWAwaDrEhXi9XjkcDnk8HqWlpZmOA0QVp9PJnCUAGEAUS4DFBYPBridSz/VUKgDg4oQ9DAcgulAkAcDAYnIRYHEul0tlZWVyuVymowBATKJYAgAACIE5SwAAACFEdWeppKRE48eP15QpU0xHAaJWMBhUIBCQBf7eAwCWRGcJsDiWDgCAgRXVnSUAF+ZwODRnzhw5HA7TUQAgJrF0AGBxycnJmjRpkukYABCz6CwBFtfa2qrKykq1traajgIAMYliCbA4t9utHTt2yO12m44CADGJCd6AxQUCAXV0dCgxMZFNrAFgADBnCbC4uLg42e120zEAIGbx11DA4hobG7Vz5041NjaajgIAMYliCbC4QCCg5uZmBQIB01EAICYxZwkAACCEqO4ssd0JAAAwjc4SYHFOp1NbtmzRXXfdxXYnADAAorqzBODC0tLSVFhYyF8kAGCA0FkCAAAIgc4SYHFtbW2qqqpSW1ub6SgAEJMolgCLa2xsVFlZGessAcAAYRgOsDi/36+WlhYlJycrPj7edBwAiDlsdwJYXHx8vIYNG2Y6BgDELIbhAItzu93avXu33G636SgAEJMolgCL6+zs1KlTp9TZ2Wk6CgDEJOYsAQAAhEBnCQAAIISoLpbYGw64sNraWj3xxBOqra01HQUAYhLDcIDFNTU16eOPP9bEiROVmppqOg4AxByKJQAAgBCiehgOwIX5fD5VV1fL5/OZjgIAMYliCbA4l8ulrVu3yuVymY4CADGJYTjA4jo7O+X1epWWlqaEBBblB4D+xu+sgMUlJCRoxIgRpmMAQMxiGA6wOI/Ho71798rj8ZiOAgAxqU/F0saNG5Wbm6ukpCQVFBTonXfeOe9nN2/erJkzZyo9PV3p6em64YYbdPDgwT4HRhRwu8+8EBXa29tVXV2t9vZ201EAICaFXSxt375dS5cu1apVq/TRRx9p5syZKioqUk1NzTk/v2/fPs2fP19///vf9Y9//EOjR49WYWGhvv7664sOjwhraJDmzpUuueTMa+7cM8dgVGZmpu677z5lZmaajgIAMSnsCd5Tp07Vtddeq02bNnUdGzdunObOnavi4uILnu/3+5Wenq5nn31WCxcu7NV3MsE7Svz3f0u7dnU/Nnduz2MAAMSQsCZ4t7e36/Dhw1qxYkW344WFhXr//fd7dY2WlhZ1dHSEnJDq8/m6rRnj9XrDiYmB4PVKb7whSepITFR9RsaZ4wcPSkePSqwcbYzL5dLevXs1f/58jRo1ynQcAIg5YRVL9fX18vv9ysrK6nY8Kyur1/tSrVixQqNGjdINN9xw3s8UFxdrzZo14URDJNhskqT6jAyV3nPPN8dfe81QIHxbW1ub6QgAEJP6tHSA7T9/aJ4VDAZ7HDuXJ598UmVlZdq3b5+SkpLO+7mVK1dq+fLlXe+9Xq9ycnL6EhX9JS3tzJDbzp0aUV+vn73wghI6OhT3ox9JTz1lOh0kZZzt9gEA+lVYxVJGRobi4+N7dJHq6up6dJu+6ze/+Y0ee+wx/fWvf9WECRNCftZut8tut4cTDZGwebMkyXXggF66804tOnpU2b//vZSebjgYAAADJ6yn4YYMGaKCggKVl5d3O15eXq7p06ef97ynnnpK69at05tvvqnJkyf3LSnMS0+XduzQ8I8+0q3/9V8avnkzhRIAIOaFPQy3fPlyLViwQJMnT9a0adNUWlqqmpoa3XvvvZKkhQsXatSoUV1Pxj355JN6+OGH9dprr+nyyy/v6kqlpqYqlUnBljQ0M1N5PKYOABgkwi6W5s2bp4aGBq1du1ZOp1P5+fnas2ePxowZI0mqqalRXNw3DauNGzeqvb1dt956a7frPPLII3r00UcvLj2MaGlpUVVVlcaOHavk5GTTcQAAGFBspIuwOZ1OlZaWatGiRcrOzjYdBwCAAUWxhLAFg8GuJyB78xQkAABW1qelAzC4USQBAAaTPm2ki8HN5XKprKxMLpfLdBQAAAYcxRIAAEAIzFkCAAAIIao7SyUlJRo/frymTJliOgq+JRgMKhAIyAJ1NgAAF43OEsLG0gEAgMEkqjtLiE4Oh0Nz5syRw+EwHQUAgAHH0gEIW3JysiZNmmQ6BgAAEUFnCWFrbW1VZWWlWltbTUcBAGDAUSwhbG63Wzt27JDb7TYdBQCAAccEb4QtEAioo6NDiYmJ3TZNBgAgFjFnCWGLi4uT3W43HQMAgIigLYCwNTY2aufOnWpsbDQdBQCAAUexhLAFAgE1NzcrEAiYjgIAwIBjzhIAAEAIdJYAAABCiOpiib3hopPT6dT69evldDpNRwEAYMAxDIewNTc3q7KyUnl5eUpJSTEdBwCAAUWxBAAAEEJUD8MhOrW1tamqqkptbW2mowAAMOAolhC2xsZGlZWVsc4SAGBQYBgOYfP7/WppaVFycrLi4+NNxwEAYECx3QnCFh8fr2HDhpmOAQBARDAMh7C53W7t3r1bbrfbdBQAAAYcxRLC1tnZqVOnTqmzs9N0FAAABhxzlgAAAEKgswQAABBCVBdLbHcSnWpra/XEE0+otrbWdBQAAAYcw3AIW1NTkz7++GNNnDhRqamppuMAADCgKJYAAABCiOphOEQnn8+n6upq+Xw+01EAABhwFEsIm8vl0tatW+VyuUxHAQBgwDEMh7B1dnbK6/UqLS1NCQksAg8AiG38SYewJSQkaMSIEaZjAAAQEQzDIWwej0d79+6Vx+MxHQUAgAFHsYSwtbe3q7q6Wu3t7aajAAAw4JizBAAAEAKdJQAAgBCMTfAOBoM6ffr0OX/N5/N1W8OnsbFRknTs2DFWjO6Dzs7Ofp1f5PV6deDAAd1yyy0aPXp0v10XAIBIGDZsmGw2W68/b2wY7uzQGgAAQCSFO63HWLEUTmfpxIkTmj59uvbv36+srKx++f4FCxbolVde6ZdrRfv1Ojs79Ytf/ELr1q3rl+u1trZqyZIlev311/vtfkyZMkWHDh3ql2sNtut5vV7l5OToxIkT/TanL5p/3oG4Xn9fk3sSXdfjfkTX9aLhfoTbWTI2DGez2cL+l5Sbm6vLLrusX77/9OnTGjt2bL9cywrX+/e//61p06b1y7W8Xq8+++wzZWVl9dt/6PHx8f06eX+wXU+S0tLSuB9Rdk3uSfRcT+J+RNP1pOi+H981aCd4L168mOtFkWj/eaP9ev0t2n/egfj3xz2J7ev1t2j/eaP9ev1toPNZYumAr776qqtl11+dJfQdSzlEF+5H9OGeRBfuR3Sx4v2wRGfJbrd3+yfMstvteuSRR7gfUYL7EX24J9GF+xFdrHg/LNFZOjsZPNwJWQAAABfLEsUSAACAKZYYhgMAADCFYgkAACAEiiUAAIAQKJbQa/v379fs2bN16aWXymaz6fXXXzcdaVArLi7WlClTNGzYMI0cOVJz587V559/bjrWoLVp0yZNmDCha6G9adOmae/evaZj4T+Ki4tls9m0dOlS01EGrUcffVQ2m63b63vf+57pWL1CsYRea25u1sSJE/Xss8+ajgJJb7/9thYvXqwDBw6ovLxcnZ2dKiwsVHNzs+log9Jll12mxx9/XB988IE++OAD/ehHP9KcOXNUWVlpOtqgd+jQIZWWlmrChAmmowx6eXl5cjqdXa9PP/3UdKReMbbdCaynqKhIRUVFpmPgP958881u71988UWNHDlShw8f1g9+8ANDqQav2bNnd3u/YcMGbdq0SQcOHFBeXp6hVGhqatLtt9+uzZs3a/369abjDHoJCQmW6SZ9G50lIEZ4PB5J0ogRIwwngd/v17Zt29Tc3NxvezKibxYvXqybbrpJN9xwg+kokHT06FFdeumlys3N1W233abjx4+bjtQrdJaAGBAMBrV8+XLNmDFD+fn5puMMWp9++qmmTZumtrY2paamateuXRo/frzpWIPWtm3b9OGHH4a1Gz0GztSpU/Xyyy9r7NixOnnypNavX6/p06ersrJSl1xyiel4IVEsATHg/vvv1yeffKJ3333XdJRB7aqrrlJFRYXcbrd27typO+64Q2+//TYFkwEnTpzQAw88oLfeektJSUmm40DqNo3jmmuu0bRp03TFFVdo69atWr58ucFkF0axBFjckiVLtHv3bu3fv5+Npg0bMmSIrrzySknS5MmTdejQIf3ud7/Tc889ZzjZ4HP48GHV1dWpoKCg65jf79f+/fv17LO6hd1dAAABt0lEQVTPyufzKT4+3mBCpKSk6JprrtHRo0dNR7kgiiXAooLBoJYsWaJdu3Zp3759ys3NNR0J3xEMBuXz+UzHGJR+/OMf93jS6uc//7muvvpqPfTQQxRKUcDn8+nIkSOaOXOm6SgXRLGEXmtqatKxY8e63n/xxReqqKjQiBEjNHr0aIPJBqfFixfrtdde0xtvvKFhw4aptrZWkuRwODR06FDD6QafX/3qVyoqKlJOTo5Onz6tbdu2ad++fT2eWkRkDBs2rMf8vZSUFF1yySXM6zPkwQcf1OzZszV69GjV1dVp/fr18nq9uuOOO0xHuyCKJfTaBx98oFmzZnW9PzvGfMcdd+ill14ylGrw2rRpkyTphz/8YbfjL774on72s59FPtAgd/LkSS1YsEBOp1MOh0MTJkzQm2++qRtvvNF0NCAqfPXVV5o/f77q6+uVmZmp6667TgcOHNCYMWNMR7sgWzAYDJoOAQAAEK1YZwkAACAEiiUAAIAQKJYAAABCoFgCAAAIgWIJAAAgBIolAACAECiWAAAAQqBYAgAACIFiCQAAIASKJQAAgBAolgAAAEKgWAIAAAjh/wFVjgQaooYWtwAAAABJRU5ErkJggg==\n",
"text/plain": [
"Graphics object consisting of 18 graphics primitives"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show(ecdfPlot(deMs), figsize=[6,3]) # use hidden ecdfPlot function to plot"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can use the empirical cumulative distribution function $\\widehat{F}_n$ for our non-parametric estimate because this kind of estimation is possible in infinite-dimensional contexts due to the following two theorems:\n",
"\n",
"- Glivenko-Cantelli Theorem (*Fundamental Theorem of Statistics*)\n",
"- Dvoretsky-Kiefer-Wolfowitz (DKW) Inequality\n",
"\n",
"# Glivenko-Cantelli Theorem\n",
"\n",
"Let $X_1, X_2, \\ldots, X_n \\overset{IID}{\\sim} F^* \\in \\{\\text{all DFs}\\}$\n",
"\n",
"and the empirical distribution function (EDF) is $\\widehat{F}_n(x) := \\displaystyle\\frac{1}{n} \\sum_{i=1}^n \\mathbf{1}_{[X_i,+\\infty)}(x)$, then\n",
"\n",
"$$\\sup_x { | \\widehat{F}_n(x) - F^*(x) | } \\overset{P}{\\rightarrow} 0$$\n",
"\n",
"Remember that the EDF is a statistic of the data, a statistic is an RV, and (from our work the convergence of random variables), $\\overset{P}{\\rightarrow}$ means \"converges in probability\". The proof is beyond the scope of this course, but we can gain an appreciation of what it means by looking at what happens to the ECDF for $n$ simulations from:\n",
"\n",
"- $de\\,Moivre(1/5,1/5,1/5,1/5,1/5)$ and \n",
"- $Uniform(0,1)$ as $n$ increases:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "94944fe01bad4a66bdf4a3c57a2cc0d0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"SW50ZXJhY3RpdmUgZnVuY3Rpb24gPGZ1bmN0aW9uIF8gYXQgMHg3ZjhlNTYwOGVlNjA+IHdpdGggMSB3aWRnZXQKICBuOiBTZWxlY3Rpb25TbGlkZXIoZGVzY3JpcHRpb249dSduJywgaW5kZXjigKY=\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"@interact\n",
"def _(n=(10,(0..200))):\n",
" '''Interactive function to plot ecdf for obs from de Moirve (5).'''\n",
" if (n > 0):\n",
" us = [randint(1,5) for i in range(n)]\n",
" p=ecdfPlot(us) # use hidden ecdfPlot function to plot\n",
" #p+=line([(-0.2,0),(0,0),(1,1),(1.2,1)],linestyle=':')\n",
" p.show(figsize=[8,2])"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "55b6ba536ed34196b7a9894da04dfa5f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"SW50ZXJhY3RpdmUgZnVuY3Rpb24gPGZ1bmN0aW9uIF8gYXQgMHg3ZjhlNTYwOGVkZTg+IHdpdGggMSB3aWRnZXQKICBuOiBTZWxlY3Rpb25TbGlkZXIoZGVzY3JpcHRpb249dSduJywgaW5kZXjigKY=\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"@interact\n",
"def _(n=(10,(0..200))):\n",
" '''Interactive function to plot ecdf for obs from Uniform(0,1).'''\n",
" if (n > 0):\n",
" us = [random() for i in range(n)]\n",
" p=ecdfPlot(us) # use hidden ecdfPlot function to plot\n",
" p+=line([(-0.2,0),(0,0),(1,1),(1.2,1)],linestyle='-')\n",
" p.show(figsize=[3,3],aspect_ratio=1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It is clear, that as $n$ increases, the ECDF $\\widehat{F}_n$ gets closer and closer to the true DF $F^*$, $\\displaystyle\\sup_x { | \\widehat{F}_n(x) - F^*(x) | } \\overset{P}{\\rightarrow} 0$.\n",
"\n",
"This will hold no matter what the (possibly unknown) $F^*$ is. Thus, $\\widehat{F}_n$ is a point estimate of $F^*$.\n",
"\n",
"We need to add the DKW Inequality be able to get confidence sets or a 'confidence band' that traps $F^*$ with high probability.\n",
"\n",
"# Dvoretsky-Kiefer-Wolfowitz (DKW) Inequality\n",
"\n",
"Let $X_1, X_2, \\ldots, X_n \\overset{IID}{\\sim} F^* \\in \\{\\text{all DFs}\\}$\n",
"\n",
"and the empirical distribution function (EDF) is $\\widehat{F}_n(x) := \\displaystyle\\frac{1}{n} \\sum_{i=1}^n \\mathbf{1}_{[X_i,+\\infty)}(x)$,\n",
"\n",
"then, for any $\\varepsilon > 0$,\n",
"\n",
"$$P\\left( \\sup_x { | \\widehat{F}_n(x) - F^*(x) | > \\varepsilon }\\right) \\leq 2 \\exp(-2n\\varepsilon^2) $$\n",
"\n",
"We can use this inequality to get a $1-\\alpha$ confidence band $C_n(x) := \\left[\\underline{C}_n(x), \\overline{C}_n(x)\\right]$ about our point estimate $\\widehat{F}_n$ of our possibly unknown $F^*$ such that the $F^*$ is 'trapped' by the band with probability at least $1-\\varepsilon$.\n",
"\n",
"$$\\begin{eqnarray} \\underline{C}_{\\, n}(x) &=& \\max \\{ \\widehat{F}_n(x)-\\varepsilon_n, 0 \\}, \\notag \\\\ \\overline{C}_{\\, n}(x) &=& \\min \\{ \\widehat{F}_n(x)+\\varepsilon_n, 1 \\}, \\notag \\\\ \\varepsilon_n &=& \\sqrt{ \\frac{1}{2n} \\log \\left( \\frac{2}{\\alpha}\\right)} \\\\ \\end{eqnarray}$$\n",
"\n",
"and\n",
"\n",
"$$P\\left(\\underline{C}_n(x) \\leq F^*(x) \\leq \\overline{C}_n(x)\\right) \\geq 1-\\alpha$$\n",
"\n",
" \n",
"\n",
"### YouTry in class\n",
"\n",
"Try this out for a simple sample from the $Uniform(0,1)$, which you can generate using random. First we will just make the point estimate for $F^*$, the EDF $\\widehat{F}_n$"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0.025589067765354856, 0.7269486429797201, 0.6891971339209981, 0.971923744038594, 0.5843896717815658, 0.6491350946802029, 0.31997566787083853, 0.07967737365806449, 0.13906972179081, 0.4277481338507123]\n"
]
}
],
"source": [
"n=10\n",
"uniformSample = [random() for i in range(n)]\n",
"print(uniformSample)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In one of the assessments, you did a question that took you through the steps for getting the list of points that you would plot for an empirical distribution function (EDF). We will do exactly the same thing here.\n",
"\n",
"First we find the unique values in the sample, in order from smallest to largest, and get the frequency with which each unique value occurs:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0.025589067765354856, 0.07967737365806449, 0.13906972179081, 0.31997566787083853, 0.4277481338507123, 0.5843896717815658, 0.6491350946802029, 0.6891971339209981, 0.7269486429797201, 0.971923744038594]\n"
]
}
],
"source": [
"sortedUniqueValuesUniform = sorted(list(set(uniformSample)))\n",
"print(sortedUniqueValuesUniform)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"freqsUniform = [uniformSample.count(i) for i in sortedUniqueValuesUniform]\n",
"freqsUniform"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we accumulate the frequences to get the cumulative frequencies:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from pylab import cumsum\n",
"cumFreqsUniform = list(cumsum(freqsUniform)) # accumulate\n",
"cumFreqsUniform"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And the the relative cumlative frequencies:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[1/10, 1/5, 3/10, 2/5, 1/2, 3/5, 7/10, 4/5, 9/10, 1]"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# cumulative rel freqs as rationals\n",
"cumRelFreqsUniform = [ZZ(i)/len(uniformSample) for i in cumFreqsUniform] \n",
"cumRelFreqsUniform"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And finally zip these up with the sorted unique values to get a list of points we can plot:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[(0.025589067765354856, 1/10),\n",
" (0.07967737365806449, 1/5),\n",
" (0.13906972179081, 3/10),\n",
" (0.31997566787083853, 2/5),\n",
" (0.4277481338507123, 1/2),\n",
" (0.5843896717815658, 3/5),\n",
" (0.6491350946802029, 7/10),\n",
" (0.6891971339209981, 4/5),\n",
" (0.7269486429797201, 9/10),\n",
" (0.971923744038594, 1)]"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ecdfPointsUniform = zip(sortedUniqueValuesUniform, cumRelFreqsUniform)\n",
"ecdfPointsUniform"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is a function that you can just use to do a ECDF plot:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"# ECDF plot given a list of points to plot\n",
"def ecdfPointsPlot(listOfPoints, colour='grey', lines_only=False):\n",
" '''Returns an empirical probability mass function plot from a list of points to plot.\n",
" \n",
" Param listOfPoints is the list of points to plot.\n",
" Param colour is used for plotting the lines, defaulting to grey.\n",
" Param lines_only controls wether only lines are plotted (true) or points are added (false, the default value).\n",
" Returns an ecdf plot graphic.'''\n",
" \n",
" ecdfP = point((0,0), pointsize=\"0\")\n",
" if not lines_only: ecdfP = point(listOfPoints, rgbcolor = \"red\", faceted = false, pointsize=\"20\")\n",
" for k in range(len(listOfPoints)):\n",
" x, kheight = listOfPoints[k] # unpack tuple\n",
" previous_x = 0\n",
" previous_height = 0\n",
" if k > 0:\n",
" previous_x, previous_height = listOfPoints[k-1] # unpack previous tuple\n",
" ecdfP += line([(previous_x, previous_height),(x, previous_height)], rgbcolor=colour)\n",
" ecdfP += line([(x, previous_height),(x, kheight)], rgbcolor=colour, linestyle=\":\")\n",
" if not lines_only: \n",
" ecdfP += points((x, previous_height),rgbcolor = \"white\", faceted = true, pointsize=\"20\")\n",
" # padding\n",
" max_index = len(listOfPoints)-1\n",
" ecdfP += line([(listOfPoints[0][0]-0.2, 0),(listOfPoints[0][0], 0)], rgbcolor=colour)\n",
" ecdfP += line([(listOfPoints[max_index][0], listOfPoints[max_index][1]),(listOfPoints[max_index][0]+0.2,\\\n",
" listOfPoints[max_index][1])],rgbcolor=colour)\n",
" return ecdfP"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This makes the plot of the $\\widehat{F}_{10}$, the point estimate for $F^*$ for these $n=10$ simulated samples."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAk0AAAEgCAYAAACpeEdBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3X90VPWd//FXnPwiJEHTCSHGhCLdKhHFGiMFQqmt0KUuyp62YlYBe6wIjXvUtOcYFpUfrqYR67FbJ2iQbmMXI+qhbLZNWuMPlEq7eFigrlDUxmGsJUvCZGYScjKTO9zvH3wJppBkJiRzbzLPxzk5dT7ce+c1nw7w5vP+zJ0E0zRNAQAAYEAXWB0AAABgNKBoAgAAiABFEwAAQAQomgAAACJA0QQAABABiiYAAIAIUDQBAABEgKIJAAAgAhRNAAAAEaBoAgAAiABFEwAAQAQommzGNE0FAgHxlYAAANgLRZPNdHR0aMKECero6LA6CgAA+AyKpmH29ttva9GiRbr44ouVkJCgHTt2WB0JAGAn4bD03/8tHTxodRJEiaJpmJ04cUIzZszQ008/bXUUAIDd7NkjXXqp9OUvS1dcIc2bJx0/bnUqRCjBZPPMiElISNAvf/lLLV68OOJzAoGAJkyYIL/fr8zMzBFMBwCIqZMnTxVMR470HV++XPr5zy2JhOgkWh0g3gWDQQWDwd7HgUDAwjQAgBGzb19vwdSTlKQ2p/PU+O7d0tGjFgYbfZxOp5KSkmL+vBRNFqusrNT69eutjgEAGGmf6R4cyc/X1mXLzvxaTY0FgUavFStWKDc3N+bPS3tuBEXSnjvXSlN+fj7tOQAYixYskJqa5MvM1LvFxZra3Kxxd90l3XGH1clGFVaa4lRKSopSUlKsjgEAiIWXXpIeeEAXbt+u+c3N0qpV0g9/KCUkWJ0MEaBoAgAgVi68UHr2WXX/5CfyeDwqKChQKgXTqMEtB4ZZZ2en9u/fr/3790uSPv74Y+3fv18ej8fiZAAAu2hvb1ddXZ3a29utjoIosKdpmO3cuVPXX3/9WePLly/XzyP4SCm3HACAsS8cDqurq0tpaWlyOBxWx0GEKJpshqIJAAB7oj0HAECM+Xw+1dfXy+fzWR0FUaBoAgAgxgzDUGtrqwzDsDoKokB7zmZozwEAYE+sNAEAAESAogkAgBhraWlRVVWVWlparI6CKFA02YTL5VJhYaGKi4utjgIAGGHp6ekqKSlRenq61VEQBfY02Qx7mgAAsCdWmgAAiLFgMCi3293nC9thfxRNAADEmNfrVW1trbxer9VREAXaczZDew4Axj7DMBQIBJSZmanExESr4yBC/D8FAECMJSYmKisry+oYiBLtOQAAYszv96uxsVF+v9/qKIgCRRMAADEWCoXkdrsVCoWsjoIosKfJZtjTBGDEdXRIP/+59P77UnGxdNttUmqq1akA26NoshmKJgAjyueTZs+WDh06MzZnjvTGG1JysnW5gFGAjeAAEE82b5YOHVJPUpLanM5TY83N0tat0t//vbXZ4ojX61VjY6NKS0uVl5dndRxEiKLJJlwul1wul8LhsNVRAIxlBw5IktqcTtXcffeZcY9HqqmxKFT86u7utjoCokDRZBNlZWUqKyvrbc8BwIi4+mpp61YZDocuMAwt3rFDzuPHpccek77xDavTxR3n6dU+jArsabIZ9jQBGFF+vzR7tk643Xr/iit0xfvva3xRkfT661JSktXpAFujaLIZiiYAI66zU3r+eengwVOfnrv1ViklxepUgO1RNNkMRROAWOju7pbH41FBQYFSud0AEBFubgkAcai9vV11dXVqb2+3OgowarDSZDOsNAGIhXA4rK6uLqWlpcnhcFgdBxgV+PQcAMQhh8OhjIwMq2MAowrtOQCIQz6fT/X19fL5fFZHAUYNiiYAiEOGYai1tVWGYVgdBRg12NNkM+xpAgDAnlhpAgAAiABFk024XC4VFhaquLjY6igA4kBLS4uqqqrU0tJidRRg1KA9ZzO05wDEQmdnpw4cOKAZM2YoPT3d6jjAqEDRZDMUTQAA2BPtOQCIQ8FgUG63W8Fg0OoowKhB0QQAccjr9aq2tlZer9fqKMCoQXvOZmjPAYgFwzAUCASUmZmpxES+HAKIBL9TACAOJSYmKisry+oYwKhCew4A4pDf71djY6P8fr/VUYBRg6JpENXV1ZoyZYpSU1NVVFSkXbt2DXj8U089pcsuu0zjxo1Tfn6+7r//fnV3d8coLQBEJhQKye12KxQKWR0FGDUomgawbds23XfffVqzZo327dunuXPnauHChfJ4POc8fuvWraqoqNDatWt16NAhbdmyRdu2bdPq1atjnBzAmOfxSN/5jpSZKV12mbRlS1SnZ2dna9WqVcrOzh6hgMDYw0bwAcycOVPXXHONNm3a1Ds2bdo0LV68WJWVlWcdf8899+jQoUN6/fXXe8d+8IMfaM+ePYOuUJ3GRnAAgwqHpcJC6YMP+o6/+KK0ZIk1mYA4wEbwfoRCIe3du1cVFRV9xhcsWKDdu3ef85ySkhL9x3/8h/bs2aPrrrtOzc3Namho0PLly/t9nmAw2Oc+KYFAYHheAICx6/XXewumnqQktTmdp8aff176ylciuoTX61VjY6NKS0uVl5c3UkmBMYWiqR9tbW0Kh8PKycnpM56Tk9PvdzXdeuutam1tVUlJiUzTlGEYWrVq1VmF12dVVlZq/fr1w5odwBj3mX9ctTmdqrn77jO/VlMT1aXYcwlEjqJpEAkJCX0em6Z51thpO3fu1KOPPqrq6mrNnDlTH330ke69917l5ubqoYceOuc5q1evVnl5ee/jQCCg/Pz84XsBAMae+fOl9HSps1OGw6ELDEOLd+yQ8zvfke66K6pLOU+vUgEYFEVTP5xOpxwOx1mrSseOHTtr9em0hx56SEuXLtX3vvc9SdKVV16pEydOaMWKFVqzZo0uuODsffcpKSlKSUkZ/hcAYOyaMEHatk264w5leb36xquv6tIrr9T4Bx6Q+PMEGDF8eq4fycnJKioqUlNTU5/xpqYmzZ49+5zndHV1nVUYORwOmaYp9tsDGFbf/Kb0l79o/Ouv67qXXtL4F16gYAJGGCtNAygvL9fSpUt17bXXatasWaqpqZHH49HKlSslScuWLVNeXl7vJ+kWLVqkJ598Ul/60pd623MPPfSQbrrpJjkcDitfCoCxKDlZ3VdfLY/Ho4LubqWmplqdCBjTKJoGsGTJEh0/flwbNmzQ0aNHNX36dDU0NGjy5MmSJI/H02dl6cEHH1RCQoIefPBBffrpp8rOztaiRYv06KOPWvUSAIxx7e3tqqur04oVK5Sbm2t1HGBM4z5NNsN9mgBEIxwOq6urS2lpaaxoAyOMlSYAGMUcDocyMjKsjgHEBTaCA8Ao5vP5VF9fL5/PZ3UUYMyjaAKAUcwwDLW2tsowDKujAGMee5pshj1NAADYEytNAAAAEaBosgmXy6XCwkIVFxdbHQXAKNLS0qKqqqp+vxMTwPChPWcztOcARKOzs1MHDhzQjBkzlJ6ebnUcYEyjaLIZiiYAAOyJ9hwAjGLBYFBut1vBYNDqKMCYR9EEAKOY1+tVbW2tvF6v1VGAMY/2nM3QngMQDcMwFAgElJmZqcREvuQBGEn8DgOAUSwxMVFZWVlWxwDiAu05ABjF/H6/Ghsb5ff7rY4CjHkUTQBgFdOUfvc7qaFB6uoa0iVCoZDcbrdCodAwhwPwt9jTZDPsaQLixF//Ki1cKP3xj6ceZ2VJL70kff3r1uYC0C9WmgDACj/4wZmCSZK8XmnpUqmnx7pMAAbERnAAsEJDgySpJylJbU7nmfGdO6Xp0yO+jNfrVWNjo0pLS5WXlzfMIQF8FkWTTbhcLrlcLoXDYaujAIiF7GwpENCR/HxtXbbszPju3ad+otTd3T2M4QCcC3uabIY9TUCcqK6Wysrky8zUu8XFmtrcrHFf/rL04x8P6XJOp1NJSUnDHBLAZ7HSBABW+P73pXHjdGF1teZ/8ol0663Sgw9KaWlWJwPQD1aabIaVJiC+dHd3y+PxqKCgQKmpqVbHATAAPj0HABZqb29XXV2d2tvbrY4CYBCsNNkMK01AfAmHw+rq6lJaWpocDofVcQAMgD1NAGAhh8OhjIwMq2MAiADtOQCwkM/nU319vXw+n9VRAAyCogkALGQYhlpbW2UYhtVRAAyCPU02w54mAADsiZUmAACACFA0AYCFWlpaVFVVpZaWFqujABgERZNNuFwuFRYWqri42OooAGIoPT1dJSUlSk9PtzoKgEGwp8lm2NMEAIA9sdIEABYKBoNyu90KBoNWRwEwCIomALCQ1+tVbW2tvF6v1VEADIL2nM3QngPii2EYCgQCyszMVGIiX9IA2Bm/QwHAQomJicrKyrI6BoAI0J4DAAv5/X41NjbK7/dbHQXAICiaBlFdXa0pU6YoNTVVRUVF2rVr14DH+3w+lZWVKTc3V6mpqZo2bZoaGhpilBbAaBMKheR2uxUKhayOAmAQtOcGsG3bNt13332qrq7WnDlz9Oyzz2rhwoU6ePCgCgoKzjo+FApp/vz5mjhxol555RVdcskl+uSTT/gGc2A0+q//krZtk8aNk+68U/ryl0fkabKzs7Vq1aoRuTaA4cVG8AHMnDlT11xzjTZt2tQ7Nm3aNC1evFiVlZVnHf/MM89o48aN+tOf/qSkpKQhPScbwQEb2LBBWrv2zOMLLjhVQH3729ZlAmA5iqZ+hEIhpaWl6eWXX9Y//uM/9o7fe++92r9/v956662zzvnmN7+prKwspaWl6T//8z+VnZ2tf/qnf9IDDzwgh8NxzucJBoN97s8SCASUn59P0QRYJRCQcnOlri71JCWpzek8NT55srR9+7A/ndfrVWNjo0pLS5WXlzfs1wcwfGjP9aOtrU3hcFg5OTl9xnNycvr9jqjm5ma98cYbuu2229TQ0KAPP/xQZWVlMgxDDz/88DnPqays1Pr164c9P4Ah8nikri5JUpvTqZq77z7zazU1I/a03d3dI3ZtAMODomkQCQkJfR6bpnnW2GknT57UxIkTVVNTI4fDoaKiIv31r3/Vxo0b+y2aVq9erfLy8t7Hp1eaAFjkC1+QPvc56fhxGQ6HLjAMLd6xQ85Jk6Tnnx+xp3WeXtECYFsUTf1wOp1yOBxnrSodO3bsrNWn03Jzc5WUlNSnFTdt2jS1tLQoFAopOTn5rHNSUlKUkpIyvOEBDF1qqvTkk9J3v6ssr1ff+O1vden//Z/Gb9p0qm0HIG5xy4F+JCcnq6ioSE1NTX3Gm5qaNHv27HOeM2fOHH300Uc6efJk79gHH3yg3NzccxZMAGxq2TLp/fc1/l/+RdctWaLx+/dLJSVWpwJgMYqmAZSXl+u5557Tz372Mx06dEj333+/PB6PVq5cKUlatmyZVq9e3Xv8qlWrdPz4cd1777364IMP9Otf/1qPPfaYysrKrHoJAIbq8svV/YMf6INFi9TNHbsBiPbcgJYsWaLjx49rw4YNOnr0qKZPn66GhgZNnjxZkuTxeHTBBWfqzvz8fL366qu6//77ddVVVykvL0/33nuvHnjgAateAoDz0N7errq6Oq1YsUK5tOaAuMctB2yG+zQB9hEOh9XV1aW0tLR+bxsCIH6w0gQA/XA4HNzRH0Av9jQBQD98Pp/q6+vl8/msjgLABiiaAKAfhmGotbVVhmFYHQWADbCnyWbY0wQAgD2x0gQAABABiiabcLlcKiwsVHFxsdVRAPx/LS0tqqqq6vf7JgHEF9pzNkN7DrCPzs5OHThwQDNmzFB6errVcQBYjKLJZiiaAACwJ9pzANCPYDAot9utYDBodRQANkDRBAD98Hq9qq2tldfrtToKABugPWcztOcA+zAMQ4FAQJmZmUpM5AsUgHjHnwIA0I/ExERlZWVZHQOATdCeA4B++P1+NTY2yu/3Wx0FgA1QNAGIjGFIJ09anSKmQqGQ3G63QqGQ1VEA2ABFE4CBHTsmfec70rhx0oQJ0v33S3FSRGRnZ2vVqlXKzs62OgoAG2BPE4CBLVki7dx56r87O6WnnpISE6WNGy2NBQCxxqfnbIZPz8FWmpulqVMlST1JSWpzOk+NZ2ScKaTGMK/Xq8bGRpWWliovL8/qOAAsxkqTTbhcLrlcLoXDYaujAGd8pg3X5nSq5u67z/xaTY0FgazR3d1tdQQANsBKk82w0gTbufpq6cABfXLJJfr5HXdo8Y4dcpaUSA8/bHWymHE6nUpKSrI6BgCLUTTZDEUTbOfDD6XSUp04dEjvT5+uK/LzNf5nP5N4fwKIMxRNNkPRBNtyu6XUVGnSJKuTAIAluOUAgIh0T5qkDwIB9vcAiFsUTQAi0t7errq6OrW3t1sdBQAsQXvOZmjPwa7C4bC6urqUlpYmh8NhdRwAiDluOQAgIg6HQxkZGVbHAADL0J4DEBGfz6f6+nr5fD6rowCAJSiaAETEMAy1trbKMAyrowCAJdjTZDPsaQIAwJ5YaQIAAIgARZNNuFwuFRYWqri42OoowDm1tLSoqqpKLS0tVkcBAEvQnrMZ2nOwq87OTh04cEAzZsxQenq61XEAIOYommyGogkAAHuiPQcgIsFgUG63W8Fg0OooAGAJiiYAEfF6vaqtrZXX67U6CgBYgvaczdCeg10ZhqFAIKDMzEwlJvJlAgDiD3/yAYhIYmKisrKyrI4BAJahPQcgIn6/X42NjfL7/VZHAQBLUDRFoLq6WlOmTFFqaqqKioq0a9euiM578cUXlZCQoMWLF49wQoxaPp/0wgvS9u1Sd7fVaQYUCoXkdrsVCoWsjgIAlmBP0yC2bdumpUuXqrq6WnPmzNGzzz6r5557TgcPHlRBQUG/5x05ckRz5szRpZdeqqysLO3YsSOi52NPUxx59VXpW9+SOjtPPb74YqmpSSostDYXAOCcKJoGMXPmTF1zzTXatGlT79i0adO0ePFiVVZWnvOccDisefPm6bvf/a527doln89H0YS+wmHp85+X/vKXvuNf/7r02muWRAIADIyN4AMIhULau3evKioq+owvWLBAu3fv7ve8DRs2KDs7W3feeeegrbxgMNjnvjeBQOD8QmN0OHy4t2DqSUpSm9N5avzgQenTT6UL7Nc593q9amxsVGlpqfLy8qyOAwAxR9E0gLa2NoXDYeXk5PQZz8nJ6ff7t9555x1t2bJF+/fvj+g5KisrtX79+vPOilEmJ0dKSpJ6enQkP19bly0782vPPWddrgh023zvFQCMFIqmCCQkJPR5bJrmWWOS1NHRodtvv12bN2+W8/TKwSBWr16t8vLy3seBQED5+fnnFxj297nPSXfdJVVXy9nWptm7dmlqc7PG3XuvdMstVqcbUKTvbQAYayiaBuB0OuVwOM5aVTp27NhZq0+S9Oc//1lut1uLFi3qHTt58qSkU/e4OXz4sKZOndrnnJSUFKWkpIxAetjev/2bdNlluvCllzRfkiorpSVLrE4FAOgHG8EHMXPmTBUVFam6urp3rLCwUDfffPNZG8G7u7v10Ucf9Rl78MEH1dHRoZ/85Cf64he/qOTk5AGfj43g8ae7u1sej0cFBQVKTU21Og4AoB+sNA2ivLxcS5cu1bXXXqtZs2appqZGHo9HK1eulCQtW7ZMeXl5qqysVGpqqqZPn97n/AsvvFCSzhoHTmtvb1ddXZ1WrFih3Nxcq+MAAPpB0TSIJUuW6Pjx49qwYYOOHj2q6dOnq6GhQZMnT5YkeTweXWDDTzph9Jg4caLKy8uVlpZmdRQAwABoz9kM7TkAAOyJJRLAYj6fT/X19fL5fFZHAQAMgKIJsJhhGGptbZVhGFZHAQAMgPaczdCeAwDAnlhpAgAAiABFE2CxlpYWVVVV9fvVPAAAe6BosgmXy6XCwkIVFxdbHQUxlp6erpKSEqWnp1sdBQAwAPY02Qx7mgAAsCdWmgCLBYNBud1uBYNBq6MAAAZA0QRYzOv1qra2Vl6v1+ooAIAB0J6zGdpz8ccwDAUCAWVmZioxkW82AgC74k9owGKJiYnKysqyOgYAYBC05wCL+f1+NTY2yu/3Wx0FADAAiibAYqFQSG63W6FQyOooAIABsKfJZsbknqYjR6Qnn5T+93+l4mKpvFyaONHqVAAARIWiyWbGXNF09Kh09dXSsWNnxqZOlfbtkzIyrMsFAECU2AiOkfXMM9KxY+pJSlKb03lqrKtLqq2VvvUta7PZhNfrVWNjo0pLS5WXl2d1HABAPyiabMLlcsnlcikcDlsdZXh9/LEkqc3pVM3dd58ZP35cqqmxKJQ9dXd3Wx0BADAA2nM2M+bac889J911lz655BL9/I47tHjHDjmPH5d+8hOppMTqdLbidDqVlJRkdQwAQD8ommxmzBVNoZD0D/+gE++8o/evuEJXvP++xt98s7R1q5SQYHU6AAAiRtFkM2OuaJIk05Ref/3Mp+fmzLE6EQAAUWNPE0ZeQoK6S0rkKShQQUGBUq3OAwDAEHBzS8REe3u76urq1N7ebnUUAACGhPaczYzJ9pykcDisrq4upaWlyeFwWB0HAICo0Z5DTDgcDmVwM0sAwChGew4x4fP5VF9fL5/PZ3UUAACGhKIJMWEYhlpbW2UYhtVRAAAYEvY02cxY3dMEAMBox0oTAABABCiabMLlcqmwsFDFxcVWRxkRLS0tqqqqUktLi9VRAAAYEtpzNjNW23OdnZ06cOCAZsyYofT0dKvjAAAQNYommxmrRRMAAKMd7TnERDAYlNvtVjAYtDoKAABDQtGEmPB6vaqtrZXX67U6CgAAQ0J7zmbGanvOMAwFAgFlZmYqMZEb0QMARh/+9kJMJCYmKisry+oYAAAMGe05xITf71djY6P8fr/VUQAAGBKKpghUV1drypQpSk1NVVFRkXbt2tXvsZs3b9bcuXN10UUX6aKLLtINN9ygPXv2xDDteejulkboPkqhUEhut1uhUGhErg8AwEijaBrEtm3bdN9992nNmjXat2+f5s6dq4ULF8rj8Zzz+J07d6q0tFRvvvmmfv/736ugoEALFizQp59+GuPkUdqwQcrJkXJzpSuukH73u2G9fHZ2tlatWqXs7OxhvS4AALHCRvBBzJw5U9dcc402bdrUOzZt2jQtXrxYlZWVg54fDod10UUX6emnn9ayZcsGPd6SjeDPPy8tX953bMIE6ciRU/8LAADYCD6QUCikvXv3qqKios/4ggULtHv37oiu0dXVpZ6enn43QQeDwT73LgoEAkMPPFS/+IUkqScpSW1O55nxl1+WbrxxWJ7C6/WqsbFRpaWlysvLG5ZrAgAQSxRNA2hra1M4HFZOTk6f8ZycnIi/Q62iokJ5eXm64YYbzvnrlZWVWr9+/XlnPS8JCZKkNqdTNXfffWb800+lmpphfaru7u5hvR4AALFC0RSBhP9fVJxmmuZZY+fy+OOPq66uTjt37lRqauo5j1m9erXKy8t7HwcCAeXn559f4GgtWyY1NclwOHSBYWjxjh1yBoPSr38tDfP3xDk/u5IFAMAoQtE0AKfTKYfDcdaq0rFjx85affpbTzzxhB577DG99tpruuqqq/o9LiUlRSkpKcOSd8huv13yeJT19NP6xm9/q0tTUzW+pkb6u7+zNhcAADbCp+cGkJycrKKiIjU1NfUZb2pq0uzZs/s9b+PGjXrkkUf0m9/8Rtdee+1Ixxwe//IvGu9267qGBo1/911p1iyrEwEAYCusNA2ivLxcS5cu1bXXXqtZs2appqZGHo9HK1eulCQtW7ZMeXl5vZ+ke/zxx/XQQw/phRde0Oc///neVar09HSlD3Ora7h1nzwpj9ergvT0ftuJAADEK1aaBrFkyRI99dRT2rBhg66++mq9/fbbamho0OTJkyVJHo9HR48e7T2+urpaoVBI3/72t5Wbm9v788QTT1j1EiLW3t6uuro6tbe3Wx0FAADb4T5NNmPlF/aGw2F1dXUpLS1NDocjps8NAIDd0Z5DL4fDoYyMDKtjAABgS7Tn0Mvn86m+vl4+n8/qKAAA2A5FE3oZhqHW1lYZhmF1FAAAbIc9TTZj5Z4mAADQP1aaAAAAIkDRZBMul0uFhYUqLi62LENLS4uqqqoi/l49AADiCe05m7GyPdfZ2akDBw5oxowZtr8RJwAAsUbRZDPsaQIAwJ5oz6FXMBiU2+1WMBi0OgoAALZD0YReXq9XtbW18nq9VkcBAMB2aM/ZjJXtOcMwFAgElJmZqcREbhYPAMBn8TcjeiUmJiorK8vqGAAA2BLtOfTy+/1qbGyU3++3OgoAALZD0TQadXdLdXXSk09KBw8O22VDoZDcbrdCodCwXRMAgLGCPU02M+iepk8/lb7yFam5+cxYZaVUURG7kAAAxCFWmkab9ev7FkyS9OCDp4opAAAwYtgIPtrs2iVJ6klKUpvTeWb89del+fPP69Jer1eNjY0qLS1VXl7eeV0LAICxhqLJJlwul1wul8Lh8MAHTp0q/elPOpKfr63Llp0Z//hjqaZmWLJ0d3cPy3UAABhL2NNkM4Puadq9W7r+evlSU/VucbGmNjdrXHGx9NRTw5bB6XQqKSlp2K4HAMBYQNFkMxHd3HLfPumnP5WOHpUWLpRWrpSSk2MbFACAOEPRZDOR3hG8u7tbHo9HBQUFSk1NjWFCAADiE5+eG6Xa29tVV1en9vZ2q6MAABAXWGmymUhXmsLhsLq6upSWliaHwxHDhAAAxCc+PTdKORwOZWRkWB0DAIC4QXtulPL5fKqvr5fP57M6CgAAcYGiaZQyDEOtra0yDMPqKAAAxAX2NNlMpHuaAABAbLHSBAAAEAGKJptwuVwqLCxUcXFxRMe3tLSoqqpKLS0tI5wMAABItOdsJ9L2XGdnpw4cOKAZM2YoPT09hgkBAIhPFE02w54mAADsifbcKBUMBuV2uxUMBq2OAgBAXKBoGqW8Xq9qa2vl9XqtjgIAQFygPWczkbbnDMNQIBBQZmamEhO5sTsAACONv21HqcTERGVlZVkdAwCAuEF7bpTy+/1qbGyU3++3OgoAAHGBommUCoVCcrvdCoVCVkcBACAusKfJZrjlAAAA9sRKEwAootvtAAAKG0lEQVQAQARGbCO4aZrq6OgY9Lienh4dP358pGLYXigUUk9PT+9jn88nSdqzZ4/Gjx/f73mBQEB/+MMf9K1vfUsFBQUjnhMAgNEmIyNDCQkJw3a9EWvPtba2auLEiSNxaQAAgEFVVFRo3bp1SklJGZbrjVjR5Pf7deGFF+qTTz4ZcG9ONCtNS5cu1S9+8YthyTec1zqf6/3tStPRo0d14403qr6+Xk6nc9Dzp0yZorS0tAGPKS4u1rvvvht1tpG+1nBeLxAIKD8/f9D3WzSYt6Fh3qzLNRLXG85rMW9Dw7wNzel5G849wiPWnju9HJaZmTlo2M997nMRXbOjo0Nf/OIXzzvbcF9rOK+XnZ0tSfrSl76kSy655LyvJ0kOh2PY3jDDea2RuF4k77dIMW9Dw7xFz86vc7izSczbUDFv1htVG8HLyspsea2RuN5wYt6GhnkbGuYtenZ+nXadM4l5GyrmbehGrD3HR+eH5i9/+UvvMuxwrTTFA95vQ8O8DQ3zNjTM29Awb0MzEvPmWLdu3bphudK5Lu5w6Ktf/SrfjRaFYDCojRs3avXq1QN+eg5n4/02NMzb0DBvQ8O8DQ3zNjTDPW/c3NJmTt+qYbg/JgkAAM4PRRMAAEAERtVGcAAAAKtQNAEAAESAogkAACAC5100maapdevW6eKLL9a4ceP01a9+Ve+///6A51RWVqq4uFgZGRmaOHGiFi9erMOHD59vFIxx1dXVmjJlilJTU1VUVKRdu3b1e+zmzZs1d+5cXXTRRbrooot0ww03aM+ePTFMax/RzNtnvfjii0pISNDixYtHOKE9RTtvPp9PZWVlys3NVWpqqqZNm6aGhoYYpbWPaOftqaee0mWXXaZx48YpPz9f999/v7q7u2OU1t7efvttLVq0SBdffLESEhK0Y8cOqyPZRrRzs337ds2fP1/Z2dnKzMzUrFmz9Nvf/jbq5z3vounxxx/Xk08+qaefflrvvvuuJk2apPnz5w/4Zb1vvfWWysrK9Ic//EFNTU0yDEMLFizQiRMnzjcOxqht27bpvvvu05o1a7Rv3z7NnTtXCxculMfjOefxO3fuVGlpqd588039/ve/V0FBgRYsWKBPP/00xsmtFe28nXbkyBH98Ic/1Ny5c2OU1F6inbdQKKT58+fL7XbrlVde0eHDh7V582bl5eXFOLm1op23rVu3qqKiQmvXrtWhQ4e0ZcsWbdu2TatXr45xcns6ceKEZsyYoaefftrqKLYT7dy8/fbbmj9/vhoaGrR3715df/31WrRokfbt2xfdE5vn4eTJk+akSZPMH/3oR71j3d3d5oQJE8xnnnkm4uscO3bMlGS+9dZb5xMHY9h1111nrly5ss/Y5ZdfblZUVER0vmEYZkZGhllbWzsS8WxrKPNmGIY5Z84c87nnnjOXL19u3nzzzSMd03ainbdNmzaZl156qRkKhWIRz7ainbeysjLza1/7Wp+x8vJys6SkZMQyjlaSzF/+8pdWx7Cloc5NYWGhuX79+qjOOa+Vpo8//lgtLS1asGBB71hKSormzZun3bt3R3wdv98vScrKyjqfOBijQqGQ9u7d2+d9JkkLFiyI+H3W1dWlnp6euHqPDXXeNmzYoOzsbN15550jHdGWhjJv9fX1mjVrlsrKypSTk6Pp06frscceUzgcjkVkWxjKvJWUlGjv3r29rfPm5mY1NDToxhtvHPG8iG8nT55UR0dH1H8nnNctMltaWiRJOTk5fcZzcnJ05MiRiK5hmqbKy8tVUlKi6dOnn08cjFFtbW0Kh8PnfJ+dfg8OpqKiQnl5ebrhhhtGIqItDWXe3nnnHW3ZskX79++PRURbGsq8NTc364033tBtt92mhoYGffjhhyorK5NhGHr44YdjEdtyQ5m3W2+9Va2trSopKZFpmjIMQ6tWrVJFRUUsIiOO/fjHP9aJEyd0yy23RHVeVCtNW7duVXp6eu9PT0+PJJ1152rTNCO+m/U999yjP/7xj6qrq4smCuLQUN9njz/+uOrq6rR9+3alpqaOVDzbinTeOjo6dPvtt2vz5s1yOp2ximdb0bzfTp48qYkTJ6qmpkZFRUW69dZbtWbNGm3atCkWUW0lmnnbuXOnHn30UVVXV+t//ud/tH37dv3qV7/SI488EouoiFN1dXVat26dtm3bpokTJ0Z1blQrTTfddJNmzpzZ+zgYDEo6teKUm5vbO37s2LGz/rVxLv/8z/+s+vp6vf3223w5LfrldDrlcDjO+tdqJO+zJ554Qo899phee+01XXXVVSMZ03ainbc///nPcrvdWrRoUe/YyZMnJUmJiYk6fPiwpk6dOrKhbWAo77fc3FwlJSXJ4XD0jk2bNk0tLS0KhUJKTk4e0cx2MJR5e+ihh7R06VJ973vfkyRdeeWVOnHihFasWKE1a9boggu4Kw6G17Zt23TnnXfq5ZdfHlLnIap3ZEZGhr7whS/0/hQWFmrSpElqamrqPSYUCumtt97S7Nmz+72OaZq65557tH37dr3xxhuaMmVK1MERP5KTk1VUVNTnfSZJTU1NA77PNm7cqEceeUS/+c1vdO211450TNuJdt4uv/xyvffee9q/f3/vz0033aTrr79e+/fvV35+fqyiW2oo77c5c+boo48+6i0yJemDDz5Qbm5uXBRM0tDmraur66zCyOFwyDRNmXzDF4ZZXV2d7rjjDr3wwgtD3zcX9Xbzv/GjH/3InDBhgrl9+3bzvffeM0tLS83c3FwzEAj0HvO1r33N/OlPf9r7eNWqVeaECRPMnTt3mkePHu396erqOt84GKNefPFFMykpydyyZYt58OBB87777jPHjx9vut1u0zRNc+nSpX0+oVNVVWUmJyebr7zySp/3WEdHh1UvwRLRztvfitdPz0U7bx6Px0xPTzfvuece8/Dhw+avfvUrc+LEiea//uu/WvUSLBHtvK1du9bMyMgw6+rqzObmZvPVV181p06dat5yyy1WvQRb6ejoMPft22fu27fPlGQ++eST5r59+8wjR45YHc1yg81NRUWFuXTp0t7jX3jhBTMxMdF0uVx9/k7w+XxRPe95F00nT540165da06aNMlMSUkxv/KVr5jvvfden2MmT55srl279syTSuf8+fd///fzjYMxzOVymZMnTzaTk5PNa665ps8tKubNm2cuX7689/HkyZPP+R777PswXkQzb38rXosm04x+3nbv3m3OnDnTTElJMS+99FLz0UcfNQ3DiHFq60Uzbz09Pea6devMqVOnmqmpqWZ+fr75/e9/32xvb7cguf28+eab5/xzbKDfs/FisLlZvny5OW/evN7j582bNyxzmWCarIECAAAMhl12AAAAEaBoAgAAiABFEwAAQAQomgAAACJA0QQAABABiiYAAIAIUDQBAABEgKIJAAAgAhRNAAAAEaBoAgAAiABFEwAAQAQomgAAACLw/wCmfIpSOu6oNwAAAABJRU5ErkJggg==\n",
"text/plain": [
"Graphics object consisting of 33 graphics primitives"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"show(ecdfPointsPlot(ecdfPointsUniform), figsize=[6,3])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What about adding those confidence bands? You will do essentially the same thing, but adjusting for the required $\\varepsilon$. First we need to decide on an $\\alpha$ and calculate the $\\varepsilon$ corresponding to this alpha. Here is some of our code to calculate the $\\varepsilon$ corresponding to $\\alpha=0.05$ (95% confidence bands), using a hidden function calcEpsilon: "
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.429469408346738"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"alpha = 0.05\n",
"epsilon = calcEpsilon(alpha, n)\n",
"epsilon"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"See if you can write your own code to do this calculation, $\\varepsilon_n = \\sqrt{ \\frac{1}{2n} \\log \\left( \\frac{2}{\\alpha}\\right)}$. For completeness, do the whole thing:assign the value 0.05 to a variable named alpha, and then use this and the variable called n that we have already declared to calculate a value for $\\varepsilon$. Call the variable to which you assign the value for $\\varepsilon$ epsilon so that it replaces the value we calculated in the cell above (you should get the same value as us!)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we need to use this to adjust the EDF plot. In the two cells below we first of all do the adjustment for $\\underline{C}_{\\,n}(x) =\\max \\{ \\widehat{F}_n(x)-\\varepsilon_n, 0 \\}$, and then use zip again to get the points to actually plot for the lower boundary of the 95% confidence band.\n",
"\n",
"Now we need to use this to adjust the EDF plot. In the two cells below we first of all do the adjustment for $\\overline{C}_{\\,n}(x) =\\min \\{ \\widehat{F}_n(x)+\\varepsilon_n, 1 \\}$, and then use zip again to get the points to actually plot for the lower boundary of the 95% confidence band."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0, 0, 0, 0, 0.0705305916532624, 0.170530591653262, 0.270530591653262, 0.370530591653262, 0.470530591653262, 0.570530591653262]\n"
]
}
],
"source": [
"# heights for the lower band\n",
"cumRelFreqsUniformLower = [max(crf - epsilon, 0) for crf in cumRelFreqsUniform] \n",
"print(cumRelFreqsUniformLower)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[(0.025589067765354856, 0),\n",
" (0.07967737365806449, 0),\n",
" (0.13906972179081, 0),\n",
" (0.31997566787083853, 0),\n",
" (0.4277481338507123, 0.0705305916532624),\n",
" (0.5843896717815658, 0.170530591653262),\n",
" (0.6491350946802029, 0.270530591653262),\n",
" (0.6891971339209981, 0.370530591653262),\n",
" (0.7269486429797201, 0.470530591653262),\n",
" (0.971923744038594, 0.570530591653262)]"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ecdfPointsUniformLower = zip(sortedUniqueValuesUniform, cumRelFreqsUniformLower)\n",
"ecdfPointsUniformLower"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We carefully gave our `ecdfPointsPlo`t function the flexibility to be able to plot bands, by having a colour parameter (which defaults to 'grey') and a `lines_only` parameter (which defaults to `false`). Here we can plot the lower bound of the confidence interval by adding `ecdfPointsPlot(ecdfPointsUniformLower, colour='green', lines_only=true)` to the previous plot:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAk0AAAEgCAYAAACpeEdBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3X9409W9B/B3SdqGkhbIUtpaWgbohIriKIUBBWSzODZxvXeb2CngHhFh9Wc3ru0Q+eHVruJ82J0pWmR3dRdq1YcxHM20/qhlsnv1drR6+VmpaVDJaMmvljxJ+g3f+wejWKFt0iY53yTv1/P08clpvifvHkP5cM7J+cbJsiyDiIiIiAY0QnQAIiIiokjAoomIiIjIDyyaiIiIiPzAoomIiIjIDyyaiIiIiPzAoomIiIjIDyyaiIiIiPzAoomIiIjIDyyaiIiIiPzAoomIiIjIDyyaiIiIiPzAoklhZFmG0+kEbwlIRESkLCyaFKarqwujR49GV1eX6ChERET0JSyagqyxsRFLly7FVVddhbi4OOzdu1d0JCIiUhKfD/if/wGOHBGdhALEoinIzp07h+nTp+O5554THYWIiJTmgw+ASZOAb30LuO46YOFC4OxZ0anIT3EyN8+ETFxcHP74xz+isLDQ72ucTidGjx4Nh8OBlJSUEKYjIqKwOn/+QsHU3t63feVK4Pe/FxKJAqMWHSDWeTweeDye3sdOp1NgGiIiCplDh3oLpp74eHTq9RfaDx4ETp8WGCzy6PV6xMfHh/11WTQJVl5ejs2bN4uOQUREofal1YP2rCzsWrHi0veqqgQEilyrV69GRkZG2F+Xy3Mh5M/y3JVmmrKysrg8R0QUjRYvBurrYU9JwYd5eZjc1oaR994L3H236GQRhTNNMSoxMRGJiYmiYxARUTi88grw6KMYs2cPCtragLVrgV/8AoiLE52M/MCiiYiIKFzGjAFeeAHu3/wGZrMZ2dnZ0LBgihg8ciDIuru70dzcjObmZgDAp59+iubmZpjNZsHJiIhIKWw2G2pqamCz2URHoQBwT1OQNTQ0YNGiRZe1r1y5Er/34yOlPHKAiCj6+Xw+uFwuJCUlQaVSiY5DfmLRpDAsmoiIiJSJy3NERERhZrfbsW/fPtjtdtFRKAAsmoiIiMJMkiR0dHRAkiTRUSgAXJ5TGC7PERERKRNnmoiIiIj8wKKJiIgozCwWCyoqKmCxWERHoQCwaFIIg8GAnJwc5OXliY5CREQhptVqkZ+fD61WKzoKBYB7mhSGe5qIiIiUiTNNREREYebxeGAymfrcsJ2Uj0UTERFRmFmtVlRXV8NqtYqOQgHg8pzCcHmOiCj6SZIEp9OJlJQUqNVq0XHIT/w/RUREFGZqtRo6nU50DAoQl+eIiIjCzOFwwGg0wuFwiI5CAWDRREREFGZerxcmkwler1d0FAoA9zQpDPc0EVHIdXUBv/89cPgwkJcH3HknoNGITkWkeCyaFIZFExGFlN0OzJ0LHD16qW3ePOCdd4CEBHG5iCIAN4ITEcWSHTuAo0fREx+PTr3+QltbG7BrF/Dd74rNFkOsViuMRiOKioqQmZkpOg75iUWTQhgMBhgMBvh8PtFRiCiatbQAADr1elTdd9+ldrMZqKoSFCp2ud1u0REoACyaFKK4uBjFxcW9y3NERCFx443Arl2QVCqMkCQU7t0L/dmzwFNPAbfcIjpdzNFfnO2jiMA9TQrDPU1EFFIOBzB3Ls6ZTDh83XW47vBhjMrNBd5+G4iPF52OSNFYNCkMiyYiCrnubuCll4AjRy58eu6OO4DERNGpiBSPRZPCsGgionBwu90wm83Izs6GhscNEPmFh1sSEcUgm82Gmpoa2Gw20VGIIgZnmhSGM01EFA4+nw8ulwtJSUlQqVSi4xBFBH56jogoBqlUKiQnJ4uOQRRRuDxHRBSD7HY79u3bB7vdLjoKUcRg0UREFIMkSUJHRwckSRIdhShicE+TwnBPExERkTJxpomIiIjIDyyaFMJgMCAnJwd5eXmioxBRDLBYLKioqIDFYhEdhShicHlOYbg8R0Th0N3djZaWFkyfPh1arVZ0HKKIwKJJYVg0ERERKROX54iIYpDH44HJZILH4xEdhShisGgiIopBVqsV1dXVsFqtoqMQRQwuzykMl+eIKBwkSYLT6URKSgrUat4cgsgf/JNCRBSD1Go1dDqd6BhEEYXLc0REMcjhcMBoNMLhcIiOQhQxWDQNorKyEhMnToRGo0Fubi4OHDgw4PO3bduGa6+9FiNHjkRWVhYeeeQRuN3uMKUlIvKP1+uFyWSC1+sVHYUoYrBoGkBtbS0efvhhrF+/HocOHcL8+fOxZMkSmM3mKz5/165dKC0txcaNG3H06FHs3LkTtbW1KCsrC3NyIop6ZjPw4x8DKSnAtdcCO3cGdHlqairWrl2L1NTUEAUkij7cCD6A2bNnY8aMGdi+fXtv29SpU1FYWIjy8vLLnn///ffj6NGjePvtt3vbfv7zn+ODDz4YdIbqIm4EJ6JB+XxATg5w4kTf9pdfBpYtE5OJKAZwI3g/vF4vmpqaUFpa2qd98eLFOHjw4BWvyc/Px3/913/hgw8+wKxZs9DW1oa6ujqsXLmy39fxeDx9zklxOp3B+QGIKHq9/XZvwdQTH49Ovf5C+0svAQsW+NWF1WqF0WhEUVERMjMzQ5WUKKqwaOpHZ2cnfD4f0tLS+rSnpaX1e6+mO+64Ax0dHcjPz4csy5AkCWvXrr2s8Pqy8vJybN68OajZiSjKfekfV516Paruu+/S96qqAuqKey6J/MeiaRBxcXF9HsuyfFnbRQ0NDXjyySdRWVmJ2bNn45NPPsFDDz2EjIwMbNiw4YrXlJWVoaSkpPex0+lEVlZW8H4AIoo+BQWAVgt0d0NSqTBCklC4dy/0P/4xcO+9AXWlvzhLRUSDYtHUD71eD5VKddms0pkzZy6bfbpow4YNWL58OVatWgUAuP7663Hu3DmsXr0a69evx4gRl++7T0xMRGJiYvB/ACKKXqNHA7W1wN13Q2e14pY338Sk66/HqEcfBfj7hChk+Om5fiQkJCA3Nxf19fV92uvr6zF37twrXuNyuS4rjFQqFWRZBvfbE1FQfe97wGefYdTbb2PWK69g1O7dLJiIQowzTQMoKSnB8uXLMXPmTMyZMwdVVVUwm81Ys2YNAGDFihXIzMzs/STd0qVL8eyzz+Kb3/xm7/Lchg0bcNttt0GlUon8UYgoGiUkwH3jjTCbzch2u6HRaEQnIopqLJoGsGzZMpw9exZbtmzB6dOnMW3aNNTV1WHChAkAALPZ3Gdm6bHHHkNcXBwee+wxfP7550hNTcXSpUvx5JNPivoRiCjK2Ww21NTUYPXq1cjIyBAdhyiq8ZwmheE5TUQUCJ/PB5fLhaSkJM5oE4UYZ5qIiCKYSqVCcnKy6BhEMYEbwYmIIpjdbse+fftgt9tFRyGKeiyaiIgimCRJ6OjogCRJoqMQRT3uaVIY7mkiIiJSJs40EREREfmBRZNCGAwG5OTkIC8vT3QUIoogFosFFRUV/d4Tk4iCh8tzCsPlOSIKRHd3N1paWjB9+nRotVrRcYiiGosmhWHRREREpExcniMiimAejwcmkwkej0d0FKKox6KJiCiCWa1WVFdXw2q1io5CFPW4PKcwXJ4jokBIkgSn04mUlBSo1bzJA1Eo8U8YEVEEU6vV0Ol0omMQxQQuzxERRTCHwwGj0QiHwyE6ClHUY9FERCSKLAN//StQVwe4XEPqwuv1wmQywev1BjkcEX0V9zQpDPc0EcWIL74AliwBPvrowmOdDnjlFeA73xGbi4j6xZkmIiIRfv7zSwUTAFitwPLlQE+PuExENCBuBCciEqGuDgDQEx+PTr3+UntDAzBtmt/dWK1WGI1GFBUVITMzM8ghiejLWDQphMFggMFggM/nEx2FiMIhNRVwOtGelYVdK1Zcaj948MJXgNxudxDDEdGVcE+TwnBPE1GMqKwEiothT0nBh3l5mNzWhpHf+hbw618PqTu9Xo/4+PgghySiL+NMExGRCD/7GTByJMZUVqLg1CngjjuAxx4DkpJEJyOifnCmSWE400QUW9xuN8xmM7Kzs6HRaETHIaIB8NNzREQC2Ww21NTUwGaziY5CRIPgTJPCcKaJKLb4fD64XC4kJSVBpVKJjkNEA+CeJiIigVQqFZKTk0XHICI/cHmOiEggu92Offv2wW63i45CRINg0UREJJAkSejo6IAkSaKjENEguKdJYbiniYiISJk400RERETkBxZNREQCWSwWVFRUwGKxiI5CRINg0aQQBoMBOTk5yMvLEx2FiMJIq9UiPz8fWq1WdBQiGgT3NCkM9zQREREpE2eaiIgE8ng8MJlM8Hg8oqMQ0SBYNBERCWS1WlFdXQ2r1So6ChENgstzCsPlOaLYIkkSnE4nUlJSoFbzJg1ESsY/oUREAqnVauh0OtExiMgPXJ4jIhLI4XDAaDTC4XCIjkJEg2DRNIjKykpMnDgRGo0Gubm5OHDgwIDPt9vtKC4uRkZGBjQaDaZOnYq6urowpSWiSOP1emEymeD1ekVHIaJBcHluALW1tXj44YdRWVmJefPm4YUXXsCSJUtw5MgRZGdnX/Z8r9eLgoICjBs3Dq+99hrGjx+PU6dO8Q7mRJHo9deB2lpg5EjgnnuAb30rJC+TmpqKtWvXhqRvIgoubgQfwOzZszFjxgxs3769t23q1KkoLCxEeXn5Zc9//vnnsXXrVhw7dgzx8fFDek1uBCdSgC1bgI0bLz0eMeJCAfWjH4nLRETCsWjqh9frRVJSEl599VX8y7/8S2/7Qw89hObmZrz33nuXXfO9730POp0OSUlJ+NOf/oTU1FT85Cc/waOPPgqVSnXF1/F4PH3OZ3E6ncjKymLRRCSK0wlkZAAuF3ri49Gp119onzAB2LMn6C9ntVphNBpRVFSEzMzMoPdPRMHD5bl+dHZ2wufzIS0trU97Wlpav/eIamtrwzvvvIM777wTdXV1aG1tRXFxMSRJwuOPP37Fa8rLy7F58+ag5yeiITKbAZcLANCp16Pqvvsufa+qKmQv63a7Q9Y3EQUHi6ZBxMXF9Xksy/JlbRedP38e48aNQ1VVFVQqFXJzc/HFF19g69at/RZNZWVlKCkp6X18caaJiAS5+mrga18Dzp6FpFJhhCShcO9e6NPTgZdeCtnL6i/OaBGRYrFo6oder4dKpbpsVunMmTOXzT5dlJGRgfj4+D5LcVOnToXFYoHX60VCQsJl1yQmJiIxMTG44Ylo6DQa4NlngZ/+FDqrFbe88QYm/eMfGLV9+4VlOyKKWTxyoB8JCQnIzc1FfX19n/b6+nrMnTv3itfMmzcPn3zyCc6fP9/bduLECWRkZFyxYCIihVqxAjh8GKN++UvMWrYMo5qbgfx80amISDAWTQMoKSnBiy++iN/97nc4evQoHnnkEZjNZqxZswYAsGLFCpSVlfU+f+3atTh79iweeughnDhxAvv378dTTz2F4uJiUT8CEQ3VlClw//znOLF0Kdw8sZuIwOW5AS1btgxnz57Fli1bcPr0aUybNg11dXWYMGECAMBsNmPEiEt1Z1ZWFt5880088sgjuOGGG5CZmYmHHnoIjz76qKgfgYiGwWazoaamBqtXr0YGl+aIYh6PHFAYntNEpBw+nw8ulwtJSUn9HhtCNBSuHheOdR4THSNiTdFPQVJ8UthflzNNRET9UKlUPNGfQuJY5zHkVuWKjhGxmlY3YUbGjLC/LosmIqJ+2O12NDY2YsGCBRgzZozoOBRF3JIb6hFqVBdWY4p+iug4EUfUmLFoIiLqhyRJ6OjogCRJoqNQlLlGdw223bINBZMKkDoqVXQc8hP3NCkM9zQREREpE48cICIiCjOH24H9J/bD4XaIjkIBYNGkEAaDATk5OcjLyxMdhYj+yWKxoKKiot/7TRIN1UnbSdxacytO2k6KjkIB4PKcwnB5jkg5uru70dLSgunTp0Or1YqOQ1Gkx9eDTlcn9El6xKviRcchP7FoUhgWTURERMrE5Tkion54PB6YTCZ4PB7RUSjKtNvbsWrfKrTb20VHoQCwaCIi6ofVakV1dTWsVqvoKBRl3JIbhzsOwy25RUehAHB5TmG4PEekHJIkwel0IiUlBWo1j7UjinX8LUBE1A+1Wg2dTic6BhEpBJfniIj64XA4YDQa4XDwLB0KrhZLC3QVOrRYWkRHoQCwaCIi/0gScP686BRh5fV6YTKZ4PV6RUehKJOuTUdZfhnStemio1AAuKdJYbiniRTnzBmguBjYuxfQaIBVq4CKCiAhQXQyIqKw4kwTEQ1s2TLgtdcuzDR1dwPbtgHr14tORRTRujxdaDA1oMvTJToKBYAzTQrDmSZSlLY2YPJkAEBPfDw69foL7cnJQEODuFxhYrVaYTQaUVRUhMzMTNFxKIr8/fTfkVuVi6bVTZiRMUN0HPITPz2nEAaDAQaDAT6fT3QUoku+tJenU69H1X33XfpeVZWAQGK43TxLh4IrJzUHrQ+0YnzKeNFRKACcaVIYzjSR4tx4I9DSglPjx+P3d9+Nwr17oc/PBx5/XHSysNHr9YiP5/3BiGIdiyaFYdFEitPaChQV4dzRozg8bRquy8rCqN/9DuD7k2jITjlOYevBrVg3dx2yRmeJjkN+4vIcEQ3smmuA//1fjDKZMEujAdL5EWmi4eryXtgIvmbmGtFRKACcaVIYzjSRUrndbpjNZmRnZ0Oj0YiOQ0QUdjxygIj8YrPZUFNTA5vNJjoKEZEQnGlSGM40kVL5fD64XC4kJSVBpVKJjkPD5Opx4VjnMdExYlartRUPGh/E60WvY1bmLNFxyE/c00REflGpVEhOThYdg4LkWOcx5Fblio4R82xuztxGEhZNROQXu92OxsZGLFiwAGPGjBEdh4bJLbmhHqFGdWE1puiniI4Tszj2kYVFExH5RZIkdHR0QJIk0VEoCK7RXYNtt2xDwaQCpI5KFR2HKCJwT5PCcE8TERGRMvHTc0REMcjhdmD/if1wuB2ioxBFDBZNCmEwGJCTk4O8vDzRUYiuyGKxoKKiAhaLRXQUCoKTtpO4teZWnLSdFB2FKGJweU5huDxHStXd3Y2WlhZMnz4dWq1WdBwaph5fDzpdndAn6RGv4n31iPzBoklhWDQREREpE5fniMgvHo8HJpMJHo9HdBQKgnZ7O1btW4V2e7voKEQRg0UTEfnFarWiuroaVqtVdBQKArfkxuGOw3BLbtFRiCIGl+cUhstzpFSSJMHpdCIlJQVqNY94I6LYw998ROQXtVoNnU4nOgYRkTBcniMivzgcDhiNRjgcPNcnGrRYWqCr0KHF0iI6ClHEYNHkh8rKSkycOBEajQa5ubk4cOCAX9e9/PLLiIuLQ2FhYYgTUsSy24Hdu4E9ewC3sveWeL1emEwmeL1e0VEoCNK16SjLL0O6Nl10FKKIwT1Ng6itrcXy5ctRWVmJefPm4YUXXsCLL76II0eOIDs7u9/r2tvbMW/ePEyaNAk6nQ579+716/W4pymGvPkm8MMfAt3dFx5fdRVQXw/k5IjNRUREV8SiaRCzZ8/GjBkzsH379t62qVOnorCwEOXl5Ve8xufzYeHChfjpT3+KAwcOwG63s2iivnw+4OtfBz77rG/7d74DvPWWkEgUW7o8XWg63YTcjFwkJyaLjkMUEbgRfABerxdNTU0oLS3t07548WIcPHiw3+u2bNmC1NRU3HPPPYMu5Xk8nj7n3jidzuGFpshw/HhvwdQTH49Ovf5C+5EjwOefAyOUt3JutVphNBpRVFSEzMxM0XFomFqtrVhUvQhNq5swI2OG6DhEEYFF0wA6Ozvh8/mQlpbWpz0tLa3f+2+9//772LlzJ5qbm/16jfLycmzevHnYWSnCpKUB8fFATw/as7Kwa8WKS9978UVxufzgVvjeK/JPTmoOWh9oxfiU8aKjEEUMFk1+iIuL6/NYluXL2gCgq6sLd911F3bs2AH9xZmDQZSVlaGkpKT3sdPpRFZW1vACk/J97WvAvfcClZXQd3Zi7oEDmNzWhpEPPQTcfrvodAPy971NyqZRa3C17mrRMYgiCoumAej1eqhUqstmlc6cOXPZ7BMAnDx5EiaTCUuXLu1tO3/+PIALZ9wcP34ckydP7nNNYmIiEhMTQ5CeFO8//gO49lqMeeUVFABAeTmwbJnoVBQjTjlOYevBrVg3dx2yRvMfakT+UN7GCQVJSEhAbm4u6uvr+7TX19dj7ty5lz1/ypQp+Pjjj9Hc3Nz7ddttt2HRokVobm7mDBL1pVIBDz4I91tv4URlJdw/+IHoRBRDurxdaDA1oMvbJToKUcTgTNMgSkpKsHz5csycORNz5sxBVVUVzGYz1qxZAwBYsWIFMjMzUV5eDo1Gg2nTpvW5fsyYMQBwWTvRRTabDTU1NVi9ejUyMjJEx6EYkZOag4/WfiQ6BlFEYdE0iGXLluHs2bPYsmULTp8+jWnTpqGurg4TJkwAAJjNZoxQ4CedKHKMGzcOJSUlSEpKEh2FiIgGwHOaFIbnNBFRIFw9LhzrPBbwda3WVjxofBCvF72OWZmzQpCMKPpwpolIMLvdjsbGRixYsKB3OZfIXwfaD+C7u7475OttblsQ0xBFNxZNRIJJkoSOjg5IkiQ6CkWgKfopeHTuoyiYXICxI8cO6Xoi8g+X5xSGy3NERETKxB3MREQRzOF2YP+J/XC4HaKjEEU9Fk1EglksFlRUVPR7ax6igZy0ncStNbfipO2k6ChEUY97mhTCYDDAYDDA5/OJjkJhptVqkZ+fD61WKzoKRaDrx12PL0q+gD6Jt7chCjXuaVIY7mkiIiJSJi7PEQnm8XhgMpng8XhER6EI1G5vx6p9q9BubxcdhSjqsWgiEsxqtaK6uhpWq1V0FIpAbsmNwx2H4ZbcoqMQRT0uzykMl+dijyRJcDqdSElJgVrNbYZERErF39BEgqnVauh0OtExiIhoEFyeIxLM4XDAaDTC4eA5OxS4FksLdBU6tFhaREchinosmogE83q9MJlM8Hq9oqNQBErXpqMsvwzp2nTRUYiiHvc0KUxU7mlqbweefRb4v/8D8vKAkhJg3DjRqYiIiALCoklhoq5oOn0auPFG4MyZS22TJwOHDgHJyeJyEUWJLk8Xmk43ITcjF8mJ/DNFFErcCE6h9fzzwJkz6ImPR6f+nycWu1xAdTXwwx+KzaYQVqsVRqMRRUVFyMzMFB2HIkyrtRWLqhehaXUTZmTMEB2HKKqxaFKIqL2NyqefAgA69XpU3XffpfazZ4GqKkGhlMnt5jk7FLic1By0PtCK8SnjRUchinpcnlOYqFuee/FF4N57cWr8ePz+7rtRuHcv9GfPAr/5DZCfLzqdouj1esTHx4uOQURE/WDRpDBRVzR5vcCtt+Lc++/j8HXX4brDhzHqBz8Adu0C4uJEpyOKeKccp7D14Fasm7sOWaOzRMchimo8coBCKyEBeOMNjPrTnzDrJz/BqDffBHbvZsFEFCRd3i40mBrQ5e0SHYUo6nGmSWGibqbpn9xuN8xmM7Kzs6HRaETHISIiChhnmigsbDYbampqYLPZREchIiIaEs40KUy0zjT5fD64XC4kJSVBpVKJjkOkGK4eF451Hhvy9a3WVjxofBCvF72OWZmzgpiMiL6KRw5QWKhUKiTzMEuiyxzrPIbcqtxh92NzcxaXKNRYNFFY2O12NDY2YsGCBRgzZozoOESK4ZbcUI9Qo7qwGlP0U4bcz3CuJSL/sGiisJAkCR0dHZAkSXQUIkW5RncNtt2yDQWTCpA6KlV0HCIaAPc0KUy07mkiIiKKdPz0HBGRQA63A/tP7IfD7RAdhYgGwaJJIQwGA3JycpCXlyc6SkhYLBZUVFTAYrGIjkKkKCdtJ3Frza04aTspOgoRDYLLcwoTrctz3d3daGlpwfTp06HVakXHIVKMHl8POl2d0CfpEa/ivQeJlIxFk8JEa9FEREQU6bg8R2Hh8XhgMpng8XhERyFSlHZ7O1btW4V2e7voKEQ0CBZNFBZWqxXV1dWwWq2ioxApilty43DHYbglt+goRDQILs8pTLQuz0mSBKfTiZSUFKjVPB6MiIgiD//2orBQq9XQ6XSiYxAREQ0Zl+coLBwOB4xGIxwOnkVD9GUtlhboKnRosbSIjkJEg2DR5IfKykpMnDgRGo0Gubm5OHDgQL/P3bFjB+bPn4+xY8di7NixuPnmm/HBBx+EMe0wuN1AiM5R8nq9MJlM8Hq9IemfKFKla9NRll+GdG266ChENAgWTYOora3Fww8/jPXr1+PQoUOYP38+lixZArPZfMXnNzQ0oKioCO+++y7+9re/ITs7G4sXL8bnn38e5uQB2rIFSEsDMjKA664D/vrXoHafmpqKtWvXIjWV99Yi+rI0bRrWzVuHNG2a6ChENAhuBB/E7NmzMWPGDGzfvr23berUqSgsLER5efmg1/t8PowdOxbPPfccVqxYMejzhWwEf+klYOXKvm2jRwPt7Rf+S0Qh0+XpQtPpJuRm5CI5MVl0HCIaADeCD8Dr9aKpqQmlpaV92hcvXoyDBw/61YfL5UJPT0+/m6A9Hk+fs4ucTufQAw/VH/4AAOiJj0enXn+p/dVXge9/PygvYbVaYTQaUVRUhMzMzKD0SRQNWq2tWFS9CE2rmzAjY4boOEQ0ABZNA+js7ITP50NaWt9p87S0NL/voVZaWorMzEzcfPPNV/x+eXk5Nm/ePOyswxIXBwDo1OtRdd99l9o//xyoqgrqS7ndPIuG6MtyUnPQ+kArxqeMFx2FiAbBoskPcf8sKi6SZfmytit5+umnUVNTg4aGBmg0mis+p6ysDCUlJb2PnU4nsrKyhhc4UCtWAPX1kFQqjJAkFO7dC73HA+zfDwT5PnH6L89kERE0ag2u1l0tOgYR+YFF0wD0ej1UKtVls0pnzpy5bPbpq5555hk89dRTeOutt3DDDTf0+7zExEQkJiYGJe+Q3XUXYDZD99xzuOWNNzBJo8GoqirgmmvE5iKKAaccp7D14Fasm7sOWaPD/A8mIgoIPz03gISEBOTm5qK+vr5Pe32Xfh+PAAAQgklEQVR9PebOndvvdVu3bsUTTzyBv/zlL5g5c2aoYwbHL3+JUSYTZtXVYdSHHwJz5ohORBQTurxdaDA1oMvbJToKEQ2CM02DKCkpwfLlyzFz5kzMmTMHVVVVMJvNWLNmDQBgxYoVyMzM7P0k3dNPP40NGzZg9+7d+PrXv947S6XVaqEN8lJXsLnPn4fZakW2VtvvciIRBVdOag4+WvuR6BhE5AfONA1i2bJl2LZtG7Zs2YIbb7wRjY2NqKurw4QJEwAAZrMZp0+f7n1+ZWUlvF4vfvSjHyEjI6P365lnnhH1I/jNZrOhpqYGNptNdBQiIiLF4TlNCiPyhr0+nw8ulwtJSUlQqVRhfW0iJXL1uHCs81hIX6PV2ooHjQ/i9aLXMStzVkhfi4iGh8tz1EulUiE5mYfrEV10rPMYcqtyw/JaNjdneImUjkUT9bLb7WhsbMSCBQswZswY0XGIhHNLbqhHqFFdWI0p+ikhfa1Q909Ew8eiiXpJkoSOjg5IkiQ6CpEiXKO7Bttu2YaCSQVIHcX7JhLFOu5pUhiRe5qIiIiof/z0HBFRPxxuB/af2A+H2yE6ChEpAIsmhTAYDMjJyUFeXp6wDBaLBRUVFX7fV48o2p20ncStNbfipO2k6ChEpABcnlMYkctz3d3daGlpwfTp0xV/ECdROPT4etDp6oQ+SY94VbzoOEQkGIsmheGeJiIiImXi8hz18ng8MJlM8Hg8oqMQKUK7vR2r9q1Cu71ddBQiUgAWTdTLarWiuroaVqtVdBQiRXBLbhzuOAy35BYdhYgUgMtzCiNyeU6SJDidTqSkpECt5hFeREREX8a/GamXWq2GTqcTHYOIiEiRuDxHvRwOB4xGIxwOnklDBAAtlhboKnRosbSIjkJECsCiKRK53UBNDfDss8CRI0Hr1uv1wmQywev1Bq1PokiWrk1HWX4Z0rXpoqMQkQJwT5PCDLqn6fPPgQULgLa2S23l5UBpafhCEhERxSDONEWazZv7FkwA8NhjF4opIgqqLk8XGkwN6PJ0iY5CRArAjeCR5sABAEBPfDw69fpL7W+/DRQUDKtrq9UKo9GIoqIiZGZmDqsvomjQam3FoupFaFrdhBkZM0THISLBWDQphMFggMFggM/nG/iJkycDx46hPSsLu1asuNT+6adAVVVQsrjdPJOGCAByUnPQ+kArxqeMFx2FiBSAe5oUZtA9TQcPAosWwa7R4MO8PExua8PIvDxg27agZdDr9YiP5322iIiIvoxFk8L4dbjloUPAb38LnD4NLFkCrFkDJCSENyhRDDjlOIWtB7di3dx1yBqdJToOEQnG5blI9M1vwl1ZCbPZjOzsbGhYMBGFRJf3wkbwNTPXiI5CRArAT89FKJvNhpqaGthsNtFRiKJWTmoOPlr7EXJSc0RHISIF4PKcwvh77zmfzweXy4WkpCSoVKowJiQiIopNXJ6LUCqVCsnJyaJjUAxx9bhwrPOY6Bhh1WptxYPGB/F60euYlTlLdBwiEoxFU4Sy2+1obGzEggULMGbMGNFxKAYcaD+A7+76rugYQtjcXAYnIhZNEUuSJHR0dECSJNFRKEakjUrD9eOux6abNuHrY74uOk5YTdFPER2BiBSAe5oUxt89TURERBRe/PQcERERkR9YNCmEwWBATk4O8vLy/Hq+xWJBRUUFLBZLiJMRXdBiaYGuQocWS4voKEREQnB5TmH8XZ7r7u5GS0sLpk+fDq1WG8aEFKv+0f0PvNTyElZMX4E0bZroOEREYceiSWG4p4mIiEiZuDwXoTweD0wmEzwej+goFCO6PBduKdLl6RIdhYhICBZNEcpqtaK6uhpWq1V0FIoRrdZWLKpehFZrq+goRERCcHlOYfxdnpMkCU6nEykpKVCredwWhZ5bcuMz52cYnzIeGrVGdBwiorDj37YRSq1WQ6fTiY5BMUSj1uBq3dWiYxARCcPluQjlcDhgNBrhcDhER6EYccpxCg8aH8QpxynRUYiIhGDRFKG8Xi9MJhO8Xq/oKBQjurz/3Aju5UZwIopN3NOkMDxygIiISJk400RERETkh5BtBJdlGV1dg0/jO1wOfPjph6GKoXi+Hh96pJ7ex522TgBA+R/LkZiU2O91jm4HWk624Cc3/QTfyP5GULK4JTfaHe2YMHpC0D8dFcq+I12kjM1J20n8W/2/ofZHtZh51UzRcYiIBpWcnIy4uLig9Rey5bmOjg6MGzcuFF0TERERDaq0tBSbNm1CYmL/kxCBCFnR5HA4MGbMGJw6dWrAvTmBzDQ9vuFxbHliS1DyBbOv4fT31ZmmLyxfYN2d6/BI1SNIGdP/uHl9Xpx1n8XN192MtNED3wds+fLl+MMf/jBoFn9mPPzty9++h9rfV507dw7fW/I91BnrMGrUqGH3BwQv22B9DWWmSeS4feNr30BSfFK/38/Ly8OHHwZn9jiYfQWzP6fTiaysrEF/v4U7Vyj6C2ZfHLeh4bgNzcVxC+Ye4ZAtz12cDktJSRkwbEpKCrLSs/zq87EvHsO/zvnXoOQLZl/B7O+zzz7DOqxDyZISjB8/PgjJgFFdo7DgGwsU11cw+3M6ncA/gHmT5wXtDwfHbWhUKpUi+wpFf4P9fvOXkn/OYGcDOG5DxXETL6I2ghcXFyuyr1D0F0wct6HhuA0Nxy1wSv45lTpmAMdtqDhuQxey5Tl+dH5oPvvss95p2GDNNMUCvt+GhuM2NBy3oeG4DQ3HbWhCMW6qTZs2bQpKT1fqXKXCTTfdxHujBcDj8WDr1q0oKysL2t6cWMH329Bw3IaG4zY0HLeh4bgNTbDHjYdbKszFoxqC/TFJIiIiGh4WTURERER+iKiN4ERERESisGgiIiIi8gOLJiIiIiI/DLtokmUZmzZtwlVXXYWRI0fipptuwuHDhwe8pry8HHl5eUhOTsa4ceNQWFiI48ePDzcKRbnKykpMnDgRGo0Gubm5OHDgQL/P3bFjB+bPn4+xY8di7NixuPnmm/HBBx+EMa1yBDJuX/byyy8jLi4OhYWFIU6oTIGOm91uR3FxMTIyMqDRaDB16lTU1dWFKa1yBDpu27Ztw7XXXouRI0ciKysLjzzyCNxud5jSKltjYyOWLl2Kq666CnFxcdi7d6/oSIoR6Njs2bMHBQUFSE1NRUpKCubMmYM33ngj4NcddtH09NNP49lnn8Vzzz2HDz/8EOnp6SgoKBjwZr3vvfceiouL8d///d+or6+HJElYvHgxzp07N9w4FKVqa2vx8MMPY/369Th06BDmz5+PJUuWwGw2X/H5DQ0NKCoqwrvvvou//e1vyM7OxuLFi/H555+HOblYgY7bRe3t7fjFL36B+fPnhympsgQ6bl6vFwUFBTCZTHjttddw/Phx7NixA5mZmWFOLlag47Zr1y6UlpZi48aNOHr0KHbu3Ina2lqUlZWFObkynTt3DtOnT8dzzz0nOoriBDo2jY2NKCgoQF1dHZqamrBo0SIsXboUhw4dCuyF5WE4f/68nJ6eLv/qV7/qbXO73fLo0aPl559/3u9+zpw5IwOQ33vvveHEoSg2a9Ysec2aNX3apkyZIpeWlvp1vSRJcnJyslxdXR2KeIo1lHGTJEmeN2+e/OKLL8orV66Uf/CDH4Q6puIEOm7bt2+XJ02aJHu93nDEU6xAx624uFj+9re/3aetpKREzs/PD1nGSAVA/uMf/yg6hiINdWxycnLkzZs3B3TNsGaaPv30U1gsFixevLi3LTExEQsXLsTBgwf97sfhcAAAdDrdcOJQlPJ6vWhqaurzPgOAxYsX+/0+c7lc6Onpian32FDHbcuWLUhNTcU999wT6oiKNJRx27dvH+bMmYPi4mKkpaVh2rRpeOqpp+Dz+cIRWRGGMm75+floamrqXTpva2tDXV0dvv/974c8L8W28+fPo6urK+C/E4Z1RKbFYgEApKWl9WlPS0tDe3u7X33IsoySkhLk5+dj2rRpw4lDUaqzsxM+n++K77OL78HBlJaWIjMzEzfffHMoIirSUMbt/fffx86dO9Hc3ByOiIo0lHFra2vDO++8gzvvvBN1dXVobW1FcXExJEnC448/Ho7Ywg1l3O644w50dHQgPz8fsixDkiSsXbsWpaWl4YhMMezXv/41zp07h9tvvz2g6wKaadq1axe0Wm3vV09PDwBcdnK1LMt+n2Z9//3346OPPkJNTU0gUSgGDfV99vTTT6OmpgZ79uyBRqMJVTzF8nfcurq6cNddd2HHjh3Q6/XhiqdYgbzfzp8/j3HjxqGqqgq5ubm44447sH79emzfvj0cURUlkHFraGjAk08+icrKSvz973/Hnj178Oc//xlPPPFEOKJSjKqpqcGmTZtQW1uLcePGBXRtQDNNt912G2bPnt372OPxALgw45SRkdHbfubMmcv+tXElDzzwAPbt24fGxkbenJb6pdfroVKpLvvXqj/vs2eeeQZPPfUU3nrrLdxwww2hjKk4gY7byZMnYTKZsHTp0t628+fPAwDUajWOHz+OyZMnhza0Agzl/ZaRkYH4+HioVKretqlTp8JiscDr9SIhISGkmZVgKOO2YcMGLF++HKtWrQIAXH/99Th37hxWr16N9evXY8QInopDwVVbW4t77rkHr7766pBWHgJ6RyYnJ+Pqq6/u/crJyUF6ejrq6+t7n+P1evHee+9h7ty5/fYjyzLuv/9+7NmzB++88w4mTpwYcHCKHQkJCcjNze3zPgOA+vr6Ad9nW7duxRNPPIG//OUvmDlzZqhjKk6g4zZlyhR8/PHHaG5u7v267bbbsGjRIjQ3NyMrKytc0YUayvtt3rx5+OSTT3qLTAA4ceIEMjIyYqJgAoY2bi6X67LCSKVSQZZlyLzDFwVZTU0N7r77buzevXvo++YC3m7+Fb/61a/k0aNHy3v27JE//vhjuaioSM7IyJCdTmfvc7797W/Lv/3tb3sfr127Vh49erTc0NAgnz59uvfL5XINNw5FqZdfflmOj4+Xd+7cKR85ckR++OGH5VGjRskmk0mWZVlevnx5n0/oVFRUyAkJCfJrr73W5z3W1dUl6kcQItBx+6pY/fRcoONmNptlrVYr33///fLx48flP//5z/K4cePkf//3fxf1IwgR6Lht3LhRTk5OlmtqauS2tjb5zTfflCdPnizffvvton4ERenq6pIPHTokHzp0SAYgP/vss/KhQ4fk9vZ20dGEG2xsSktL5eXLl/c+f/fu3bJarZYNBkOfvxPsdntArzvsoun8+fPyxo0b5fT0dDkxMVFesGCB/PHHH/d5zoQJE+SNGzdeelHgil//+Z//Odw4FMUMBoM8YcIEOSEhQZ4xY0afIyoWLlwor1y5svfxhAkTrvge+/L7MFYEMm5fFatFkywHPm4HDx6UZ8+eLScmJsqTJk2Sn3zySVmSpDCnFi+Qcevp6ZE3bdokT548WdZoNHJWVpb8s5/9TLbZbAKSK8+77757xd9jA/2ZjRWDjc3KlSvlhQsX9j5/4cKFQRnLOFnmHCgRERHRYLjLjoiIiMgPLJqIiIiI/MCiiYiIiMgPLJqIiIiI/MCiiYiIiMgPLJqIiIiI/MCiiYiIiMgPLJqIiIiI/MCiiYiIiMgPLJqIiIiI/MCiiYiIiMgPLJqIiIiI/PD/PQP0hjCuXVUAAAAASUVORK5CYII=\n",
"text/plain": [
"Graphics object consisting of 56 graphics primitives"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"pointEstimate = ecdfPointsPlot(ecdfPointsUniform)\n",
"lowerBound = ecdfPointsPlot(ecdfPointsUniformLower, colour='green', lines_only=true)\n",
"show(pointEstimate + lowerBound, figsize=[6,3])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### YouTry \n",
"You try writing the code to create the list of points needed for plotting the upper band $\\overline{C}_{\\,n}(x) =\\min \\{ \\widehat{F}_n(x)+\\varepsilon_n, 1 \\}$. You will need to first of all get the upper heights (call them say `cumRelFreqsUniformUpper`) and then `zip` them up with the `sortedUniqueValuesUniform` to get the points to plot."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"# heights for the upper band\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once you have got done this you can add them to the plot by altering the code below:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAk0AAAEgCAYAAACpeEdBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3X9409W9B/B3SdqGkhbIUtpaWgbohIriKIUBBWSzODZxvXeb2CngHhFh9Wc3ru0Q+eHVruJ82J0pWmR3dRdq1YcxHM20/qhlsnv1drR6+VmpaVDJaMmvljxJ+g3f+wejWKFt0iY53yTv1/P08clpvifvHkP5cM7J+cbJsiyDiIiIiAY0QnQAIiIiokjAoomIiIjIDyyaiIiIiPzAoomIiIjIDyyaiIiIiPzAoomIiIjIDyyaiIiIiPzAoomIiIjIDyyaiIiIiPzAoomIiIjIDyyaiIiIiPzAoklhZFmG0+kEbwlIRESkLCyaFKarqwujR49GV1eX6ChERET0JSyagqyxsRFLly7FVVddhbi4OOzdu1d0JCIiUhKfD/if/wGOHBGdhALEoinIzp07h+nTp+O5554THYWIiJTmgw+ASZOAb30LuO46YOFC4OxZ0anIT3EyN8+ETFxcHP74xz+isLDQ72ucTidGjx4Nh8OBlJSUEKYjIqKwOn/+QsHU3t63feVK4Pe/FxKJAqMWHSDWeTweeDye3sdOp1NgGiIiCplDh3oLpp74eHTq9RfaDx4ETp8WGCzy6PV6xMfHh/11WTQJVl5ejs2bN4uOQUREofal1YP2rCzsWrHi0veqqgQEilyrV69GRkZG2F+Xy3Mh5M/y3JVmmrKysrg8R0QUjRYvBurrYU9JwYd5eZjc1oaR994L3H236GQRhTNNMSoxMRGJiYmiYxARUTi88grw6KMYs2cPCtragLVrgV/8AoiLE52M/MCiiYiIKFzGjAFeeAHu3/wGZrMZ2dnZ0LBgihg8ciDIuru70dzcjObmZgDAp59+iubmZpjNZsHJiIhIKWw2G2pqamCz2URHoQBwT1OQNTQ0YNGiRZe1r1y5Er/34yOlPHKAiCj6+Xw+uFwuJCUlQaVSiY5DfmLRpDAsmoiIiJSJy3NERERhZrfbsW/fPtjtdtFRKAAsmoiIiMJMkiR0dHRAkiTRUSgAXJ5TGC7PERERKRNnmoiIiIj8wKKJiIgozCwWCyoqKmCxWERHoQCwaFIIg8GAnJwc5OXliY5CREQhptVqkZ+fD61WKzoKBYB7mhSGe5qIiIiUiTNNREREYebxeGAymfrcsJ2Uj0UTERFRmFmtVlRXV8NqtYqOQgHg8pzCcHmOiCj6SZIEp9OJlJQUqNVq0XHIT/w/RUREFGZqtRo6nU50DAoQl+eIiIjCzOFwwGg0wuFwiI5CAWDRREREFGZerxcmkwler1d0FAoA9zQpDPc0EVHIdXUBv/89cPgwkJcH3HknoNGITkWkeCyaFIZFExGFlN0OzJ0LHD16qW3ePOCdd4CEBHG5iCIAN4ITEcWSHTuAo0fREx+PTr3+QltbG7BrF/Dd74rNFkOsViuMRiOKioqQmZkpOg75iUWTQhgMBhgMBvh8PtFRiCiatbQAADr1elTdd9+ldrMZqKoSFCp2ud1u0REoACyaFKK4uBjFxcW9y3NERCFx443Arl2QVCqMkCQU7t0L/dmzwFNPAbfcIjpdzNFfnO2jiMA9TQrDPU1EFFIOBzB3Ls6ZTDh83XW47vBhjMrNBd5+G4iPF52OSNFYNCkMiyYiCrnubuCll4AjRy58eu6OO4DERNGpiBSPRZPCsGgionBwu90wm83Izs6GhscNEPmFh1sSEcUgm82Gmpoa2Gw20VGIIgZnmhSGM01EFA4+nw8ulwtJSUlQqVSi4xBFBH56jogoBqlUKiQnJ4uOQRRRuDxHRBSD7HY79u3bB7vdLjoKUcRg0UREFIMkSUJHRwckSRIdhShicE+TwnBPExERkTJxpomIiIjIDyyaFMJgMCAnJwd5eXmioxBRDLBYLKioqIDFYhEdhShicHlOYbg8R0Th0N3djZaWFkyfPh1arVZ0HKKIwKJJYVg0ERERKROX54iIYpDH44HJZILH4xEdhShisGgiIopBVqsV1dXVsFqtoqMQRQwuzykMl+eIKBwkSYLT6URKSgrUat4cgsgf/JNCRBSD1Go1dDqd6BhEEYXLc0REMcjhcMBoNMLhcIiOQhQxWDQNorKyEhMnToRGo0Fubi4OHDgw4PO3bduGa6+9FiNHjkRWVhYeeeQRuN3uMKUlIvKP1+uFyWSC1+sVHYUoYrBoGkBtbS0efvhhrF+/HocOHcL8+fOxZMkSmM3mKz5/165dKC0txcaNG3H06FHs3LkTtbW1KCsrC3NyIop6ZjPw4x8DKSnAtdcCO3cGdHlqairWrl2L1NTUEAUkij7cCD6A2bNnY8aMGdi+fXtv29SpU1FYWIjy8vLLnn///ffj6NGjePvtt3vbfv7zn+ODDz4YdIbqIm4EJ6JB+XxATg5w4kTf9pdfBpYtE5OJKAZwI3g/vF4vmpqaUFpa2qd98eLFOHjw4BWvyc/Px3/913/hgw8+wKxZs9DW1oa6ujqsXLmy39fxeDx9zklxOp3B+QGIKHq9/XZvwdQTH49Ovf5C+0svAQsW+NWF1WqF0WhEUVERMjMzQ5WUKKqwaOpHZ2cnfD4f0tLS+rSnpaX1e6+mO+64Ax0dHcjPz4csy5AkCWvXrr2s8Pqy8vJybN68OajZiSjKfekfV516Paruu+/S96qqAuqKey6J/MeiaRBxcXF9HsuyfFnbRQ0NDXjyySdRWVmJ2bNn45NPPsFDDz2EjIwMbNiw4YrXlJWVoaSkpPex0+lEVlZW8H4AIoo+BQWAVgt0d0NSqTBCklC4dy/0P/4xcO+9AXWlvzhLRUSDYtHUD71eD5VKddms0pkzZy6bfbpow4YNWL58OVatWgUAuP7663Hu3DmsXr0a69evx4gRl++7T0xMRGJiYvB/ACKKXqNHA7W1wN13Q2e14pY338Sk66/HqEcfBfj7hChk+Om5fiQkJCA3Nxf19fV92uvr6zF37twrXuNyuS4rjFQqFWRZBvfbE1FQfe97wGefYdTbb2PWK69g1O7dLJiIQowzTQMoKSnB8uXLMXPmTMyZMwdVVVUwm81Ys2YNAGDFihXIzMzs/STd0qVL8eyzz+Kb3/xm7/Lchg0bcNttt0GlUon8UYgoGiUkwH3jjTCbzch2u6HRaEQnIopqLJoGsGzZMpw9exZbtmzB6dOnMW3aNNTV1WHChAkAALPZ3Gdm6bHHHkNcXBwee+wxfP7550hNTcXSpUvx5JNPivoRiCjK2Ww21NTUYPXq1cjIyBAdhyiq8ZwmheE5TUQUCJ/PB5fLhaSkJM5oE4UYZ5qIiCKYSqVCcnKy6BhEMYEbwYmIIpjdbse+fftgt9tFRyGKeiyaiIgimCRJ6OjogCRJoqMQRT3uaVIY7mkiIiJSJs40EREREfmBRZNCGAwG5OTkIC8vT3QUIoogFosFFRUV/d4Tk4iCh8tzCsPlOSIKRHd3N1paWjB9+nRotVrRcYiiGosmhWHRREREpExcniMiimAejwcmkwkej0d0FKKox6KJiCiCWa1WVFdXw2q1io5CFPW4PKcwXJ4jokBIkgSn04mUlBSo1bzJA1Eo8U8YEVEEU6vV0Ol0omMQxQQuzxERRTCHwwGj0QiHwyE6ClHUY9FERCSKLAN//StQVwe4XEPqwuv1wmQywev1BjkcEX0V9zQpDPc0EcWIL74AliwBPvrowmOdDnjlFeA73xGbi4j6xZkmIiIRfv7zSwUTAFitwPLlQE+PuExENCBuBCciEqGuDgDQEx+PTr3+UntDAzBtmt/dWK1WGI1GFBUVITMzM8ghiejLWDQphMFggMFggM/nEx2FiMIhNRVwOtGelYVdK1Zcaj948MJXgNxudxDDEdGVcE+TwnBPE1GMqKwEiothT0nBh3l5mNzWhpHf+hbw618PqTu9Xo/4+PgghySiL+NMExGRCD/7GTByJMZUVqLg1CngjjuAxx4DkpJEJyOifnCmSWE400QUW9xuN8xmM7Kzs6HRaETHIaIB8NNzREQC2Ww21NTUwGaziY5CRIPgTJPCcKaJKLb4fD64XC4kJSVBpVKJjkNEA+CeJiIigVQqFZKTk0XHICI/cHmOiEggu92Offv2wW63i45CRINg0UREJJAkSejo6IAkSaKjENEguKdJYbiniYiISJk400RERETkBxZNREQCWSwWVFRUwGKxiI5CRINg0aQQBoMBOTk5yMvLEx2FiMJIq9UiPz8fWq1WdBQiGgT3NCkM9zQREREpE2eaiIgE8ng8MJlM8Hg8oqMQ0SBYNBERCWS1WlFdXQ2r1So6ChENgstzCsPlOaLYIkkSnE4nUlJSoFbzJg1ESsY/oUREAqnVauh0OtExiMgPXJ4jIhLI4XDAaDTC4XCIjkJEg2DRNIjKykpMnDgRGo0Gubm5OHDgwIDPt9vtKC4uRkZGBjQaDaZOnYq6urowpSWiSOP1emEymeD1ekVHIaJBcHluALW1tXj44YdRWVmJefPm4YUXXsCSJUtw5MgRZGdnX/Z8r9eLgoICjBs3Dq+99hrGjx+PU6dO8Q7mRJHo9deB2lpg5EjgnnuAb30rJC+TmpqKtWvXhqRvIgoubgQfwOzZszFjxgxs3769t23q1KkoLCxEeXn5Zc9//vnnsXXrVhw7dgzx8fFDek1uBCdSgC1bgI0bLz0eMeJCAfWjH4nLRETCsWjqh9frRVJSEl599VX8y7/8S2/7Qw89hObmZrz33nuXXfO9730POp0OSUlJ+NOf/oTU1FT85Cc/waOPPgqVSnXF1/F4PH3OZ3E6ncjKymLRRCSK0wlkZAAuF3ri49Gp119onzAB2LMn6C9ntVphNBpRVFSEzMzMoPdPRMHD5bl+dHZ2wufzIS0trU97Wlpav/eIamtrwzvvvIM777wTdXV1aG1tRXFxMSRJwuOPP37Fa8rLy7F58+ag5yeiITKbAZcLANCp16Pqvvsufa+qKmQv63a7Q9Y3EQUHi6ZBxMXF9Xksy/JlbRedP38e48aNQ1VVFVQqFXJzc/HFF19g69at/RZNZWVlKCkp6X18caaJiAS5+mrga18Dzp6FpFJhhCShcO9e6NPTgZdeCtnL6i/OaBGRYrFo6oder4dKpbpsVunMmTOXzT5dlJGRgfj4+D5LcVOnToXFYoHX60VCQsJl1yQmJiIxMTG44Ylo6DQa4NlngZ/+FDqrFbe88QYm/eMfGLV9+4VlOyKKWTxyoB8JCQnIzc1FfX19n/b6+nrMnTv3itfMmzcPn3zyCc6fP9/bduLECWRkZFyxYCIihVqxAjh8GKN++UvMWrYMo5qbgfx80amISDAWTQMoKSnBiy++iN/97nc4evQoHnnkEZjNZqxZswYAsGLFCpSVlfU+f+3atTh79iweeughnDhxAvv378dTTz2F4uJiUT8CEQ3VlClw//znOLF0Kdw8sZuIwOW5AS1btgxnz57Fli1bcPr0aUybNg11dXWYMGECAMBsNmPEiEt1Z1ZWFt5880088sgjuOGGG5CZmYmHHnoIjz76qKgfgYiGwWazoaamBqtXr0YGl+aIYh6PHFAYntNEpBw+nw8ulwtJSUn9HhtCNBSuHheOdR4THSNiTdFPQVJ8UthflzNNRET9UKlUPNGfQuJY5zHkVuWKjhGxmlY3YUbGjLC/LosmIqJ+2O12NDY2YsGCBRgzZozoOBRF3JIb6hFqVBdWY4p+iug4EUfUmLFoIiLqhyRJ6OjogCRJoqNQlLlGdw223bINBZMKkDoqVXQc8hP3NCkM9zQREREpE48cICIiCjOH24H9J/bD4XaIjkIBYNGkEAaDATk5OcjLyxMdhYj+yWKxoKKiot/7TRIN1UnbSdxacytO2k6KjkIB4PKcwnB5jkg5uru70dLSgunTp0Or1YqOQ1Gkx9eDTlcn9El6xKviRcchP7FoUhgWTURERMrE5Tkion54PB6YTCZ4PB7RUSjKtNvbsWrfKrTb20VHoQCwaCIi6ofVakV1dTWsVqvoKBRl3JIbhzsOwy25RUehAHB5TmG4PEekHJIkwel0IiUlBWo1j7UjinX8LUBE1A+1Wg2dTic6BhEpBJfniIj64XA4YDQa4XDwLB0KrhZLC3QVOrRYWkRHoQCwaCIi/0gScP686BRh5fV6YTKZ4PV6RUehKJOuTUdZfhnStemio1AAuKdJYbiniRTnzBmguBjYuxfQaIBVq4CKCiAhQXQyIqKw4kwTEQ1s2TLgtdcuzDR1dwPbtgHr14tORRTRujxdaDA1oMvTJToKBYAzTQrDmSZSlLY2YPJkAEBPfDw69foL7cnJQEODuFxhYrVaYTQaUVRUhMzMTNFxKIr8/fTfkVuVi6bVTZiRMUN0HPITPz2nEAaDAQaDAT6fT3QUoku+tJenU69H1X33XfpeVZWAQGK43TxLh4IrJzUHrQ+0YnzKeNFRKACcaVIYzjSR4tx4I9DSglPjx+P3d9+Nwr17oc/PBx5/XHSysNHr9YiP5/3BiGIdiyaFYdFEitPaChQV4dzRozg8bRquy8rCqN/9DuD7k2jITjlOYevBrVg3dx2yRmeJjkN+4vIcEQ3smmuA//1fjDKZMEujAdL5EWmi4eryXtgIvmbmGtFRKACcaVIYzjSRUrndbpjNZmRnZ0Oj0YiOQ0QUdjxygIj8YrPZUFNTA5vNJjoKEZEQnGlSGM40kVL5fD64XC4kJSVBpVKJjkPD5Opx4VjnMdExYlartRUPGh/E60WvY1bmLNFxyE/c00REflGpVEhOThYdg4LkWOcx5Fblio4R82xuztxGEhZNROQXu92OxsZGLFiwAGPGjBEdh4bJLbmhHqFGdWE1puiniI4Tszj2kYVFExH5RZIkdHR0QJIk0VEoCK7RXYNtt2xDwaQCpI5KFR2HKCJwT5PCcE8TERGRMvHTc0REMcjhdmD/if1wuB2ioxBFDBZNCmEwGJCTk4O8vDzRUYiuyGKxoKKiAhaLRXQUCoKTtpO4teZWnLSdFB2FKGJweU5huDxHStXd3Y2WlhZMnz4dWq1WdBwaph5fDzpdndAn6RGv4n31iPzBoklhWDQREREpE5fniMgvHo8HJpMJHo9HdBQKgnZ7O1btW4V2e7voKEQRg0UTEfnFarWiuroaVqtVdBQKArfkxuGOw3BLbtFRiCIGl+cUhstzpFSSJMHpdCIlJQVqNY94I6LYw998ROQXtVoNnU4nOgYRkTBcniMivzgcDhiNRjgcPNcnGrRYWqCr0KHF0iI6ClHEYNHkh8rKSkycOBEajQa5ubk4cOCAX9e9/PLLiIuLQ2FhYYgTUsSy24Hdu4E9ewC3sveWeL1emEwmeL1e0VEoCNK16SjLL0O6Nl10FKKIwT1Ng6itrcXy5ctRWVmJefPm4YUXXsCLL76II0eOIDs7u9/r2tvbMW/ePEyaNAk6nQ579+716/W4pymGvPkm8MMfAt3dFx5fdRVQXw/k5IjNRUREV8SiaRCzZ8/GjBkzsH379t62qVOnorCwEOXl5Ve8xufzYeHChfjpT3+KAwcOwG63s2iivnw+4OtfBz77rG/7d74DvPWWkEgUW7o8XWg63YTcjFwkJyaLjkMUEbgRfABerxdNTU0oLS3t07548WIcPHiw3+u2bNmC1NRU3HPPPYMu5Xk8nj7n3jidzuGFpshw/HhvwdQTH49Ovf5C+5EjwOefAyOUt3JutVphNBpRVFSEzMxM0XFomFqtrVhUvQhNq5swI2OG6DhEEYFF0wA6Ozvh8/mQlpbWpz0tLa3f+2+9//772LlzJ5qbm/16jfLycmzevHnYWSnCpKUB8fFATw/as7Kwa8WKS9978UVxufzgVvjeK/JPTmoOWh9oxfiU8aKjEEUMFk1+iIuL6/NYluXL2gCgq6sLd911F3bs2AH9xZmDQZSVlaGkpKT3sdPpRFZW1vACk/J97WvAvfcClZXQd3Zi7oEDmNzWhpEPPQTcfrvodAPy971NyqZRa3C17mrRMYgiCoumAej1eqhUqstmlc6cOXPZ7BMAnDx5EiaTCUuXLu1tO3/+PIALZ9wcP34ckydP7nNNYmIiEhMTQ5CeFO8//gO49lqMeeUVFABAeTmwbJnoVBQjTjlOYevBrVg3dx2yRvMfakT+UN7GCQVJSEhAbm4u6uvr+7TX19dj7ty5lz1/ypQp+Pjjj9Hc3Nz7ddttt2HRokVobm7mDBL1pVIBDz4I91tv4URlJdw/+IHoRBRDurxdaDA1oMvbJToKUcTgTNMgSkpKsHz5csycORNz5sxBVVUVzGYz1qxZAwBYsWIFMjMzUV5eDo1Gg2nTpvW5fsyYMQBwWTvRRTabDTU1NVi9ejUyMjJEx6EYkZOag4/WfiQ6BlFEYdE0iGXLluHs2bPYsmULTp8+jWnTpqGurg4TJkwAAJjNZoxQ4CedKHKMGzcOJSUlSEpKEh2FiIgGwHOaFIbnNBFRIFw9LhzrPBbwda3WVjxofBCvF72OWZmzQpCMKPpwpolIMLvdjsbGRixYsKB3OZfIXwfaD+C7u7475OttblsQ0xBFNxZNRIJJkoSOjg5IkiQ6CkWgKfopeHTuoyiYXICxI8cO6Xoi8g+X5xSGy3NERETKxB3MREQRzOF2YP+J/XC4HaKjEEU9Fk1EglksFlRUVPR7ax6igZy0ncStNbfipO2k6ChEUY97mhTCYDDAYDDA5/OJjkJhptVqkZ+fD61WKzoKRaDrx12PL0q+gD6Jt7chCjXuaVIY7mkiIiJSJi7PEQnm8XhgMpng8XhER6EI1G5vx6p9q9BubxcdhSjqsWgiEsxqtaK6uhpWq1V0FIpAbsmNwx2H4ZbcoqMQRT0uzykMl+dijyRJcDqdSElJgVrNbYZERErF39BEgqnVauh0OtExiIhoEFyeIxLM4XDAaDTC4eA5OxS4FksLdBU6tFhaREchinosmogE83q9MJlM8Hq9oqNQBErXpqMsvwzp2nTRUYiiHvc0KUxU7mlqbweefRb4v/8D8vKAkhJg3DjRqYiIiALCoklhoq5oOn0auPFG4MyZS22TJwOHDgHJyeJyEUWJLk8Xmk43ITcjF8mJ/DNFFErcCE6h9fzzwJkz6ImPR6f+nycWu1xAdTXwwx+KzaYQVqsVRqMRRUVFyMzMFB2HIkyrtRWLqhehaXUTZmTMEB2HKKqxaFKIqL2NyqefAgA69XpU3XffpfazZ4GqKkGhlMnt5jk7FLic1By0PtCK8SnjRUchinpcnlOYqFuee/FF4N57cWr8ePz+7rtRuHcv9GfPAr/5DZCfLzqdouj1esTHx4uOQURE/WDRpDBRVzR5vcCtt+Lc++/j8HXX4brDhzHqBz8Adu0C4uJEpyOKeKccp7D14Fasm7sOWaOzRMchimo8coBCKyEBeOMNjPrTnzDrJz/BqDffBHbvZsFEFCRd3i40mBrQ5e0SHYUo6nGmSWGibqbpn9xuN8xmM7Kzs6HRaETHISIiChhnmigsbDYbampqYLPZREchIiIaEs40KUy0zjT5fD64XC4kJSVBpVKJjkOkGK4eF451Hhvy9a3WVjxofBCvF72OWZmzgpiMiL6KRw5QWKhUKiTzMEuiyxzrPIbcqtxh92NzcxaXKNRYNFFY2O12NDY2YsGCBRgzZozoOESK4ZbcUI9Qo7qwGlP0U4bcz3CuJSL/sGiisJAkCR0dHZAkSXQUIkW5RncNtt2yDQWTCpA6KlV0HCIaAPc0KUy07mkiIiKKdPz0HBGRQA63A/tP7IfD7RAdhYgGwaJJIQwGA3JycpCXlyc6SkhYLBZUVFTAYrGIjkKkKCdtJ3Frza04aTspOgoRDYLLcwoTrctz3d3daGlpwfTp06HVakXHIVKMHl8POl2d0CfpEa/ivQeJlIxFk8JEa9FEREQU6bg8R2Hh8XhgMpng8XhERyFSlHZ7O1btW4V2e7voKEQ0CBZNFBZWqxXV1dWwWq2ioxApilty43DHYbglt+goRDQILs8pTLQuz0mSBKfTiZSUFKjVPB6MiIgiD//2orBQq9XQ6XSiYxAREQ0Zl+coLBwOB4xGIxwOnkVD9GUtlhboKnRosbSIjkJEg2DR5IfKykpMnDgRGo0Gubm5OHDgQL/P3bFjB+bPn4+xY8di7NixuPnmm/HBBx+EMe0wuN1AiM5R8nq9MJlM8Hq9IemfKFKla9NRll+GdG266ChENAgWTYOora3Fww8/jPXr1+PQoUOYP38+lixZArPZfMXnNzQ0oKioCO+++y7+9re/ITs7G4sXL8bnn38e5uQB2rIFSEsDMjKA664D/vrXoHafmpqKtWvXIjWV99Yi+rI0bRrWzVuHNG2a6ChENAhuBB/E7NmzMWPGDGzfvr23berUqSgsLER5efmg1/t8PowdOxbPPfccVqxYMejzhWwEf+klYOXKvm2jRwPt7Rf+S0Qh0+XpQtPpJuRm5CI5MVl0HCIaADeCD8Dr9aKpqQmlpaV92hcvXoyDBw/61YfL5UJPT0+/m6A9Hk+fs4ucTufQAw/VH/4AAOiJj0enXn+p/dVXge9/PygvYbVaYTQaUVRUhMzMzKD0SRQNWq2tWFS9CE2rmzAjY4boOEQ0ABZNA+js7ITP50NaWt9p87S0NL/voVZaWorMzEzcfPPNV/x+eXk5Nm/ePOyswxIXBwDo1OtRdd99l9o//xyoqgrqS7ndPIuG6MtyUnPQ+kArxqeMFx2FiAbBoskPcf8sKi6SZfmytit5+umnUVNTg4aGBmg0mis+p6ysDCUlJb2PnU4nsrKyhhc4UCtWAPX1kFQqjJAkFO7dC73HA+zfDwT5PnH6L89kERE0ag2u1l0tOgYR+YFF0wD0ej1UKtVls0pnzpy5bPbpq5555hk89dRTeOutt3DDDTf0+7zExEQkJiYGJe+Q3XUXYDZD99xzuOWNNzBJo8GoqirgmmvE5iKKAaccp7D14Fasm7sOWaPD/A8mIgoIPz03gISEBOTm5qK+vr5Pe32Xfh+PAAAQgklEQVR9PebOndvvdVu3bsUTTzyBv/zlL5g5c2aoYwbHL3+JUSYTZtXVYdSHHwJz5ohORBQTurxdaDA1oMvbJToKEQ2CM02DKCkpwfLlyzFz5kzMmTMHVVVVMJvNWLNmDQBgxYoVyMzM7P0k3dNPP40NGzZg9+7d+PrXv947S6XVaqEN8lJXsLnPn4fZakW2VtvvciIRBVdOag4+WvuR6BhE5AfONA1i2bJl2LZtG7Zs2YIbb7wRjY2NqKurw4QJEwAAZrMZp0+f7n1+ZWUlvF4vfvSjHyEjI6P365lnnhH1I/jNZrOhpqYGNptNdBQiIiLF4TlNCiPyhr0+nw8ulwtJSUlQqVRhfW0iJXL1uHCs81hIX6PV2ooHjQ/i9aLXMStzVkhfi4iGh8tz1EulUiE5mYfrEV10rPMYcqtyw/JaNjdneImUjkUT9bLb7WhsbMSCBQswZswY0XGIhHNLbqhHqFFdWI0p+ikhfa1Q909Ew8eiiXpJkoSOjg5IkiQ6CpEiXKO7Bttu2YaCSQVIHcX7JhLFOu5pUhiRe5qIiIiof/z0HBFRPxxuB/af2A+H2yE6ChEpAIsmhTAYDMjJyUFeXp6wDBaLBRUVFX7fV48o2p20ncStNbfipO2k6ChEpABcnlMYkctz3d3daGlpwfTp0xV/ECdROPT4etDp6oQ+SY94VbzoOEQkGIsmheGeJiIiImXi8hz18ng8MJlM8Hg8oqMQKUK7vR2r9q1Cu71ddBQiUgAWTdTLarWiuroaVqtVdBQiRXBLbhzuOAy35BYdhYgUgMtzCiNyeU6SJDidTqSkpECt5hFeREREX8a/GamXWq2GTqcTHYOIiEiRuDxHvRwOB4xGIxwOnklDBAAtlhboKnRosbSIjkJECsCiKRK53UBNDfDss8CRI0Hr1uv1wmQywev1Bq1PokiWrk1HWX4Z0rXpoqMQkQJwT5PCDLqn6fPPgQULgLa2S23l5UBpafhCEhERxSDONEWazZv7FkwA8NhjF4opIgqqLk8XGkwN6PJ0iY5CRArAjeCR5sABAEBPfDw69fpL7W+/DRQUDKtrq9UKo9GIoqIiZGZmDqsvomjQam3FoupFaFrdhBkZM0THISLBWDQphMFggMFggM/nG/iJkycDx46hPSsLu1asuNT+6adAVVVQsrjdPJOGCAByUnPQ+kArxqeMFx2FiBSAe5oUZtA9TQcPAosWwa7R4MO8PExua8PIvDxg27agZdDr9YiP5322iIiIvoxFk8L4dbjloUPAb38LnD4NLFkCrFkDJCSENyhRDDjlOIWtB7di3dx1yBqdJToOEQnG5blI9M1vwl1ZCbPZjOzsbGhYMBGFRJf3wkbwNTPXiI5CRArAT89FKJvNhpqaGthsNtFRiKJWTmoOPlr7EXJSc0RHISIF4PKcwvh77zmfzweXy4WkpCSoVKowJiQiIopNXJ6LUCqVCsnJyaJjUAxx9bhwrPOY6Bhh1WptxYPGB/F60euYlTlLdBwiEoxFU4Sy2+1obGzEggULMGbMGNFxKAYcaD+A7+76rugYQtjcXAYnIhZNEUuSJHR0dECSJNFRKEakjUrD9eOux6abNuHrY74uOk5YTdFPER2BiBSAe5oUxt89TURERBRe/PQcERERkR9YNCmEwWBATk4O8vLy/Hq+xWJBRUUFLBZLiJMRXdBiaYGuQocWS4voKEREQnB5TmH8XZ7r7u5GS0sLpk+fDq1WG8aEFKv+0f0PvNTyElZMX4E0bZroOEREYceiSWG4p4mIiEiZuDwXoTweD0wmEzwej+goFCO6PBduKdLl6RIdhYhICBZNEcpqtaK6uhpWq1V0FIoRrdZWLKpehFZrq+goRERCcHlOYfxdnpMkCU6nEykpKVCredwWhZ5bcuMz52cYnzIeGrVGdBwiorDj37YRSq1WQ6fTiY5BMUSj1uBq3dWiYxARCcPluQjlcDhgNBrhcDhER6EYccpxCg8aH8QpxynRUYiIhGDRFKG8Xi9MJhO8Xq/oKBQjurz/3Aju5UZwIopN3NOkMDxygIiISJk400RERETkh5BtBJdlGV1dg0/jO1wOfPjph6GKoXi+Hh96pJ7ex522TgBA+R/LkZiU2O91jm4HWk624Cc3/QTfyP5GULK4JTfaHe2YMHpC0D8dFcq+I12kjM1J20n8W/2/ofZHtZh51UzRcYiIBpWcnIy4uLig9Rey5bmOjg6MGzcuFF0TERERDaq0tBSbNm1CYmL/kxCBCFnR5HA4MGbMGJw6dWrAvTmBzDQ9vuFxbHliS1DyBbOv4fT31ZmmLyxfYN2d6/BI1SNIGdP/uHl9Xpx1n8XN192MtNED3wds+fLl+MMf/jBoFn9mPPzty9++h9rfV507dw7fW/I91BnrMGrUqGH3BwQv22B9DWWmSeS4feNr30BSfFK/38/Ly8OHHwZn9jiYfQWzP6fTiaysrEF/v4U7Vyj6C2ZfHLeh4bgNzcVxC+Ye4ZAtz12cDktJSRkwbEpKCrLSs/zq87EvHsO/zvnXoOQLZl/B7O+zzz7DOqxDyZISjB8/PgjJgFFdo7DgGwsU11cw+3M6ncA/gHmT5wXtDwfHbWhUKpUi+wpFf4P9fvOXkn/OYGcDOG5DxXETL6I2ghcXFyuyr1D0F0wct6HhuA0Nxy1wSv45lTpmAMdtqDhuQxey5Tl+dH5oPvvss95p2GDNNMUCvt+GhuM2NBy3oeG4DQ3HbWhCMW6qTZs2bQpKT1fqXKXCTTfdxHujBcDj8WDr1q0oKysL2t6cWMH329Bw3IaG4zY0HLeh4bgNTbDHjYdbKszFoxqC/TFJIiIiGh4WTURERER+iKiN4ERERESisGgiIiIi8gOLJiIiIiI/DLtokmUZmzZtwlVXXYWRI0fipptuwuHDhwe8pry8HHl5eUhOTsa4ceNQWFiI48ePDzcKRbnKykpMnDgRGo0Gubm5OHDgQL/P3bFjB+bPn4+xY8di7NixuPnmm/HBBx+EMa1yBDJuX/byyy8jLi4OhYWFIU6oTIGOm91uR3FxMTIyMqDRaDB16lTU1dWFKa1yBDpu27Ztw7XXXouRI0ciKysLjzzyCNxud5jSKltjYyOWLl2Kq666CnFxcdi7d6/oSIoR6Njs2bMHBQUFSE1NRUpKCubMmYM33ngj4NcddtH09NNP49lnn8Vzzz2HDz/8EOnp6SgoKBjwZr3vvfceiouL8d///d+or6+HJElYvHgxzp07N9w4FKVqa2vx8MMPY/369Th06BDmz5+PJUuWwGw2X/H5DQ0NKCoqwrvvvou//e1vyM7OxuLFi/H555+HOblYgY7bRe3t7fjFL36B+fPnhympsgQ6bl6vFwUFBTCZTHjttddw/Phx7NixA5mZmWFOLlag47Zr1y6UlpZi48aNOHr0KHbu3Ina2lqUlZWFObkynTt3DtOnT8dzzz0nOoriBDo2jY2NKCgoQF1dHZqamrBo0SIsXboUhw4dCuyF5WE4f/68nJ6eLv/qV7/qbXO73fLo0aPl559/3u9+zpw5IwOQ33vvveHEoSg2a9Ysec2aNX3apkyZIpeWlvp1vSRJcnJyslxdXR2KeIo1lHGTJEmeN2+e/OKLL8orV66Uf/CDH4Q6puIEOm7bt2+XJ02aJHu93nDEU6xAx624uFj+9re/3aetpKREzs/PD1nGSAVA/uMf/yg6hiINdWxycnLkzZs3B3TNsGaaPv30U1gsFixevLi3LTExEQsXLsTBgwf97sfhcAAAdDrdcOJQlPJ6vWhqaurzPgOAxYsX+/0+c7lc6Onpian32FDHbcuWLUhNTcU999wT6oiKNJRx27dvH+bMmYPi4mKkpaVh2rRpeOqpp+Dz+cIRWRGGMm75+floamrqXTpva2tDXV0dvv/974c8L8W28+fPo6urK+C/E4Z1RKbFYgEApKWl9WlPS0tDe3u7X33IsoySkhLk5+dj2rRpw4lDUaqzsxM+n++K77OL78HBlJaWIjMzEzfffHMoIirSUMbt/fffx86dO9Hc3ByOiIo0lHFra2vDO++8gzvvvBN1dXVobW1FcXExJEnC448/Ho7Ywg1l3O644w50dHQgPz8fsixDkiSsXbsWpaWl4YhMMezXv/41zp07h9tvvz2g6wKaadq1axe0Wm3vV09PDwBcdnK1LMt+n2Z9//3346OPPkJNTU0gUSgGDfV99vTTT6OmpgZ79uyBRqMJVTzF8nfcurq6cNddd2HHjh3Q6/XhiqdYgbzfzp8/j3HjxqGqqgq5ubm44447sH79emzfvj0cURUlkHFraGjAk08+icrKSvz973/Hnj178Oc//xlPPPFEOKJSjKqpqcGmTZtQW1uLcePGBXRtQDNNt912G2bPnt372OPxALgw45SRkdHbfubMmcv+tXElDzzwAPbt24fGxkbenJb6pdfroVKpLvvXqj/vs2eeeQZPPfUU3nrrLdxwww2hjKk4gY7byZMnYTKZsHTp0t628+fPAwDUajWOHz+OyZMnhza0Agzl/ZaRkYH4+HioVKretqlTp8JiscDr9SIhISGkmZVgKOO2YcMGLF++HKtWrQIAXH/99Th37hxWr16N9evXY8QInopDwVVbW4t77rkHr7766pBWHgJ6RyYnJ+Pqq6/u/crJyUF6ejrq6+t7n+P1evHee+9h7ty5/fYjyzLuv/9+7NmzB++88w4mTpwYcHCKHQkJCcjNze3zPgOA+vr6Ad9nW7duxRNPPIG//OUvmDlzZqhjKk6g4zZlyhR8/PHHaG5u7v267bbbsGjRIjQ3NyMrKytc0YUayvtt3rx5+OSTT3qLTAA4ceIEMjIyYqJgAoY2bi6X67LCSKVSQZZlyLzDFwVZTU0N7r77buzevXvo++YC3m7+Fb/61a/k0aNHy3v27JE//vhjuaioSM7IyJCdTmfvc7797W/Lv/3tb3sfr127Vh49erTc0NAgnz59uvfL5XINNw5FqZdfflmOj4+Xd+7cKR85ckR++OGH5VGjRskmk0mWZVlevnx5n0/oVFRUyAkJCfJrr73W5z3W1dUl6kcQItBx+6pY/fRcoONmNptlrVYr33///fLx48flP//5z/K4cePkf//3fxf1IwgR6Lht3LhRTk5OlmtqauS2tjb5zTfflCdPnizffvvton4ERenq6pIPHTokHzp0SAYgP/vss/KhQ4fk9vZ20dGEG2xsSktL5eXLl/c+f/fu3bJarZYNBkOfvxPsdntArzvsoun8+fPyxo0b5fT0dDkxMVFesGCB/PHHH/d5zoQJE+SNGzdeelHgil//+Z//Odw4FMUMBoM8YcIEOSEhQZ4xY0afIyoWLlwor1y5svfxhAkTrvge+/L7MFYEMm5fFatFkywHPm4HDx6UZ8+eLScmJsqTJk2Sn3zySVmSpDCnFi+Qcevp6ZE3bdokT548WdZoNHJWVpb8s5/9TLbZbAKSK8+77757xd9jA/2ZjRWDjc3KlSvlhQsX9j5/4cKFQRnLOFnmHCgRERHRYLjLjoiIiMgPLJqIiIiI/MCiiYiIiMgPLJqIiIiI/MCiiYiIiMgPLJqIiIiI/MCiiYiIiMgPLJqIiIiI/MCiiYiIiMgPLJqIiIiI/MCiiYiIiMgPLJqIiIiI/PD/PQP0hjCuXVUAAAAASUVORK5CYII=\n",
"text/plain": [
"Graphics object consisting of 56 graphics primitives"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"pointEstimate = ecdfPointsPlot(ecdfPointsUniform)\n",
"lowerBound = ecdfPointsPlot(ecdfPointsUniformLower,colour='green', lines_only=true)\n",
"show(pointEstimate + lowerBound, figsize=[6,3])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(end of YouTry)\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we are doing lots of collections of EDF points we may as well define a function to do it, rather than repeating the same code again and again. We use an offset parameter to give us the flexibility to use this to make points for confidence bands as well."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"def makeEDFPoints(myDataList, offset=0):\n",
" '''Make a list empirical distribution plotting points from from a data list.\n",
" \n",
" Param myDataList, list of data to make ecdf from.\n",
" Param offset is an offset to adjust the edf by, used for doing confidence bands.\n",
" Return list of tuples comprising (data value, cumulative relative frequency(with offset)) \n",
" ordered by data value.'''\n",
" \n",
" sortedUniqueValues = sorted(list(set(myDataList)))\n",
" freqs = [myDataList.count(i) for i in sortedUniqueValues]\n",
" from pylab import cumsum\n",
" cumFreqs = list(cumsum(freqs)) \n",
" cumRelFreqs = [ZZ(i)/len(myDataList) for i in cumFreqs] # get cumulative relative frequencies as rationals\n",
" if offset > 0: # an upper band\n",
" cumRelFreqs = [min(i+offset ,1) for i in cumRelFreqs]\n",
" if offset < 0: # a lower band\n",
" cumRelFreqs = [max(i+offset, 0) for i in cumRelFreqs] \n",
" return zip(sortedUniqueValues, cumRelFreqs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## NZ EartQuakes\n",
"\n",
"Now we will try looking at the Earthquakes data we have used before to get a confidence band around an EDF for that. We start by bringing in the data and the function we wrote earlier to parse that data.\n",
"\n",
"First check if you have already `unzip`-ped `data/earthquakes.csv.zip` file by dropping in shell via `%%sh`."
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"data/earthquakes.csv.zip\n",
"data/earthquakes.tgz\n",
"data/earthquakes_small.csv\n",
"earthquakes.csv\n"
]
}
],
"source": [
"%%sh\n",
"ls data/earthquakes* # the '*' just lists all files that start with earthquakes in data/ directory\n",
"cd data\n",
"tar zxvf earthquakes.tgz"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"-rw-r--r-- 1 sage sage 4085555 Jan 23 14:06 earthquakes.csv\n",
"-rw-r--r-- 1 sage sage 1344114 Jun 9 17:53 earthquakes.csv.zip\n",
"-rw-r--r-- 1 sage sage 1344959 Jun 9 17:53 earthquakes.tgz\n",
"-rw-r--r-- 1 sage sage 77786 Jun 9 17:53 earthquakes_small.csv\n"
]
}
],
"source": [
"%%sh\n",
"# only do this once! So, you don't need to do this step if you see earthquakes.csv file above\n",
"cd data\n",
"# windows and mac users should first try to unzip by uncommenting next line\n",
"# unzip earthquakes.csv.zip\n",
"## if unzip is not found try tar by uncommenting next line and commenting the above line\n",
"#tar zxvf earthquakes.tgz\n",
"ls -al earthquakes*"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"21015"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def getLonLatMagDepTimes(NZEQCsvFileName):\n",
" '''returns longitude, latitude, magnitude, depth and the origin time as unix time\n",
" for each observed earthquake in the csv filr named NZEQCsvFileName'''\n",
" from datetime import datetime\n",
" import time\n",
" from dateutil.parser import parse\n",
" import numpy as np\n",
" \n",
" with open(NZEQCsvFileName) as f:\n",
" reader = f.read() \n",
" dataList = reader.split('\\n')\n",
" \n",
" myDataAccumulatorList =[]\n",
" for data in dataList[1:-1]:\n",
" dataRow = data.split(',')\n",
" myTimeString = dataRow[2] # origintime\n",
" # let's also grab longitude, latitude, magnitude, depth\n",
" myDataString = [dataRow[4],dataRow[5],dataRow[6],dataRow[7]]\n",
" try: \n",
" myTypedTime = time.mktime(parse(myTimeString).timetuple())\n",
" myFloatData = [float(x) for x in myDataString]\n",
" myFloatData.append(myTypedTime) # append the processed timestamp\n",
" myDataAccumulatorList.append(myFloatData)\n",
" except TypeError, e: # error handling for type incompatibilities\n",
" print 'Error: Error is ', e\n",
" #return np.array(myDataAccumulatorList)\n",
" return myDataAccumulatorList\n",
"\n",
"myProcessedList = getLonLatMagDepTimes('data/earthquakes.csv')\n",
"\n",
"def interQuakeTimes(quakeTimes):\n",
" '''Return a list inter-earthquake times in seconds from earthquake origin times\n",
" Date and time elements are expected to be in the 5th column of the array\n",
" Return a list of inter-quake times in seconds. NEEDS sorted quakeTimes Data'''\n",
" import numpy as np\n",
" retList = []\n",
" if len(quakeTimes) > 1:\n",
" retList = [quakeTimes[i]-quakeTimes[i-1] for i in range(1,len(quakeTimes))]\n",
" #return np.array(retList)\n",
" return retList\n",
"\n",
"interQuakesSecs = interQuakeTimes(sorted([x[4] for x in myProcessedList]))\n",
"len(interQuakesSecs)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[531.0, 551.0, 764.0, 294.0, 678.0, 1538.0, 376.0, 364.0, 208.0, 242.0]"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"interQuakesSecs[0:10]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There is a lot of data here, so let's use an interactive plot to do the non-parametric DF estimation just for some of the last data:"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "934cd962189043889c3960087b12be41",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"SW50ZXJhY3RpdmUgZnVuY3Rpb24gPGZ1bmN0aW9uIF8gYXQgMHg3ZjhlNGZlMDQ2Njg+IHdpdGggMiB3aWRnZXRzCiAgdGFrZUxhc3Q6IFNlbGVjdGlvblNsaWRlcihkZXNjcmlwdGlvbj11J3TigKY=\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"@interact\n",
"def _(takeLast=(500,(0..min(len(interQuakesSecs),1999))), alpha=input_box(0.05.n(digits=3),label='$\\\\alpha$')):\n",
" '''Interactive function to plot the edf estimate and confidence bands for inter earthquake times.'''\n",
" if takeLast > 0 and alpha > 0 and alpha < 1:\n",
" lastInterQuakesSecs = interQuakesSecs[len(interQuakesSecs)-takeLast:len(interQuakesSecs)]\n",
" interQuakePoints = makeEDFPoints(lastInterQuakesSecs)\n",
" p=ecdfPointsPlot(interQuakePoints, lines_only=true)\n",
" epQuakes = calcEpsilon(alpha, len(lastInterQuakesSecs))\n",
" interQuakePointsLower = makeEDFPoints(lastInterQuakesSecs, offset=-epQuakes)\n",
" lowerQuakesBound = ecdfPointsPlot(interQuakePointsLower, colour='green', lines_only=true)\n",
" interQuakePointsUpper = makeEDFPoints(lastInterQuakesSecs, offset=epQuakes)\n",
" upperQuakesBound = ecdfPointsPlot(interQuakePointsUpper, colour='green', lines_only=true)\n",
" show(p + lowerQuakesBound + upperQuakesBound, figsize=[6,3])\n",
" else:\n",
" print \"check your input values\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Plug-in Estimator: A nonparametric Point Estimator \n",
"\n",
"A function $h$ of another function $g$ is called a *functional*. A **statistical functional** $T(F)$ is any function of the distribution function $F$.\n",
"\n",
"Suppose we are not directly interested in estimating the unknown $F^*$ in the experiment with $n$ IID samples: \n",
"$$X_1,\\ldots,X_n \\overset{IID}{\\sim}F^* \\in \\{ \\text{ all DFs } \\}$$\n",
"But rather in estimating a statistical functional of interest, say, $\\theta^* := T(F^*)$.\n",
"\n",
"NOTE: Here $\\theta^*$ is not a parameter, but just some unknown quantity of interest that can be obtained as a statistical functional $T(F^*)$ of the unknown $F^*$.\n",
"\n",
"Some examples of statistical functionals you have already seen include:\n",
"\n",
"- The expectation of $g(X)$ is a functional as it merely involves the integration against $dF(x)$\n",
" - Mean with $g(X)=X$: \n",
" $$E(X) = \\int x \\, d F(x) = \\begin{cases} \\int x \\, f(x) dx & \\text{ if $X$ is a continuous distribution with probability density function $f(x)$}\\\\ \\sum_x x \\, f(x) & \\text{ if $X$ is a discrete distribution with probability mass function $f(x)$}\\end{cases}$$\n",
" - Variance with $g(X)=(X-E(X))^2$:\n",
" $$V(X) = \\int (x-E(X))^2 \\, d F(x)$$\n",
"- Another interesting staistical function is the median:\n",
" - Median as the inverse of the DF at $0.5$:\n",
" $$\\mathrm{median}(X) = F^{[-1]}(0.5)$$\n",
"\n",
"By substituting in the point estimate $\\widehat{F}_n$ for the unknown $F^*$, we can obtain the **plug-in estimator** as:\n",
"$$\n",
"\\boxed{\n",
"\\widehat{\\Theta}_n = T(\\widehat{F}_n) \\quad\n",
"\\text{ of the quantity of interest } \\, \\theta^* := T(F^*)\n",
"}\n",
"$$\n",
"In other words, just plug-in $\\widehat{F}_n$ for the unknown $F^*$ in $T(F^*) =: \\theta^*$. \n",
"\n",
"If $T(F)=\\int r(x) \\, d F(x)$ for some function $r(x)$ then $T$ is called a **linear functional** since $T$ is linear in its arguments: $T(aF+bG)=aT(F)+bT(G)$, where $(a,b) \\in \\mathbb{R}^2$.\n",
"\n",
"Thus, the plug-in estimator for any linear statistical functional $\\theta^* = T(F^*)=\\int r(x) dF^*(x)$ is:\n",
"\n",
"$$\n",
"\\boxed{\\widehat{\\Theta}_n = T(\\widehat{F}_n) = \\frac{1}{n} \\sum_{i=1}^n r(x_i)}\n",
"$$\n",
"because:\n",
"$$\n",
"\\widehat{\\Theta}_n = T(\\widehat{F}_n) = \\int r(x) d \\widehat{F}_n(x) = \\sum_{i=1}^n r(x_i) \\, P(X=x_i) = \\sum_{i=1}^n r(x_i) \\, f(x_i) = \\sum_{i=1}^n r(x_i) \\, \\frac{1}{n} = \\frac{1}{n} \\sum_{i=1}^n r(x_i)\n",
"$$\n",
"\n",
"Sample mean and sample variance are actually examples of plug-in estimators of the linear statistical functionals $E(X)$ and $V(X)$\n",
" - Sample Mean: \n",
" $$\\overline{X}_n = \\int x \\, d \\widehat{F}_n(x) = \\frac{1}{n} \\sum_{i=1}^n x_i$$\n",
" - Sample Variance:\n",
" $$\\mathrm{var}(X_n) = \\int (x-\\overline{X}_n)^2 \\, d \\widehat{F}_n(x) = \\frac{1}{n} \\sum_{i=1}^n \\left( x_i - \\overline{X}_n\\right)^2$$\n",
" \n",
"Quantiles generalise the notion of a median and are very useful statistics when we want to summarise the values taken by a random variable. For a given $q \\in (0,1)$, **the $q$-th quantile** of a random variable with DF $F$ is the statistical functional: \n",
"$$T(F)={F}^{[-1]}(q) := \\inf\\{x: F(x) \\geq q\\}$$\n",
"We can use the plug-in estimator:\n",
"$$\\widehat{F}_n^{[-1]}(q) := \\inf\\{x: \\widehat{F}_n(x) \\geq q\\}$$\n",
"to estimate $T(F)={F}^{[-1]}(q)$, and $\\widehat{F}_n^{[-1]}(q)$ is called the **$q$-th sample quantile**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Example: Estimating quantiles of inter-earthquake times\n",
"\n",
"We can simply call the quantile function knowing that it is just a plug-in estimator of the $q$-th quantile.\n",
"\n",
"#### Quantiles and Percentiles\n",
"If $q \\in [0,100]$ then we call $(\\frac{q}{100})$-th quantile as the $q$-th percentile."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"75.255"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"iQMinutes = np.array(interQuakesSecs)/60.0\n",
"np.percentile(iQMinutes,95.0) # percentile\n",
"# is in range (0,100) corresponding to quantile in (0,1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So only $5\\%=0.05$ of the inter-EQ times are longer than 75 minutes. If we assume this non-parametric model that models the times between EQs as IID RVs, then the probability of not have any earthquakes in New Zealand (during the observation period) for longer than 150 minutes is no smaller than $0.05 \\times 0.05 = 0.0025 = 1/400$, a pretty unlikely event!\n",
"\n",
"Quantiles as plug-in estimates can give us a lot of basic insights into the data under the more general nonparametric view."
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"minimum, maximum inter EQ-time in minues = 0.0 327.8\n",
"(0.05-th, 0.50-th or median, 0.95-th) quantiles = (1.3166666666666667, 15.516666666666667, 75.255)\n"
]
}
],
"source": [
"print \"minimum, maximum inter EQ-time in minues = \", min(iQMinutes), max(iQMinutes)\n",
"\n",
"print \"(0.05-th, 0.50-th or median, 0.95-th) quantiles = \",\\\n",
"(np.percentile(iQMinutes,5.0),np.percentile(iQMinutes,50.0), np.percentile(iQMinutes,95.0))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Bootstrap for Confidence Sets in Nonparametric Experiments \n",
"\n",
"> This is an adaptation of *All of Statistics (2003)* by Wasserman of Carnegie Mellon University, Pittsburgh, PA, USA (one of the first universities that started presenting an integrated research and educational view of mathematical statistics and machine learning, along both theoretical and applied fronts).\n",
"\n",
"Plug-in estimators are point estimators. Next we will see a very general way to obtain set estimators of $\\theta^* = T(F^*)$, some statistical functional of interest, when we do not make any parametric assumptions about the underlying unknown distribution function $F^*$ in our IID experiment:\n",
"\n",
"$$X_1,\\ldots,X_n \\overset{IID}{\\sim}F^* \\in \\{ \\text{ all DFs } \\}$$\n",
"\n",
"NOTE: To get confidence sets (or intervals in 1D) we need to obtain estimated standard error $\\widehat{se}_n$ of our point estimator, such as the plug-in estimator, $\\widehat{\\Theta}_n$ of $\\theta^*=T(F^*)$. This is generally not possible. However, the bootstrap due to Efron gives us a helping hand.\n",
"\n",
"The **bootstrap** is a statistical method for estimating standard errors and confidence sets of statistics, such as estimators.\n",
"\n",
"Let $T_n := T_n((X_1,X_2,\\ldots,X_n))$ be a statistic, i.e, any function of the data $X_1,X_2,\\ldots,X_n \\overset{IID}{\\sim} F^*$. \n",
"\n",
"Suppose we want to know its variance $V_{F^*}(T_n)$, which clearly depends on the fixed and possibly unknown DF $F^*$ (hence the subscripting by $F^*$ to emphasise this dependence).\n",
"\n",
"If our statistic $T_n$ is one with an analytically unknown variance, then we can use the bootstrap to estimate it. \n",
"\n",
"The bootstrap idea has the following **two basic steps** with their underlying assumptions:\n",
"\n",
"- $\\mathsf{Step~1}$: *Estimate* $V_{F^*}(T_n)$ with $V_{\\widehat{F}_n}(T_n)$.\n",
" - Here we are assuming that $n$ is large enough so that $\\widehat{F}_n$ is a good approximation of $F^*$.\n",
"- $\\mathsf{Step~2}$: *Approximate* $V_{\\widehat{F}_n}(T_n)$ using simulated data from the \"Bootstrap World.\" (Glivenko-Cantelli Theorem and Dvoretzky-Kiefer-Wolfowitz Inequality)\n",
" - Here we are assuming that the simulated data is available $B$ times for large enough $B$ to approximate $V_{\\widehat{F}_n}(T_n)$ (Law of Large Numbers)\n",
"\n",
"\n",
"For example, if $T_n=\\overline{X}_n$, in $\\mathsf{Step~1}$, $V_{\\widehat{F}_n}(T_n) = s_n^2/n$, where $s_n^2=n^{-1} \\sum_{i=1}^n (x_i-\\overline{x}_n)$ is the sample variance and $\\overline{x}_n$ is the sample mean. In this case, $\\mathsf{Step~1}$ is enough. However, when the statistic $T_n$ is more complicated (e.g. $T_n=\\widetilde{X}_n = F^{[-1]}(0.5)$), the sample median, then we may not be able to find a simple expression for $V_{\\widehat{F}_n}(T_n)$ and may need $\\mathsf{Step~2}$ of the bootstrap.\n",
"\n",
"$$\n",
"\\boxed{\n",
"\\begin{eqnarray}\n",
"\\text{Real World Data come from} & F^* \\quad \\implies X_1,X_2,\\ldots,X_n & \\implies\n",
"T_n((X_1,X_2,\\ldots,X_n))=t_n \\notag \\\\\n",
"\\text{Bootstrap World Data come from} & \\widehat{F}_n \\quad \\implies X^{\\bullet}_1,X^{\\bullet}_2,\\ldots,X^{\\bullet}_n & \\implies\n",
"T_n((X^{\\bullet}_1,X^{\\bullet}_2,\\ldots,X^{\\bullet}_n))=t^{\\bullet}_n \\notag\n",
"\\end{eqnarray}\n",
"}\n",
"$$\n",
"\n",
"Observe that drawing an observation from the empirical DF $\\widehat{F}_n$ is equivalent to drawing one point at random from the original data (think of the indices $[n] := \\{ 1,2,\\ldots,n \\}$ of the original data $X_1,X_2,\\ldots,X_n$ being drawn according to the equi-probable $de~Moivre(1/n,1/n,\\ldots,1/n)$ RV on $[n]$). Thus, to simulate $X^{\\bullet}_1,X^{\\bullet}_2,\\ldots,X^{\\bullet}_n$ from $\\widehat{F}_n$, it is enough to drawn $n$ observations with replacement from the dataset $\\{X_1,X_2,\\ldots,X_n\\}$.\n",
"\n",
"In summary, the algorithm for Bootstrap Variance Estimation is:\n",
"\n",
"\n",
"> - $\\mathsf{Step~1}$: Draw $X^{\\bullet}_1,X^{\\bullet}_2,\\ldots,X^{\\bullet}_n \\sim \\widehat{F}_n$\n",
"> - $\\mathsf{Step~2}$: Compute $t_n^{\\bullet} = T_n((X^{\\bullet}_1,X^{\\bullet}_2,\\ldots,X^{\\bullet}_n))$\n",
"> - $\\mathsf{Step~3}$: Repeat $\\mathsf{Step~1}$ and $\\mathsf{Step~2}$ $B$ times, for some large $B$, say $B>1000$, to get $t_{n,1}^{\\bullet}, t_{n,2}^{\\bullet},\\ldots,t_{n,B}^{\\bullet}$\n",
"> - $\\mathsf{Step~4}$: Several ways of estimating the bootstrap confidence intervals are possible, we use the one based on percentiles:\n",
"> - The $1-\\alpha$ percentile-based bootstrap confidence interval is:\n",
"$$\n",
"C_n=[\\widehat{G^{\\bullet}}_{n}^{-1}(\\alpha/2),\\widehat{G^{\\bullet}}_{n}^{-1}(1-\\alpha/2)] ,\n",
"$$\n",
"> where $\\widehat{G^{\\bullet}}_{n}$ is the empirical DF of the bootstrapped $t_{n,1}^{\\bullet}, t_{n,2}^{\\bullet},\\ldots,t_{n,B}^{\\bullet}$ and $\\widehat{G^{\\bullet}}_{n}^{-1}(q)$ is the $q^{\\text{th}}$ sample quantile of $t_{n,1}^{\\bullet}, t_{n,2}^{\\bullet},\\ldots,t_{n,B}^{\\bullet}$.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Confidence Interval via Bootstrap - Median of inter-EQ Times in Minutes\n",
"\n",
"Use Bootstrap to obtain a 95% confidence interval of the median inter earth-quate time in minutes. Use this interval to test the null hypothesis that the median inter earth-quate time is 10 minutes.\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The sample median = 15.516666666666667\n",
"The inner 95% percentile based Confidence Interval for the Median = \n",
"[ 15.166666666666666 , 15.866666666666667 ]\n"
]
}
],
"source": [
"# Sample Exam Problem x - Solution\n",
"# Be patient! this can take a couple minutes; reduce B to 100 to save time\n",
"import numpy as np\n",
"iQMinutes = np.array(interQuakesSecs)/60.0\n",
"sampleMedian = np.percentile(iQMinutes,50.0) # median \n",
"print \"The sample median = \",sampleMedian\n",
"\n",
"B = 1000 # Number of Bootstrap replications\n",
"n = len(iQMinutes) # sample size of the original dataset\n",
"bootstrappedSampleMedians=[] # list to store the sample medians from each bootstrapped data\n",
"for b in range(B):\n",
" #sample indices at random between 0 and len(iQMinutes)-1 to make the bootstrapped dataset\n",
" randIndices=[randint(0,n-1) for i in range(n)] \n",
" bootstrappedDataset = iQMinutes[randIndices] # resample with replacement from original dataset\n",
" bootstrappedMedian = np.percentile(bootstrappedDataset,50.0)\n",
" bootstrappedSampleMedians.append(bootstrappedMedian)\n",
"\n",
"lower95BootstrapCIForMedian = np.percentile(bootstrappedSampleMedians,2.5)\n",
"upper95BootstrapCIForMedian = np.percentile(bootstrappedSampleMedians,97.5)\n",
"\n",
"print \"The inner 95% percentile based Confidence Interval for the Median = \"\n",
"print \"[ \"+str(lower95BootstrapCIForMedian) + \" , \" + str(upper95BootstrapCIForMedian) +\" ]\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Testing via Bootstrap \n",
"\n",
"We can use bootstrap-based confidence interval in conjunction with Wald test in nonparametric experiments just as we used confidence interval from an asymptotically normal estimator such as the MLE in the parametric experiments.\n",
"\n",
"### The $\\mathsf{size}$ Wald test from bootstrapped $1-\\alpha$ Confidence Interval $C_n$ for a statistic $t$\n",
"\n",
"The $\\mathsf{size}$ $\\alpha$ Wald test rejects:\n",
"\n",
"$$\n",
"\\boxed{\n",
"\\text{ $H_0: t^*=t_0$ versus $H_1: t^* \\neq t_0$ if and only if $t_0 \\notin C_n := C_n=[\\widehat{G^{\\bullet}}_{n}^{-1}(\\alpha/2),\\widehat{G^{\\bullet}}_{n}^{-1}(1-\\alpha/2)]$.\n",
"}}\n",
"$$\n",
"\n",
"$$\\boxed{\\text{Therefore, testing the hypothesis is equivalent to verifying whether the null value $t_0$ is in the bootstrapped confidence interval.}}$$\n",
"\n",
"\n",
"\n",
"### Example: Wald Test that median of inter-EQ times is 10 minutes\n",
"\n",
"We reject the null hypothesis that the median is $10$ minutes, if $10$ is not inside the 95% confidence interval for the median we just obtained via bootstrap. This is just like the Wald test except that we use the bootstrap instead of the asymptotic normality of the maximum likelihood estimator.\n",
"\n",
"Let us see this more explicitly in the next cell."
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The null value for median = 10.0000000000000\n",
"is NOT inside the 95% Bootstrapped Confidence Interval: [ 15.166666666666666 , 15.866666666666667 ]\n",
" Therefore, we reject the null hypothesis H_0: median = 10.0000000000000\n"
]
}
],
"source": [
"### Example: Wald Test that median of inter-EQ times is 10 minutes\n",
"#### Be patient if you see a 'In [*]' to the left top corner of the cell ... your computer is working...\n",
"nullValueForMedian = 10.0\n",
"\n",
"# check if the null value is in the 95% CI or not to fail to reject or reject the null hypothesis\n",
"if nullValueForMedian >= lower95BootstrapCIForMedian and nullValueForMedian <= upper95BootstrapCIForMedian:\n",
" print \"The null value for median = \"+str(nullValueForMedian)+\\\n",
" \"\\nis inside the 95% Bootstrapped Confidence Interval: \"+\\\n",
" \"[ \"+str(lower95BootstrapCIForMedian) + \" , \" + str(upper95BootstrapCIForMedian) +\" ]\"\n",
" print \" Therefore, we fail to reject the null hypothesis H_0: median=1.0\"\n",
"else:\n",
" print \"The null value for median = \"+str(nullValueForMedian)+\\\n",
" \"\\nis NOT inside the 95% Bootstrapped Confidence Interval: \"+\\\n",
" \"[ \"+str(lower95BootstrapCIForMedian) + \" , \" + str(upper95BootstrapCIForMedian) +\" ]\"\n",
" print \" Therefore, we reject the null hypothesis H_0: median = \"+str(nullValueForMedian)\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Making a Generic Bootstrap Function for $1-\\alpha$ Confidence Interval of any Statistic\n",
"\n",
"Making a generic function `makeBootstrappedConfidenceIntervalOfStatisticT` as done below can be helpful in other problems."
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"def makeBootstrappedConfidenceIntervalOfStatisticT(dataset, statT, alpha, B=100):\n",
" '''make a bootstrapped 1-alpha confidence interval for ANY given statistic statT \n",
" from the dataset with B Bootstrap replications for 0 < alpha < 1, and \n",
" return lower CI, upper CI, bootstrapped_samples '''\n",
" n = len(dataset) # sample size of the original dataset\n",
" bootstrappedStatisticTs=[] # list to store the statistic T from each bootstrapped data\n",
" for b in range(B):\n",
" #sample indices at random between 0 and len(iQMinutes)-1 to make the bootstrapped dataset\n",
" randIndices=[randint(0,n-1) for i in range(n)] \n",
" bootstrappedDataset = dataset[randIndices] # resample with replacement from original dataset\n",
" bootstrappedStatisticT = statT(bootstrappedDataset)\n",
" bootstrappedStatisticTs.append(bootstrappedStatisticT)\n",
" # noe get the [2.5%, 97.5%] percentile-based CI\n",
" alpaAsPercentage=alpha*100.0\n",
" lowerBootstrap1MinusAlphaCIForStatisticT = np.percentile(bootstrappedStatisticTs,alpaAsPercentage/2)\n",
" upperBootstrap1MinusAlphaCIForStatisticT = np.percentile(bootstrappedStatisticTs,100-alpaAsPercentage/2)\n",
" return (lowerBootstrap1MinusAlphaCIForStatisticT,upperBootstrap1MinusAlphaCIForStatisticT,\\\n",
" np.array(bootstrappedStatisticTs))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Demonstrating the use of bootstrap for different statistics of interest\n",
"\n",
"We can obtain the bootstrap-based $1-\\alpha$ confidence intervals quite easily for any statistic and any dataset as demonstrated below.\n",
"\n",
"### Bootstrapped $1-\\alpha$ Confidence Interval for Sample Median of inter-EQ Time in Minutes\n",
"\n",
"All we need to do is follow these **three steps to perform a bootstrap**:\n",
"\n",
"1. define a lambda expression for the statistic of interest: `statTMedian = lambda dataset : np.percentile(dataset,50.0)` \n",
"- get the plugin estimate : `statTMedian = lambda dataset : np.percentile(dataset,50.0)`\n",
"- and call the generic function we made:\n",
"> `lowerCIMedian,upperCIMedian,bootValuesMedian = \\`\n",
"> ` makeBootstrappedConfidenceIntervalOfStatisticT(iQMinutes, statTMedian, alpha, B)`"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The Plug-in Point Estimate of the Population Median of inter-EQ Times = 15.516666666666667\n",
"1-alpha Bootstrapped CI for the media of inter-EQ Times = (15.099166666666669, 15.834166666666665)\n",
" where alpha = 0.050 and bootstrap replicates = 100\n"
]
}
],
"source": [
"#### Be patient if you see a 'In [*]' to the left top corner of the cell ... your computer is working...\n",
"# dataset\n",
"iQMinutes = np.array(interQuakesSecs)/60.0\n",
"\n",
"# our statistic T as one example, say the 50-th percentile or median as \n",
"# via anonymous function / lambda expression\n",
"statTMedian = lambda dataset : np.percentile(dataset,50.0)\n",
"\n",
"# now obtain the bootstrapped 1-alpha confidence interval with alpha=0.5, i.e. 95% conf Interval\n",
"alpha=0.05\n",
"B=100 # number of bootstrap samples\n",
"\n",
"# plug-in point estimate of population median of inter-EQ times\n",
"plugInEstimateOfMedian = statTMedian(iQMinutes)\n",
"\n",
"# Let's call our convenient function\n",
"# and get the bootstrapped samples and build 1-alpha confidence interval\n",
"lowerCIMedian,upperCIMedian,bootValuesMedian = \\\n",
" makeBootstrappedConfidenceIntervalOfStatisticT(iQMinutes, statTMedian, alpha, B)\n",
"\n",
"# print the results\n",
"print \"The Plug-in Point Estimate of the Population Median of inter-EQ Times = \", plugInEstimateOfMedian\n",
"print \"1-alpha Bootstrapped CI for the media of inter-EQ Times = \", (lowerCIMedian,upperCIMedian)\n",
"print \" where alpha = \",alpha.n(digits=2),\" and bootstrap replicates = \", B"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAAEICAYAAACQ18pCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xe4ZVV5+PHvCzMUqSITI3UUlQgKKhNFMRE7TU0UFRNRVIKNoP5IFIyV2FJsCQkEFMESwCBRFOyKxO4MYkHUEBhgqEMdmkh5f3+sdZ09Z8659+xbzp4z8/08z33u2WeX86691l77PbudyEwkSZI0vPW6DkCSJGncmEBJkiS1ZAIlSZLUkgmUJElSSyZQkiRJLZlASZIktTT2CVREvDMiPlVf7xARt0XE+rP8GedGxKFTff6A8RdGxN6zGc+4iYg/j4grat08ps/4jIiHdhHbuIqIkyPi3V3HMUhELKz1Om+En7lzRPwkIm6NiCOGnGckbW+qfmKGy35LRHx0LpbdIoY56XtHISJeExHX1vgfMOLP/v2+JSL+MiK+Okefc1tEPGQulr0mGXU5p0ygImJpbVybNN47NCLOndPIpiEzL8/MTTPz3q5jmZCZu2bmuZNN08XOZsT+GTi81s1PRvWhte0+fchpBybJGhtvAs7NzM0y8196R66tdZyZ783MkZard9uay753jpPP+cAHgWfW+G+Yi88ZRmZ+OjOfOUfL3jQzLxlm2rn4UlHr8O6a4Ez83dwYHxHxtxHxvxFxZ0RcHhHvjYgNJlnmattzm3LOhmGPQM0DXj/TD6sraeyPeo2bNSAx2xG4sOMYZmQNWIea2ti3M43cA4GNsN3Miin6ydNrgjPxt2Vj3L8AhwEvBTYD9gWeDpw2d9HOgsyc9A9YChwF3AhsWd87lPJNb2KaJwI/Bm6p/5/YGHcu8B7gu8CdwEPre+8GvgfcBnwBeADwaWBFXcbCxjI+AlxRxy0B/qQx7p3Ap+rrhUAC8+rwIcAlwK3ApcBfNuZ7BXARcBPwFWDHxrhnAL+q5TkW+DZw6ID1807gM8An6udcCCzqWX9Pr68fByyu5bgW+GB9//Ia92317wmU5PatwGXAdXX5WzSW+9I67gbgbT2f807gDOBT9bMOrZ/9feBm4Oparg0ay0vgtcD/1nL8PbBTnWdFLeMGA9ZB31iBDWt5Ergd+L8B8ydwRK2r64F/AtabbNmNeZ9T1/nNlHb1iPr+J4H7KG3uNsrRiY3qOrmhTv9jSgf6HuBe4Ld12mMbcb2urpNLh2yLZwCn13V4PrB7T1s4Gvglpd19HNioMf4A4IIa2/eA3RrjHlOXd2td/mnAuwesz4dS2uwtdX2e3mJb+q+6jm4Ffg48vMZ8XZ3vmT3b9vuAH9XP+jyw1YBtcQvgY5S2dyVl+19/qnj7lG1QfX+zpw4f3jPfZHX86lrHNwH/BsQw/UTP8ifKexhwVS3nkQP6qb2BZX362Yntd2PglPqZF1Ha7rJJ1klz2RNxvIzSr1wP/N0k825IOUJ8OaVPOh7YuI7bGvhiXdc3Av9D2R77bVu99X0us9DHA/sAvwPursv56VTtaUAZP1zr5ar6ekNK276dlX3vNyep15fX+G6q7eWPgZ/VdXNszzzT2rdQ9lffabGtDtzvDOhjH1pfn0xp52fXeX8I7FTHncfK/vo24EVD9E1LgTfX9XHXRBsY1Eb7jHsYZdt8XM/729flPbnPPJNtz81y/jvwpTrNd4E/rPV/U62HxzSWuQ3wWWA5JV84YtD6/P08U05QN2zgTGqHTSOBAraqwRxMOVL14jr8gMaGdDmwax0/v753MWUHvQVlh/Kb+jnzaqP4eCOGl1A2vnnAkcA11B0PAxIoYBNKw9u5jnsQsGt9/Wf18x9Rp30r8L1Gp7ECOLDG+kbgHiZPoH4L7AesT9mh/GBAx/h94OD6elNgz964ezbCi4GH1GnPBD5Zx+1SG8STgA0oHeDdrJpA3V3LuR6lQ94D2LOWdyFlA39DzwZ2FrB5rau7gG/Uz5+oo5cNWAcDY+1t1JNs3N+itKUdals4dIj1MNEBPqPW1ZvqtBv0rvs6/CpKR36/Wld7AJs32umhfeL6Wo1rYqcyVVu8m5Vt528oG+L8Rjy/oHQMW1E26Ilt6rGUJOXxNbaX1ek3rHV8GaUtzq/Lv5vBCdSpwN/Vut8IeFKLbem3wLNYuR1eWpc1H/graiLZWGdXAo+kbG+fZfCXmc8B/1Gn+wNK0vWqqeLtKddU9b1aHfbMP6iOvwhsSWl7y4F9puon+ix7oryn1jI+qi6ruU0Om0C9n7JjvT+wHWXH1DaBOpGy3e9O2ZYfMWDeD1O2+60o3/y/ALyvjnsfJaGaX//+hJpcsvq21Vvf5zIHfXxj+oHtqU8ZjwF+UKdbQEkA/n5Q3zugXo+ntM1nUraRz9XlbUvZbp88030LqydQw2yrffc7A/rYZmJxI+VL9TxKUnvaoP6aSfqmRlu4gNKvbTxVG+0z7tXAZQPGfRt4T8vtuVnO6yn9/EaUL1mXUg4+rE9Jur9Vp12PkqS+ndLfPoTyhf5Zg9ZpZrsE6pGUrHkBqyZQBwM/6pnn+8AhjUIe06fgf9cY/gDwpcbws4ELJonpJuo3eyZPoG4Gnt9bqZSM9JWN4fWAOyinAF7KqglQAMt6K6qnYXy9MbwLcGfv+quvzwPeBWw9YCNtJlDfAF7bGN6ZstOcVyv51Ma4+1G+pTU76/OmqNc3AP/d0/D2agwvAd7cU0cfHrCsgbH22yAHbNz7NIZfC3xjiPXwNuAzPfV4JbB377qvw6+g59vTEBvjU6dYj71t8Qc98VzNym/TS4FXN8bvRz0qBxxH7dQb438NPBn4U8o35+aRke8xOIH6BHACsN0Q23dv/F/r2Q5vY+WRos3qOpk4En0u8P6etv87Sue0kJXb4gMpO/GNG9O+mJWd11DxDlHfq9XhkHXcTDA/AxxVXw/sJ/ose6K8f9R47x+BjzXW7bAJ1CodN6W/bZtAbdcY/yPgoD7zBSUh3anx3hNYebT1GMpRxdW2XYZLoGa9j6/Dk7anPsv6P2C/xvCzgKX94p6kXrdtvHcD9chMHf4s9cvoZG2GKfYt9CRQQ6yTgfudPvP2JhYfbYzbD/hVv2nr8MC+qdEWXjHosxvx/o6yT574+1Yd91YGJH+UI+0nDBh3LlMnUCc2xv01cFFj+FHAzfX144HLe5Z1NI0kv9/f0NcjZeYvKN/UjuoZtQ3l23HTZZTMfMIVfRZ5beP1nX2GN50YiIgjI+KiiLilXni2BSWbnyze24EXUbLbqyPi7Ij4ozp6R+AjEXFzXd6NlMa8bS3PFY3l5ID4m65pvL4D2GjAueBXUr5F/yoifhwRB0yyzN71ehkrd0a9Md5B2aibVok5Ih4eEV+MiGsiYgXwXlZfh0PXSYtYh9WM97K6zKmWvcq4zLyvLqfZ9po+STmkflpEXBUR/1gvIh02rmHaYrNe7qN0kNv0G99Tzh2BIyfaZF329nX8NsCVtS025x3kTZT2/KN6F+grWsTfW+fX58oLg++s/5vtoLc881m9Xe1Y37+6Ubb/oHyDnzTeHm3re1i92+9E+SbrJwYZVL9trLJ9N1/XO7UmLsL90iTLGFSmpgWUL19LGmX8cn0fyqn0i4GvRsQlEdHb909lrvr4qdpTr359SNt6GbYss7ZvGWKdDLvf6WeY9jFhsr5pwjBt9DOZuWXj7yn1/espZ4j6eRDlSO50tam3bXrK+Bam2Ie1vaD7HZTD+M0O5Kr64U07UL4ZTkimKSL+hHJ+9YXA/bNceHYLpVFOKjO/kpnPoFTCryiHtaFU9qt6KnPjzPwe5YjB9o3Pj+bwTGTm/2bmiykb+j8AZ9S7G/utn971ugPlcO+1NcbtGjFuTDnUu8rH9QwfR1kHD8vMzSmNY8p1OKTJYh1Wcx3vUJc51bJXGdeoq4m2t8o6yMy7M/NdmbkL5bq9AyjfClebtjlbY/nDtMVm21mPUk9X9RvfU84rKIeqm23yfpl5KqW+t63la87bP+DMazLzrzJzG8ppy3+PiIfOZFuaRG957qZ0iE1XUI4YbN0o2+aZuetk8fb5rKnqeypt+6HJ+olBBtVv0+2UxAWAKLf+L2iMX2X7bi4zy51aExfh7tumMH1cT9mJ7Noo3xaZuWn9rFsz88jMfAjliNH/i4inTYQyw8/+vSHaZe9nTdqe+ujXh/Srl9kwK/uWOdpWp2uyvmnC7+toGm30m8D2EfG45psRsT3lspNvD5hv1togpYyX9pRxs8zcb7KZWiVQmXkx5QLW5jNWzgEeHhF/ERHzIuJFlMOJX2xZgEE2o+wwlwPzIuLtlOt0JhURD4yI59QE5S7KqYiJb9LHA0dHxK512i0i4gV13NnArhHxvJrNH0G58GzGIuIlEbGgfnOeuIXz3lq2+yjnXSecCrwxIh4cEZtSjhidnpn3UC5UfnZEPLHe5vkupt6wNqOcf7+tHol7zWyUaYhYh/W3EXH/utG8ntLOplr2Z4D9I+Jp9UjSkZS6ntjBXUtjnUbEUyLiUXWHtYKys7+337QDDNMW92i0nTfUeH7QGP+6iNguIraiJLET5TwReHVEPD6KTSJi/4jYjHJK/B7giLqNPY9y/UJfEfGCiJjYAd9E6WjuHTL+tl4SEbtExP0op3zOyJ5b2TPzauCrwAciYvOIWC8idoqIJ08Rb6+p6nsqw9Rx02T9xCBvi4j71Xlezsr6bfoN5WjB/rUcb6Vc6zbhM/Vz7x8R2wKHt4h5aLUfOhH4UET8AUBEbBsRz6qvD6iJd1C2l3tpt70Ma6p2eS2wsH4hmbI99XEq8NaIWBARW1MugZiTxyIwe/uWudhWh9Vbt5P1TTOWmb+hrLdPR8SeEbF+XX+fpWzbXx8yzpn4EbAiIt4cERvXGB4ZEX882UzTeaTAMZTriwDI8tyMAyid2Q2Uw/EHZGbvt9Dp+grlvPJvKIdef8vUp9SglO1IyjeNGynXkry2xvzflCNAp0U5nfULym2T1LhfQLmQ8wbKHQLfnaWy7ANcGBG3Ue6wOCgzf1tPwb0H+G6Uw4d7AidRTjmdR7nw7beUc7hk5oX19WmUbzW3Ui7yu2uSz/4b4C/qtCfSv2OfroGxtvB5ynVXF1A6mo9NtezM/DXlQst/pXybfjbw7Mz8XZ33fZSO8+aI+BtKZ3UGZWdwEeWbzURH+hHgwIi4KSJWe4ZQNUxb/Dzl1PHEjRXPy8y7G+P/k9L5X1L/3l3LsphydPfYOu/FlGsiqOV5Xh2+qS7/zAExQrlD6Ie1nZ0FvD4zLx0y/rY+SbnW4BrKhZqDHmD5UsrFmRN3IJ7BysP2g+JdxRD1PZVh6rj5eQP7iUl8m1J33wD+OTNXezBiZt5C6Ys+Sjl6djvlVO+EY+rwpZSdxxlMvm3PxJtrvD+oZfw65TpDKH3f1ylfPr8P/HuufKZd77Y1E1O1y/+q/2+IiPPr68naU693U+5+/hnlztLz63uzbhb3LXOxrQ7rncAptW5fOFnf1NKLYtXnQN02kbhTviR8lNIf30FZb5cBf1YT/X5abc+TqV/6ng08mrLdXV/j2WKy+SbuqNAYq0dmbqacnlttx6PRiIh3Ui5gfMmA8UspFz0O+kY1VqI8TPdTmdnpU7DXBBGxkJV3XLY58jrMsl9D+bI16AiLtFaJiGModzT+aWbePNX0XfGhlmMqIp5dTxVsQnmMwc8pd0NIGmMR8aCI2KuemtqZciT9v7uOSxqVzHw75c7cPbuOZTI+XXl8PZdy+iQoh6cPSg8nSmuDDSh3lT2YcmT5NMoDAaV1RmYe23UMU/EUniRJUkuewpMkSWrJU3hi6623zoULF3YdxtxasqT832OPbuPQGmNJbRN72CbWLGO0rS5ZsuT6zFww9ZRaG3kKTyxatCgXL17cdRhza+IZlLZ3VVHbhH3gGmaMttWIWJKZi7qOQ93wFJ4kSVJLJlBjJCK2j4hvRfl9pAsj4vV9ptk7ym8nXVD/3t5FrJIkrc28Bmq83AMcmZnn18foL4mIr2XmL3um+5/MnOyHiiVJ0gx4BGqMZObVmXl+fX0r5edIZvpL9JIkqSUTqDFVfzriMcAP+4x+QkT8NCK+NPGjln3mPywiFkfE4uXLl89hpJIkrX1MoMZQ/e27zwJvyMwVPaPPB3bMzN0pP7r6uX7LyMwTMnNRZi5asMC7cCVJasMEasxExHxK8vTpzDyzd3xmrsjM2+rrc4D5EbH1iMOUJGmtZgI1RqI8uOZjwEWZ+cEB0/xhnY6IeByljm8YXZSSJK39vAtvvOwFHAz8PCIuqO+9BdgBIDOPBw4EXhMR9wB34o8MS5I060ygxkhmfgeIKaY5Fljjf8VaGsbCo85eIz9j6fv3n4NIJI0TT+FJkiS1ZAIlSZLUkgmUJElSSyZQkiRJLZlASZIktWQCJUmS1JIJlCRJUksmUJIkSS2ZQEmSJLVkAiVJktSSCZQkSVJLJlCSJEktmUBJkiS1ZAIlSZLUkgmUJElSSyZQkiRJLZlASZIktWQCJUmS1JIJlCRJUksmUJIkSS2ZQEmSJLVkAiVJktSSCZQkSVJLJlCSJEktmUBJkiS1ZAIlSZLUkgmUJElSSyZQkiRJLZlASZIktWQCJUmS1JIJlCRJUksmUJIkSS2ZQEmSJLVkAjVGImL7iPhWRFwUERdGxOv7TBMR8S8RcXFE/CwiHttFrJIkrc3mdR2AWrkHODIzz4+IzYAlEfG1zPxlY5p9gYfVv8cDx9X/kiRplngEaoxk5tWZeX59fStwEbBtz2TPBT6RxQ+ALSPiQSMOVZKktZoJ1JiKiIXAY4Af9ozaFriiMbyM1ZMsIuKwiFgcEYuXL18+V2FKkrRWMoEaQxGxKfBZ4A2ZuaJ3dJ9ZcrU3Mk/IzEWZuWjBggVzEaYkSWstE6gxExHzKcnTpzPzzD6TLAO2bwxvB1w1itgkSVpXmECNkYgI4GPARZn5wQGTnQW8tN6NtydwS2ZePbIgJUlaB3gX3njZCzgY+HlEXFDfewuwA0BmHg+cA+wHXAzcAby8gzglSVqrmUCNkcz8Dv2vcWpOk8DrRhORJEnrJk/hSZIktWQCJUmS1JIJlCRJUksmUJIkSS2ZQEmSJLVkAiVJktSSCZQkSVJLPgdKklpaeNTZXYfQ19L37991CNI6wyNQkiRJLZlASZIktWQCJUmS1JIJlCRJUksmUJIkSS2ZQHUgIj45zHuSJGnNZALVjV2bAxGxPrBHR7FIkqSWTKBGKCKOjohbgd0iYkX9uxW4Dvh8x+FJkqQhmUCNUGa+LzM3A/4pMzevf5tl5gMy8+iu45MkScPxSeQdyMyjI2JbYEcadZCZ53UXlSRJGpYJVAci4v3AQcAvgXvr2wmYQEmSNAZMoLrx58DOmXlX14FIkqT2vAaqG5cA87sOQpIkTY9HoLpxB3BBRHwD+P1RqMw8oruQJEnSsEygunFW/ZMkSWPIBKoDmXlK1zFIkqTpM4HqQERcSrnrbhWZ+ZAOwpEkSS2ZQHVjUeP1RsALgK06ikWSJLXkXXgdyMwbGn9XZuaHgad2HZckSRqOR6A6EBGPbQyuRzkitVlH4UiSpJZMoLrxgcbre4ClwAu7CUWChUed3XUIkjRWTKA6kJlP6ToGSZI0fV4D1YGI2CIiPhgRi+vfByJii67jkiRJwzGB6sZJwK2U03YvBFYAH+80IkmSNDRP4XVjp8x8fmP4XRFxQWfRSJKkVjwC1Y07I+JJEwMRsRdw51QzRcRJEXFdRPxiwPi9I+KWiLig/r19FmOWJEmVR6C68RrglMZ1TzcBhwwx38nAscAnJpnmfzLzgBlFJ0mSJmUC1YHMvADYPSI2r8MrhpzvvIhYOIehSZKkIXgKrwMR8d6I2DIzV2Tmioi4f0S8e5YW/4SI+GlEfCkidp2lZUqSpAYTqG7sm5k3Twxk5k3AfrOw3POBHTNzd+Bfgc8NmjAiDpt4jMLy5ctn4aMlSVp3mEB1Y/2I2HBiICI2BjacZPqh1CNat9XX5wDzI2LrAdOekJmLMnPRggULZvrRkiStU7wGqhufAr4RER8HEngFcMpMFxoRfwhcm5kZEY+jJMg3zHS5kiRpVSZQHcjMf4yInwFPBwL4+8z8ylTzRcSpwN7A1hGxDHgHML8u83jgQOA1EXEP5bEIB2Vmzk0pJElad5lAdSQzvwx8ueU8L55i/LGUxxxIkqQ55DVQkiRJLZlASZIktWQCJUmS1JLXQI1QRPycctddX5m52wjDkSRJ02QCNVoTv1H3uvr/k/X/XwJ3jD4cSZI0HSZQI5SZlwFExF6ZuVdj1FER8V3gmG4ikyRJbXgNVDc2iYgnTQxExBOBTTqMR5IkteARqG68EjgpIragXBN1C+Vp5JIkaQyYQHUgM5cAu0fE5kBk5i1dxyRJkobnKbwORMQDI+JjwOmZeUtE7BIRr+w6LkmSNBwTqG6cDHwF2KYO/wZ4Q2fRSJKkVkygurF1Zn4GuA8gM+8B7u02JEmSNCwTqG7cHhEPoD5UMyL2pFxILkmSxoAXkXfj/wFnATvV5z8tAA7sNiRJkjQsE6gOZOb5EfFkYGcggF9n5t0dhyVJkobkKbwORMQLgI0z80Lgz4DTI+KxHYclSZKGZALVjbdl5q31aeTPAk4Bjus4JkmSNCQTqG5M3HG3P3BcZn4e2KDDeCRJUgsmUN24MiL+A3ghcE5EbIh1IUnS2HCn3Y0XUh6kuU9m3gxsBfxttyFJkqRheRfeCEXE5pm5AtgIOLe+txVwF7C4w9AkSVILJlCj9Z/AAcASykM0ozEugYd0EZQkSWrHBGqEMvOA+v/BXcciSZKmzwRqhKZ61lNmnj+qWCRJ0vSZQI3WByYZl8BTRxWIJEmaPhOoEcrMp3QdgyRJmjkTqI5ExCOBXSh35AGQmZ/oLiJJkjQsE6gORMQ7gL0pCdQ5wL7AdwATKEmSxoAP0uzGgcDTgGsy8+XA7sCG3YYkSZKGZQLVjTsz8z7gnojYHLgOnwElSdLY8BReNxZHxJbAiZSHat4G/KjbkCRJ0rBMoDqQma+tL4+PiC8Dm2fmz7qMSZIkDc8EqiMRsRuwkFoHEfHQzDyz06AkSdJQTKA6EBEnAbsBFwL31bcTMIGSJGkMmEB1Y8/M3KXtTDXxOgC4LjMf2Wd8AB8B9gPuAA7x52EkSZp93oXXje9HROsECjgZ2GeS8fsCD6t/hwHHTeMzJEnSFDwC1Y1TKEnUNcBdQACZmbtNNlNmnhcRCyeZ5LnAJzIzgR9ExJYR8aDMvHqW4pYkSZhAdeUk4GDg56y8Bmo2bAtc0RheVt9bLYGKiMMoR6nYYYcdZjEESZLWfiZQ3bg8M8+ag+VGn/ey34SZeQJwAsCiRYv6TiNJkvozgerGryLiP4EvUE7hATALjzFYBmzfGN4OuGqGy5QkST1MoLqxMSVxembjvdl4jMFZwOERcRrweOAWr3+SJGn2mUCNWESsD/wsMz80jXlPBfYGto6IZcA7gPkAmXk8cA7lEQYXUx5j8PJZCluSJDWYQI1YZt4bEc8BWidQmfniKcYn8LrpxiZJkoZjAtWN70XEscDpwO0Tb/rQS0mSxoMJVDeeWP8f03gvgad2EIskSWrJBKoDmfmUrmOQJEnT50+5dCAitoiID0bE4vr3gYjYouu4JEnScEygunEScCvwwvq3Avh4pxFJkqSheQqvGztl5vMbw++KiAs6i0aSJLXiEahu3BkRT5oYiIi9gDs7jEeSJLXgEahuvBr4RL3uKYAbgUM6jUgjsfCos7sOQZI0C0ygOpCZPwV2j4jN6/CKjkOSJEktmEB1ICI2BJ4PLATmRQQAmXnMJLNJkqQ1hAlUNz4P3AIsofyosCRJGiMmUN3YLjP36ToISZI0Pd6F143vRcSjug5CkiRNj0eguvEk4JCIuJRyCi+AzMzdug1LkiQNwwSqG/t2HYAkSZo+E6gOZOZlXccgSZKmz2ugJEmSWjKBkiRJaskESpIkqSUTKEmSpJZMoCRJkloygZIkSWrJxxhI0lpi4VFndx3CQEvfv3/XIUizyiNQkiRJLZlASZIktWQCJUmS1JIJlCRJUksmUJIkSS2ZQEmSJLVkAiVJktSSCZQkSVJLJlCSJEktmUBJkiS1ZAI1ZiJin4j4dURcHBFH9Rl/SEQsj4gL6t+hXcQpSdLazN/CGyMRsT7wb8AzgGXAjyPirMz8Zc+kp2fm4SMPUJKkdYRHoMbL44CLM/OSzPwdcBrw3I5jkiRpneMRqPGyLXBFY3gZ8Pg+0z0/Iv4U+A3wxsy8oneCiDgMOAxghx12mINQJWmlhUedPdR0S1tOP1NL37//SD5Hax+PQI2X6PNe9gx/AViYmbsBXwdO6begzDwhMxdl5qIFCxbMcpiSJK3dTKDGyzJg+8bwdsBVzQky84bMvKsOngjsMaLYJElaZ5hAjZcfAw+LiAdHxAbAQcBZzQki4kGNwecAF40wPkmS1gleAzVGMvOeiDgc+AqwPnBSZl4YEccAizPzLOCIiHgOcA9wI3BIZwFLkrSWMoEaM5l5DnBOz3tvb7w+Gjh61HFJkrQu8RSeJElSSyZQkiRJLZlASZIktWQCJUmS1JIJlCRJUksmUJIkSS2ZQEmSJLVkAiVJktSSCZQkSVJLJlCSJEktmUBJkiS1ZAIlSZLUkgmUJElSSyZQkiRJLZlASZIktTSv6wA03hYedXbXIQxlaf0/LvFKktZsHoGSJElqyQRKkiSpJRMoSZKklkygJEmSWjKBkiRJaskESpIkqSUTKEmSpJZMoCRJkloygZIkSWrJBEqSJKklEyhJkqSWTKAkSZJaMoGSJElqyQRKkiSpJRMoSZKklkygJEmSWjKBkiRJaskESpIkqSUTqDETEftExK8j4uKIOKrP+A0j4vQ6/ocRsXD0UUqStHYzgRojEbE+8G/AvsAuwIsjYpeeyV4J3JSZDwU+BPzDaKOUJGntZwI1Xh4HXJyZl2Tm74DTgOf2TPNc4JT6+gzgaRERI4xRkqS1XmRm1zFoSBFxILBPZh5ahw8GHp+Zhzem+UWdZlkd/r8+xRBkAAAEDElEQVQ6zfU9yzoMOKwO7gz8egRF2Bq4fsqp1l6W3/Jb/rXLjpm5oOsg1I15XQegVvodSerNgIeZhsw8AThhNoIaVkQszsxFo/zMNYnlt/yWf90tv9Y+nsIbL8uA7RvD2wFXDZomIuYBWwA3jiQ6SZLWESZQ4+XHwMMi4sERsQFwEHBWzzRnAS+rrw8Evpmep5UkaVZ5Cm+MZOY9EXE48BVgfeCkzLwwIo4BFmfmWcDHgE9GxMWUI08HdRfxakZ6ynANZPnXbZZfWot4EbkkSVJLnsKTJElqyQRKkiSpJRMozVhEnBQR19VnUE28986IuDIiLqh/+w2Yd9KfphkH0y1/RGwfEd+KiIsi4sKIeP1oI58dM6n/Ou36EfGTiPjiaCKeXTNs/1tGxBkR8avaDp4wushnxwzL/8ba9n8REadGxEaji1yaGRMozYaTgX36vP+hzHx0/Tund+SQP00zDk5mGuUH7gGOzMxHAHsCr1vHyj/h9cBFcxLZaJzM9Mv/EeDLmflHwO6M53o4melt/9sCRwCLMvORlBtj1qSbXqRJmUBpxjLzPKb3rKlhfppmjTfd8mfm1Zl5fn19K2Xnue0shzfnZlD/RMR2wP7AR2c1qBGabvkjYnPgTyl3zpKZv8vMm2c5vDk3k/qn3Am+cX1m3f1Y/bl20hrLBEpz6fCI+Fk9xH//PuO3Ba5oDC9jDBOISUxV/t+LiIXAY4AfjiKwERmm/B8G3gTcN8K4RmWq8j8EWA58vJ7C/GhEbDLiGOfSpOXPzCuBfwYuB64GbsnMr446SGm6TKA0V44DdgIeTekcP9BnmqF+dmZMDVN+ACJiU+CzwBsyc8VowptzU5Y/Ig4ArsvMJSOObRSGqf95wGOB4zLzMcDtwFheB9jHMPV/f8oR5wcD2wCbRMRLRhmkNBMmUJoTmXltZt6bmfcBJ1JO1/Ua5qdpxtKQ5Sci5lOSp09n5pmjjHEuDVn+vYDnRMRSyunbp0bEp0YY5pxp0f6XZebEUcczKAnV2Buy/E8HLs3M5Zl5N3Am8MRRxinNhAmU5kREPKgx+OfAL/pMNsxP04ylYcofEUG5/uWizPzgqGIbhWHKn5lHZ+Z2mbmQUvffzMy14gjEkOW/BrgiInaubz0N+OUIwptzQ27/lwN7RsT96rbwNMbzInqto/wpF81YRJwK7A1sHRHLgHcAe0fEoymn5JYCr6rTbgN8NDP3G/TTNB0UYUamW37KEZiDgZ9HxAV1cW+Z4o61Nc4Myr9WmGH5/xr4dP0CcQnw8tFGP3Mz2P5/GBFnAOdT7kj9Cf7ci8aIP+UiSZLUkqfwJEmSWjKBkiRJaskESpIkqSUTKEmSpJZMoCRJkloygZIkSWrJBEqSJKml/w8ugSENADpHfQAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Sample Exam Problem 7 - SOLUTION\n",
"#### Be patient if you see a 'In [*]' to the left top corner of the cell ... your computer is working...\n",
"# Do NOT change the variable or function Names - Just replace XXXs\n",
"# now obtain the bootstrapped 1-alpha confidence interval with alpha=0.5, i.e. 95% conf Interval\n",
"alpha=0.05\n",
"B=100 # number of bootstrap samples=1000\n",
"# our statistic T as one example, say the 50-th percentile or median as \n",
"# via anonymous function / lambda expression\n",
"statTMedian = lambda dataset : np.percentile(dataset,50.0)\n",
"\n",
"# plug-in point estimate of population median of inter-EQ times\n",
"plugInEstimateOfMedian = statTMedian(iQMinutes)\n",
"\n",
"# Let's call our convenient function\n",
"# and get the bootstrapped samples and build 1-alpha confidence interval\n",
"lowerCIMedian,upperCIMedian,bootValuesMedian = \\\n",
" makeBootstrappedConfidenceIntervalOfStatisticT(iQMinutes, statTMedian, alpha, B)\n",
"\n",
"# print in more details if you want to see if your values make sense\n",
"print \"The Plug-in Point Estimate of the Population Median of inter-EQ Times = \", plugInEstimateOfMedian\n",
"print \"1-alpha Bootstrapped CI for the media of inter-EQ Times = \", (lowerCIMedian,upperCIMedian)\n",
"print \" where alpha = \",alpha.n(digits=2),\" and bootstrap replicates = \", B\n",
"\n",
"print (plugInEstimateOfMedian,lowerCIMedian,upperCIMedian)\n",
"\n",
"# Hypothesis test\n",
"NullValueForMedian=20 # 20 minutes is the value of Median under the Null Hypothesis, H_0: Population Median = 20\n",
"\n",
"RejectedH0_sampleMedianIs20min = (NullValueForMedian <= upperCIMedian and NullValueForMedian >= lowerCIMedian)\n",
"\n",
"if RejectedH0_sampleMedianIs20min:\n",
" print \" RejectedH0_sampleMedianIs20min = True, so the Null Hypothesis is Rejected at size alpha = \",alpha\n",
"else:\n",
" print \" RejectedH0_sampleMedianIs20min = False, so the Null Hypothesis is NOT Rejected at size alpha = \",alpha\n",
"\n",
"# This is just extra stuff you don't need to do for this sample exam problem!\n",
"import pylab\n",
"pylab.clf() # clear current figure\n",
"Bins=10 # Number of histogram bins\n",
"n, bins, patches = pylab.hist(bootValuesMedian, Bins, density=true) \n",
"pylab.ylabel('normalised count')\n",
"pylab.title('Normalised histogram of bootstrapped samples of the plug-in estimate of median inter-EQ time')\n",
"pylab.axvline(x=plugInEstimateOfMedian, linewidth=2, color='k') # plot vertical line at plug-in point estimate\n",
"pylab.axvline(x=lowerCIMedian, linewidth=2, color='r') # plot vertical line at lower CI at (alpha/2)-th Quantile\n",
"pylab.axvline(x=upperCIMedian, linewidth=2, color='r') # plot vertical line at upper CI at (1-alpha/2)-th Quantile\n",
"#pylab.savefig('myHist') # to actually save the figure for subsequent extraction elsewhere\n",
"pylab.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"lx_assignment_number": "3",
"lx_problem_cell_type": "PROBLEM"
},
"source": [
"---\n",
"## Assignment 3, PROBLEM 7\n",
"Maximum Points = 2"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"lx_assignment_number": "3",
"lx_assignment_type": "ASSIGNMENT",
"lx_assignment_type2print": "Assignment",
"lx_problem_cell_type": "PROBLEM",
"lx_problem_number": "7",
"lx_problem_points": "2"
},
"source": [
"\n",
"Repeat the **three steps to perform a bootstrap** above as done in **Median of inter-EQ Time** example (notebook `11.ipynb`) to find the plug-in estimate and 95% CI for the *99-th Percentile of the inter-EQ time in minutes*.\n",
"\n",
"You just need to evaluate the next `REQUIRED-CELL` and replace `XXX` with the right expressions in the following cell.\n",
"\n",
"HINT: Median is the $50$-th Percentile."
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"deletable": false,
"lx_assignment_number": "3",
"lx_assignment_type": "ASSIGNMENT",
"lx_assignment_type2print": "Assignment",
"lx_problem_cell_type": "PROBLEM",
"lx_problem_number": "7",
"lx_problem_points": "2"
},
"outputs": [],
"source": [
"# REQUIRED-CELL\n",
"\n",
"# DO NOT MODIFY this cell \n",
"# Evaluate this cell before trying this PROBLEM so that the required functions and variables are loaded\n",
"import numpy as np\n",
"## Be Patient! - This will take more time, about a minute or so\n",
"###############################################################################################\n",
"def getLonLatMagDepTimes(NZEQCsvFileName):\n",
" '''returns longitude, latitude, magnitude, depth and the origin time as unix time\n",
" for each observed earthquake in the csv filr named NZEQCsvFileName'''\n",
" from datetime import datetime\n",
" import time\n",
" from dateutil.parser import parse\n",
" import numpy as np\n",
" \n",
" with open(NZEQCsvFileName) as f:\n",
" reader = f.read() \n",
" dataList = reader.split('\\n')\n",
" \n",
" myDataAccumulatorList =[]\n",
" for data in dataList[1:-1]:\n",
" dataRow = data.split(',')\n",
" myTimeString = dataRow[2] # origintime\n",
" # let's also grab longitude, latitude, magnitude, depth\n",
" myDataString = [dataRow[4],dataRow[5],dataRow[6],dataRow[7]]\n",
" try: \n",
" myTypedTime = time.mktime(parse(myTimeString).timetuple())\n",
" myFloatData = [float(x) for x in myDataString]\n",
" myFloatData.append(myTypedTime) # append the processed timestamp\n",
" myDataAccumulatorList.append(myFloatData)\n",
" except TypeError, e: # error handling for type incompatibilities\n",
" print 'Error: Error is ', e\n",
" #return np.array(myDataAccumulatorList)\n",
" return myDataAccumulatorList\n",
"\n",
"myProcessedList = getLonLatMagDepTimes('data/earthquakes.csv')\n",
"\n",
"def interQuakeTimes(quakeTimes):\n",
" '''Return a list inter-earthquake times in seconds from earthquake origin times\n",
" Date and time elements are expected to be in the 5th column of the array\n",
" Return a list of inter-quake times in seconds. NEEDS sorted quakeTimes Data'''\n",
" import numpy as np\n",
" retList = []\n",
" if len(quakeTimes) > 1:\n",
" retList = [quakeTimes[i]-quakeTimes[i-1] for i in range(1,len(quakeTimes))]\n",
" #return np.array(retList)\n",
" return retList\n",
"\n",
"def makeBootstrappedConfidenceIntervalOfStatisticT(dataset, statT, alpha, B=100):\n",
" '''make a bootstrapped 1-alpha confidence interval for ANY given statistic statT \n",
" from the dataset with B Bootstrap replications for 0 < alpha < 1, and \n",
" return lower CI, upper CI, bootstrapped_samples '''\n",
" n = len(dataset) # sample size of the original dataset\n",
" bootstrappedStatisticTs=[] # list to store the statistic T from each bootstrapped data\n",
" for b in range(B):\n",
" #sample indices at random between 0 and len(iQMinutes)-1 to make the bootstrapped dataset\n",
" randIndices=[randint(0,n-1) for i in range(n)] \n",
" bootstrappedDataset = dataset[randIndices] # resample with replacement from original dataset\n",
" bootstrappedStatisticT = statT(bootstrappedDataset)\n",
" bootstrappedStatisticTs.append(bootstrappedStatisticT)\n",
" # noe get the [2.5%, 97.5%] percentile-based CI\n",
" alpaAsPercentage=alpha*100.0\n",
" lowerBootstrap1MinusAlphaCIForStatisticT = np.percentile(bootstrappedStatisticTs,alpaAsPercentage/2)\n",
" upperBootstrap1MinusAlphaCIForStatisticT = np.percentile(bootstrappedStatisticTs,100-alpaAsPercentage/2)\n",
" return (lowerBootstrap1MinusAlphaCIForStatisticT,upperBootstrap1MinusAlphaCIForStatisticT,\\\n",
" np.array(bootstrappedStatisticTs))\n",
"\n",
"interQuakesSecs = interQuakeTimes(sorted([x[4] for x in myProcessedList]))\n",
"iQMinutes = np.array(interQuakesSecs)/60.0\n",
"###############################################################################################"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"lx_assignment_number": "3",
"lx_assignment_type": "ASSIGNMENT",
"lx_assignment_type2print": "Assignment",
"lx_problem_cell_type": "PROBLEM",
"lx_problem_number": "7",
"lx_problem_points": "2"
},
"outputs": [],
"source": [
"\n",
"statT99thPercentile = lambda dataset : XXX #statistic of interest\n",
"alpha=XXX\n",
"B=1000 # number of bootstrap samples, reduce this to 100 while debuging and back to 1000 when done\n",
"# plug-in point estimate of the 99th-Percentile of inter-EQ Times\n",
"plugInEstimateOf99thPercentile = XXX \n",
"# get the bootstrapped samples and build 1-alpha confidence interval\n",
"# do NOT change anything below\n",
"lowerCIT99P,upperCIT99P,bootValuesT99P = \\\n",
" makeBootstrappedConfidenceIntervalOfStatisticT(iQMinutes, statT99thPercentile, alpha, B)\n",
"print \"The Plug-in Point Estimate of the 99th-Percentile of inter-EQ Times = \", plugInEstimateOf99thPercentile\n",
"print \"1-alpha Bootstrapped CI for the 99th-Percentile of inter-EQ Times = \",(lowerCIT99P,upperCIT99P)\n",
"print \" for alpha = \",alpha.n(digits=2),\" and bootstrap replicates = \",B\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Correlation: A Bivariate Nonparametric Bootstrap\n",
"\n",
"Here is a classical data set used by Bradley Efron at Stanford University's Statistics Department (the inventor of bootstrap) to illustrate the method.\n",
"\n",
"The data are LSAT (Law School Admission Test in the USA) scores and GPA (grade point average) of just fifteen individuals. The inference task involved assisting the Admissions Office with their evidence-based investigations on their admissions policies.\n",
"\n",
"Thus, we have bivariate data of the form $(Y_i,Z_i)$, where $Y_i={\\rm LSAT}_i$ and $Z_i={\\rm GPA}_i$. For example, the first individual had an LSAT score of $y_1=576$ and a GPA of $z_1=3.39$ while the fifteenth individual had an LSAT score of $y_{15}=594$ and a GPA of $z_{15}=3.96$. \n",
"\n",
"We supose that the bivariate data has the following IID bivariate model:\n",
"\n",
"$$\\boxed{(Y_1,Z_1),(Y_2,Z_2),\\ldots,(Y_{15},Z_{15}) \\overset{IID}{\\sim} F^* \\in \\{ \\text{all bivariate DFs} \\}}$$\n",
"\n",
"This is a *bivariate nonparametric experiment* and its bivariate data is plotted below."
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAxQAAAEiCAYAAABgP5QIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XlcVXX+x/E3somyCJkLgkvihopLmpFo6rhEudW0mIpajTOa2zg2ozma1a/Csr3IzMpKTVrVZlTUSlRMc0kbtxRHUzNwZ3EBBL6/PxiuIpftAvcCvp6Px33MPfd8z/d8zkcm7ofz/Z6vkzHGCAAAAABsUM3RAQAAAACovCgoAAAAANiMggIAAACAzSgoAAAAANiMggIAAACAzSgoAAAAANiMggIAAACAzSgoAAAAANiMggIAAACAzSgoAAAAANiMggIAAACAzSpVQbFnzx45OztrzJgxjg6lTKSkpMjX11dhYWGODgUAAACwSaUqKKZOnSpnZ2c9+eSThbbbunWrxo0bpzZt2sjX11eurq6qXbu27rjjDv3jH//Qjh07rB7XuHFjOTk55Xl5eHioadOmevTRR7V3795ixdm2bVvLsSkpKQW28/b21sSJE7Vp0yYtX768WH0DAAAAFYmTMcY4Ooji2Lhxo7p3765Ro0ZpwYIFVttcunRJf/rTn7RkyRJJkqurq5o2bSpvb2+dO3dOR44cUVZWliQpPDxcK1euzHN848aNdfToUTVr1kx16tSRJCUlJSk+Pl4ZGRlyd3fXF198oQEDBhQY565du9ShQwfL9gcffKBHH320wPbnz59XQECAGjVqpL1798rJyal4CQEAAAAqgFLfoZg8ebLlr/k1atRQs2bNdOutt6pTp07q1KmTGjZsKCcnJzVu3LhU53n77bclSSNHjrS6/8qVK+rXr5+WLFmi+vXr68MPP9T58+e1f/9+/fjjj4qPj9eZM2f00UcfKTg4WN9//32B55o+fbri4uIUFxenPXv26NixY+rdu7fS09P1yCOP6MKFCwUeu3DhQklSrVq18mwXxNfXVwMGDND+/fsLjQkAAACoiEpdUHz99ddq1aqVVqxYoeTkZMXHx2vHjh3avn27vvnmG1WrVk3e3t6lGtJz+vRpLVu2TP7+/urevbvVNk8//bTi4uLk7++vH3/8UY888ohq1qyZp02tWrU0cuRI/fzzz5o5c2axz1+3bl0tXLhQ7u7uOnv2rNauXWu1XVZWluXuyNtvvy1nZ2etX79ex44dK7T/IUOGSJLef//9YscEAAAAVASlKih27dolFxcXxcXF6e6775arq6tlX3Jysu666y4lJCRo2bJlateunc3nWbp0qTIyMhQeHq5q1fKHnJSUpDfffFOS9OabbyowMLDQ/lxcXPTPf/6zRDHUq1dPzZo1kyTFx8dbbfPtt98qISFB9erV05AhQ9SrVy8ZY7R48eJC++7Xr59cXFy0bNkypaenlyguAAAAwJFKVVB88803eu211+Tn55fn8/T0dA0aNEh79+7VwoUL1bNnz1IFuWHDBknSbbfdZnX/ypUrdeHCBdWrV0+DBw8u1bkKU9R0k08++USS9NBDD8nZ2VnDhg2TVPSwJw8PD7Vt21ZpaWnatm1b2QQLAAAA2EGpCor9+/erf//+eT7Lzs7W0KFDtX79er322mt68MEHSxWgJP3www+SpFtvvbXQ/aGhoXJ2di71+axJTEzUoUOHJElBQUH59l+4cEHLli2TJEshcd9998nDw0P79+8v8MlSuTp37ixJiouLK8uwAQAAgHJVqoJi8eLF+YYgjR8/Xl9//bWmTZumiRMnlio4KeeuwPHjxyVJ9evXt9rmxIkTklTqid8FOXXqlCIiIpSeni5fX1/16dMnX5uvvvpKly5dUlBQkKU48PLyshRcRd2lyL22o0ePlnH0AAAAQPkpVUFxfTHx7LPPau7cuRo1apQiIyNLFViupKQkZWZmSlK+oVW5UlNTJSnfJOxc0dHR+daXcHJy0kcffWS1/QsvvKCwsDCFhYWpTZs2CgwM1LfffitXV1fNnz9fXl5e+Y7JLRiGDh2a5/PcuxVLliyxXIc1udd2+vTpAtsAAAAAFY1LWXU0f/58zZo1S3fffbfmz59faNvbbrtNY8eO1SOPPFJkv2lpaZb3bm5uVtvkfsG/ePGi1f0333yzunbtatnes2ePkpOTCzxnfHy8ZeK1m5ub6tWrp+7du2vKlClq3759vvYnTpzQunXrJOUvKMLDw+Xr66tTp05pzZo1uvvuu62e08PDQ5J0+fLlAuMCAAAAKpoyWSl7+fLlGjt2rG677TZ9/vnncnEpvE6ZMGFCoYvDXevauxIFFQENGjSQJP36669W9//hD3+wrCsRFxenTp06FXrOBQsWyBgjY4zS09N19OhRLVy40GoxIeUM/crOzlbHjh3VokWLPPvc3Nz0wAMPSCp82NO5c+ckSbVr1y40NgAAAKAiKXVBERcXpyFDhqhp06ZasWJFvmFH1u5CREREFPuLs7u7u7y9vSVd/dJ9vdDQUEk5k7NzV8K2p9xC4aeffrI6tOq9996TlFN4paSkWO0j99puvvlm+wQNAAAAlIFSFRR79+7VwIED5evrq9WrV+crEuLi4spkonTunYH9+/db3X/33XfL09NTJ0+e1NKlS0t9vpLYuXOn9uzZIycnJ9WtW7fAl5ubmy5fvqyvvvrKaj/79u2TJHXs2NGe4QMAAAClYnNBcfz4cfXr109ZWVlatWqV1cJhzpw5uvfeey3b7733nmbNmqUZM2aU6FxhYWGSpO3bt1vd7+vrq/Hjx0uSJk2aVOTK1GUp9+5E9+7dlZiYWOBrypQpedpfL3f9iW7dutkncAAAAKAM2FRQnDt3Tv369dOZM2cKXAX7iy++0L59+xQSEiJJ+vHHH+Xv769u3brp3//+d4nO17dvX0mFr9HwzDPPKDQ0VL///ru6dOmiDz/8UBcuXMjT5sqVK/ryyy914MCBEp2/IFlZWVqyZImknGFchRk+fLgkKTY21vIY3FyHDh3SyZMn1bJlyyJX+QYAAAAqkhIXFJcvX1b//v114MABq6tgx8fHa/r06Ro2bFieuxOZmZm6++679dlnnxV7Qnau7t27KygoSLGxsTp58qTVNm5ublq7dq0efPBBJSYm6rHHHpOfn59atWqlLl26KCgoSLVq1dIDDzyg3377TX379i31Ct5r165VYmKiqlevrvvvv7/QtsHBwerQoYOMMVq8eHGefZ999pkk6dFHHy1VPAAAAIC9lbigmDFjhjZv3iwXFxc9+eSTCgoKUlBQkBo2bChPT081b95ckZGRyszMtDzdSJK6du2qy5cv6/PPPy/W42Kv5eTkpNGjRysrK8vy5duamjVr6rPPPtOWLVs0ZswYNWvWTAkJCfrpp5+UlJSktm3basqUKdqxY4dWr16tRo0alfTy88gdvjRgwAD5+PgU2T73LsX1w56WLFkiV1dXjRw5slTxAAAAAPbmZIwx9jrZhx9+qMWLF+u7777Tpk2b8qwNUZSUlBQ1bdpUfn5+2r9/f75F9SqrdevWqVevXnr88ccVFRXl6HAAAACAErHrt/J//etfeuihh3TgwIECHwFbEG9vb82YMUMHDx5UdHR0OUVof88++6w8PT311FNPOToUAAAAoMTKbKXs4rjnnnu0Z88epaena8KECSU+fuzYsUpJSVF2dnY5RGd/KSkp6tGjhyZOnKi6des6OhwAAACgxOw65AkAAABA1VI1JiIAAAAAcAgKCgAAAAA2o6AAAAAAYDMKCgAAAAA2o6AAAAAAYDMKCgAAAAA2o6AAAAAAYDMKCgAAAAA2o6AAAAAAYDMKCgAAAAA2o6AAAAAAYDMKCgAAAAA2q9IFhTFGKSkpMsY4OhQAAACgSqrSBUVqaqp8fHyUmprq6FAAAACAKqlKFxQAHOvECemPf5TuuEP68ENHRwMAAMqDi6MDAFB1PfywtHFjzvstW6QWLaSuXR0bEwAAKFvcoQBQbvbuvfreGGnfPsfFAgAAygcFBYByM2DA1feenlKPHg4LBQAAlJMqOeQpKipKUVFRysrKcnQowA3t/feljh2l33+Xhg6VmjVzdEQAAKCsOZkq/EzVlJQU+fj4KDk5Wd7e3o4OBwAAAKhyGPIEAAAAwGYUFAAAAABsRkEBAAAAwGYUFAAAAABsRkEBAAAAwGYUFAAAAABsViULiqioKAUHB6tz586ODgUAAACo0liHAgAAAIDNquQdCgAAAAD2QUEBAAAAwGYUFAAAAABsRkEBAAAAwGYUFAAAAABsViULCh4bCwAAANgHj40FAAAAYLMqeYcCAAAAgH1QUAAAAACwmV0Kirlz5yokJETe3t7y9vZWaGioVq1aVegxr7/+ulq0aCEPDw8FBgZq8uTJSktLs0e4AAAAAIrJxR4nCQgI0OzZsxUUFCRJ+vjjjzVo0CDt3LlTrVu3ztd+8eLFmjZtmj788EPdcccdOnjwoEaNGiVJeu211+wRMgAAAIBicNikbD8/P82ZM0ePPfZYvn3jx4/X/v379d1331k+mzJlirZu3aqNGzcW+xxMygYAAADKl93nUGRlZSk6OloXL15UaGio1TZhYWHasWOHtm7dKkk6fPiwVq5cqXvuuafQvtPT05WSkpLnBQAAAKD82GXIkyTt3r1boaGhSktLk6enp5YuXarg4GCrbYcMGaLTp08rLCxMxhhlZmZq7NixmjZtWqHniIyM1DPPPFMe4QMAAACwwm5DnjIyMnTs2DElJSXpq6++0vvvv6/169dbLSpiY2M1ZMgQPffcc+rSpYsOHTqkSZMmafTo0Zo5c2aB50hPT1d6erplOyUlRYGBgQx5AgAAAMqJw+ZQ9O7dW02bNtW8efPy7evWrZtuv/12zZkzx/LZokWL9Oc//1kXLlxQtWrFG6nFHAoAAACgfDlsHQpjTJ67Cde6dOlSvqLB2dlZxhhV4YW9AQAAgErHLnMopk+frvDwcAUGBio1NVXR0dGKjY1VTEyMJGnEiBFq0KCBIiMjJUkDBgzQq6++qg4dOliGPM2cOVMDBw6Us7OzPUIGAAAAUAx2KShOnjypiIgIJSQkyMfHRyEhIYqJiVGfPn0kSceOHctzR2LGjBlycnLSjBkzdOLECd18880aMGCAnn/+eXuECwAAAKCYHDaHwh6YQwEAAACUL4fNoQAAAABQ+VXJgiIqKkrBwcHq3Lmzo0MBAAAAqjSGPAEAAACwWZW8QwEAAADAPigoAAAAANiMggIAAACAzSgoAAAAANiMggIAAACAzapkQcFjYwEAAAD74LGxAAAAAGxWJe9QAAAAALAPCgoAAAAANqOgAAAAAGAzCgoAAAAANqOgAAAAAGAzCgoAAAAANquSBQXrUAAAAAD2wToUAAAAAGxWJe9QAAAAALAPCgoAAAAANrNLQTF37lyFhITI29tb3t7eCg0N1apVqwo9JikpSePGjVP9+vVVvXp1tWrVSitXrrRHuAAAAACKycUeJwkICNDs2bMVFBQkSfr44481aNAg7dy5U61bt87XPiMjQ3369FGdOnX05ZdfKiAgQMePH5eXl5c9wgUAAABQTA6blO3n56c5c+bosccey7fv3Xff1Zw5c/TLL7/I1dXV5nMwKRsAAAAoX3afQ5GVlaXo6GhdvHhRoaGhVtt88803Cg0N1bhx41S3bl21adNGL7zwgrKysgrtOz09XSkpKXleAAAAAMqPXYY8SdLu3bsVGhqqtLQ0eXp6aunSpQoODrba9vDhw/r+++81bNgwrVy5UvHx8Ro3bpwyMzP11FNPFXiOyMhIPfPMM+V1CQAAAACuY7chTxkZGTp27JiSkpL01Vdf6f3339f69eutFhXNmzdXWlqajhw5ImdnZ0nSq6++qjlz5ighIaHAc6Snpys9Pd2ynZKSosDAQIY8AQAAAOXEbnco3NzcLJOyO3XqpG3btumNN97QvHnz8rWtX7++XF1dLcWEJLVq1UqJiYnKyMiQm5ub1XO4u7vL3d29fC4AAAAAQD4OW4fCGJPnbsK1unbtqkOHDik7O9vy2cGDB1W/fv0CiwkAAAAA9meXgmL69OnauHGjfv31V+3evVv//Oc/FRsbq2HDhkmSRowYoSeffNLSfuzYsTp79qwmTZqkgwcPasWKFXrhhRc0btw4e4QLAAAAoJjsMuTp5MmTioiIUEJCgnx8fBQSEqKYmBj16dNHknTs2DFVq3a1tgkMDNSaNWs0efJkhYSEqEGDBpo0aZKmTp1qj3ABAAAAFJPD1qGwB9ahAAAAAMqXw+ZQAAAAAKj8qmRBERUVpeDgYHXu3NnRoQAAAABVGkOeAAAAANisSt6hAAAAAGAfFBQAAAAAbEZBAQAAAMBmFBQAAAAAbEZBAQAAAMBmVbKg4LGxAAAAgH3w2FgAAAAANquSdygAAAAA2AcFBQAAAACbUVAAAAAAsBkFBQAAAACbUVAAAAAAsBkFBQAAuGEtXCg98YS0Zo2jIwEqrypZULAOBQAAVVN2dtn19fLL0ogR0iuvSHfdJcXElF3fwI2kShYU48aN0759+7Rt2zZHhwIAAMqAMdLYsZK7u+TvL/3wQ+n7XLEib/8rV5a+T+BGVCULCgAAULX861/Su+9KmZlSQoI0alTp+2zdOu92mzal7xO4Ebk4OgAAAICinD9f+LYtXnpJysqSfv5Z6tdPGj269H0CNyK73aGYO3euQkJC5O3tLW9vb4WGhmrVqlXFOjY6OlpOTk4aPHhwOUcJAAAqooEDpebNr24/8UTp+6xRQ5o7N2f41KxZkpNT6fsEbkR2u0MREBCg2bNnKygoSJL08ccfa9CgQdq5c6daX3/P8RpHjx7VE088oW7dutkrVAAAUMH4+krbtkmxsVK9etJttzk6IgC5nIwxxlEn9/Pz05w5c/TYY49Z3Z+VlaU777xTjzzyiDZu3KikpCQtW7as2P2npKTIx8dHycnJ8vb2LquwAQAAAPyPQyZlZ2VlKTo6WhcvXlRoaGiB7Z599lndfPPNBRYc10tPT1dKSkqeF1Ae/v1vqWNHKTRU2rLF0dEAAAA4jl0nZe/evVuhoaFKS0uTp6enli5dquDgYKttN23apA8++EC7du0qdv+RkZF65plnyipcwKoTJ6T775fS03O2+/fP+czd3bFxAQAAOIJd71C0aNFCu3bt0pYtWzR27FiNHDlS+/bty9cuNTVVw4cP1/z581W7du1i9//kk08qOTnZ8jp+/HhZhg9Ikn777WoxIUlnz5bN00YAAAAqI4fOoejdu7eaNm2qefPm5fl8165d6tChg5ydnS2fZf9vacxq1arpwIEDatq0aZH9M4cC5eHSpZzhTgcO5GyHhUkbNvB0EAAAcGNy6DoUxhilX/un3v9p2bKldu/eneezGTNmKDU1VW+88YYCAwPtFSKQT40aUlyc9OGHOcOcRo+mmABQ+SUnS/HxUlCQVKuWo6MBUJnYraCYPn26wsPDFRgYqNTUVEVHRys2NlYxMTGSpBEjRqhBgwaKjIxU9erV1ea65Spr/e+/btd/DjhC7drSP/7h6CgAoGzs2yf17CmdOiXVqSOtWycVMMURAPKxW0Fx8uRJRUREKCEhQT4+PgoJCVFMTIz69OkjSTp27JiqVXPIQ6cAALihvfhiTjEh5fzviy9KH3/s2JgAVB52Kyg++OCDQvfHxsYWuv+jjz4qu2AAAICFq2vh2wBQGG4JAABwg5sxQ2rSJOd9kyY52wBQXA6dlF1eoqKiFBUVpaysLEeHAgBAhde4sfTLL1JCglS/vuTm5uiIAFQmDn1sbHnjsbEAAABA+WLIEwAAAACbUVAAAAAAsBkFBQAAAACbUVAAAG5YZ89KvXtL3t7SwIHShQuOjggAKh8KCgDADWv6dOm776TUVOlf/5IiIx0dEQBUPlWyoIiKilJwcLA6d+7s6FAAABXYyZOFbwMAisZjYwEAN6wVK6TBg6XMTKl6dWntWikszNFRAUDlUiUXtgMAoDjuuUfatk3atUvq0kVq1crREQFA5cMdCgAAAAA2q5JzKAAAAADYBwUFAAAAAJtRUAAAAACwGQUFAAAAAJtVyYKCdSgAAAAA++ApTwAAAABsViXvUAAAAACwD7sVFHPnzlVISIi8vb3l7e2t0NBQrVq1qsD28+fPV7du3eTr6ytfX1/17t1bW7dutVe4AAAAAIrBbgVFQECAZs+ere3bt2v79u3q1auXBg0apL1791ptHxsbq4cffljr1q3T5s2b1bBhQ/Xt21cnTpywV8gAAAAAiuDQORR+fn6aM2eOHnvssSLbZmVlydfXV2+//bZGjBhRrP6ZQwEAAACULxdHnDQrK0tffPGFLl68qNDQ0GIdc+nSJV25ckV+fn7lHB0AAACA4rJrQbF7926FhoYqLS1Nnp6eWrp0qYKDg4t17LRp09SgQQP17t27wDbp6elKT0+3bKekpJQ6ZgAAAAAFs+tTnlq0aKFdu3Zpy5YtGjt2rEaOHKl9+/YVedxLL72kJUuW6Ouvv1b16tULbBcZGSkfHx/LKzAwsCzDBwAAAHAdh86h6N27t5o2bap58+YV2Obll1/Wc889p2+//VadOnUqtD9rdygCAwOZQwEAAACUE4fMochljMlTAFxvzpw5eu6557R69eoiiwlJcnd3l7u7e1mGCAAAAKAQdisopk+frvDwcAUGBio1NVXR0dGKjY1VTEyMJGnEiBFq0KCBIiMjJeUMc5o5c6Y+/fRTNW7cWImJiZIkT09PeXp62itsAAAAAIWwW0Fx8uRJRUREKCEhQT4+PgoJCVFMTIz69OkjSTp27JiqVbs6peOdd95RRkaG7r///jz9zJo1S08//bS9wgYAAABQCIfOoShvrEMBAAAAlC+7PuUJAAAAQNVSJQuKqKgoBQcHq3Pnzo4OBQAAAKjSGPIEANeZN09avlxq2VJ6/nnJw8PREQEAUHE59LGxAFDRfP21NGZMzvtVq6SLF3MKDOBGl5kpHTwo1a4t1anj6GgAVCRVcsgTANhq+/a82zt2OCYOoCJJS5N69ZJat5YCA6Uvv3R0RAAqEgoKALhGz56Sk9PV7V69HBcLUFF8/rm0cWPO+4wMacoUx8YDoGJhyBMAXKNPn5z5E998I7VqJU2a5OiIAMe7tsi2tg3gxsakbAAAUKj0dCk8XFq3TqpeXVqyRBo82NFRAagouEMBAAAK5e4uffutdPiw5OeX8wKAXFVyDgXrUAAAULaqVZOCgigmAOTHkCcAAAAANmPIE+Ag33+f80jS7t2lLl0cHQ0AAIBtKCgAB1i8WBo+POe9s3POAmp9+jg2JgAAAFtUyTkUQEX36adX32dl5TzjHQAAoDKioAAcoEmTwrcBAAAqC4Y8AQ7wwgvS6dPS9u05KzM/8YSjIwIAALANT3kCAAAAYLMqOeSJdShQEezcKa1ZI1265OhIAAAAyg93KIBy8OKL0rRpOe/bt5fi4qSaNR0bEwAAQHmokncoAEd7/vmr73ftklascFwsAAAA5alCFxRz585VSEiIvL295e3trdDQUK1atcrRYQFF8vIqfBsAAKCqqNAFRUBAgGbPnq3t27dr+/bt6tWrlwYNGqS9e/c6OjSgUAsWSLVqSU5O0ujRUni4oyMCAAAoH5VuDoWfn5/mzJmjxx57rMi2N/ociosXpWeflY4flyIi+FJrb9nZUkaGVL26oyMBAAAoP5VmHYqsrCx98cUXunjxokJDQx0dTqUwapT05Zc57z//XNq8WeLBV/ZTrRrFBAAAqPoqfEGxe/duhYaGKi0tTZ6enlq6dKmCg4Ottk1PT1d6erplOyUlxV5hVkibNl19n5UlbdlCQQEAAICyVaHnUEhSixYttGvXLm3ZskVjx47VyJEjtW/fPqttIyMj5ePjY3kFBgbaOdqK5Y47rr53dpa6dHFcLAAAAKiaKt0cit69e6tp06aaN29evn3W7lAEBgbesHMoLlyQnn766hyK/v0dHREAAACqmgo/5Ol6xpg8RcO13N3d5e7ubueIKi5PT+nllx0dBQAAAKqyCl1QTJ8+XeHh4QoMDFRqaqqio6MVGxurmJgYR4cGAAAAQBW8oDh58qQiIiKUkJAgHx8fhYSEKCYmRn369HF0aAAAAABUCedQlMSNvg4FAAAAUN4q/FOeAAAAAFRcVbKgiIqKUnBwsDqz6AIAAABQrhjyBAAAAMBmVfIOBQAAAAD7oKAAAAAAYDMKCgAAAAA2o6AAAAAAYDMKCgAAAAA2o6AAAAAAYLMqWVCwDgWKcumSdOiQlJHh6EgAAAAqN9ahwA1n926pTx/p5EmpeXMpNlaqX9/RUQEAAFROVfIOBVCYmTNziglJOnhQeuUVx8YDAABQmVFQ4IaTnV34NgAAAIqPggI3nFmzJD+/nPeNG0uTJzs0HAAAgErNxdEBAPZ2663Sf/8rHT0qNWsm1ajh6IgAAAAqLwoK3JBq1cp5AQAAoHQY8gQAAADAZlWyoGAdCgAAAMA+WIcCAAAAgM2q5B0KAAAAAPZhl4IiMjJSnTt3lpeXl+rUqaPBgwfrwIEDRR73+uuvq0WLFvLw8FBgYKAmT56stLQ0O0R84/r9d+nzz6WdOx0dCQAAACoDuxQU69ev17hx47RlyxatXbtWmZmZ6tu3ry5evFjgMYsXL9a0adM0a9Ys7d+/Xx988IE+++wzPfnkk/YI+YZ06JDUrp300ENSp07SwoWOjggAAAAVnV0eGxsTE5Nne8GCBapTp4527Nih7t27Wz1m8+bN6tq1q4YOHSpJaty4sR5++GFt3bq13OO9US1aJJ05k/M+O1t66y0pIsKxMQEAAKBic8gciuTkZEmSX+5yxVaEhYVpx44dlgLi8OHDWrlype65554Cj0lPT1dKSkqeF4rvppvybhfyzwMAAABIcsBTnowxGjRokM6fP6+NGzcW2vatt97SlClTZIxRZmamxo4dq3feeafA9k8//bSeeeaZfJ/zlKfiyciQhg+Xli2TWrSQli6VgoIcHRUAAAAqMrsXFOPGjdOKFSsUFxengICAAtvFxsZqyJAheu6559SlSxcz2Pi7AAAauElEQVQdOnRIkyZN0ujRozVz5kyrx6Snpys9Pd2ynZKSosDAQAoKAAAAoJzYtaCYMGGCli1bpg0bNqhJkyaFtu3WrZtuv/12zZkzx/LZokWL9Oc//1kXLlxQtWpFj9ZiHQoAAACgfNllUrYxRhMmTNDSpUsVGxtbZDEhSZcuXcpXNDg7O8sYoyq8Fh8AAABQqdiloBg3bpw+/fRTLV++XF5eXkpMTJQk+fj4yMPDQ5I0YsQINWjQQJGRkZKkAQMG6NVXX1WHDh0sQ55mzpypgQMHytnZ2R5hF+rIEWnOHMnJSfrHP6RGjRwdEQAAAGB/dhny5OTkZPXzBQsWaNSoUZKkHj16qHHjxvroo48kSZmZmXr++ee1cOFCnThxQjfffLMGDBig559/XrVq1SrWectryNOlS1LLltLx4znbTZpI+/ZJ1auX2SkAAACASsHuk7LtqbwKij17pLZt83524IDUvHmZnQIAAACoFByyDkVl16iRVKfO1e369aVCHliFKuCXX6Tp06VXX5WueZAYAADADc8ucyjsLSoqSlFRUcrKyiqX/r28pO++k/7v/3LmUDz1lFSjRrmcChXA8eNSaKiUlJSz/cMP0pdfOjYmAACAioIhT0ARliyRhg69uu3uLqWlOS4eAACKsmfPHrVr106jR4/Wu+++6+hwUMGlpKSoUaNGat26teLi4kp8PEOegCK0aiVd+wTj4GDHxQIAQHFMnTpVzs7OevLJJ/Pta9y4sZycnCwPwimOY8eO6W9/+5vatGmjmjVrysPDQw0bNtQdd9yhv//971q9enWRfbRt21ZOTk7y8PBQSkpKge2cnJxK/OrRo0exrwX5eXt7a+LEidq0aZOWL19e4uOr5JAnoCy1by8tWiS9/bZUu7b0xhuOjggAgIJt3LhRK1eu1KhRo9SoDJ5r//3332vw4MFKTU2Vs7OzAgMDVadOHZ07d05btmzR5s2btWDBAp05c6bAPnbt2qU9e/ZIktLS0vTll1/q0Ucftdq2a9eu+T5LTk62HG9tf9vrn5aDEvvrX/+ql19+WU8++aQGDhxY4FNaraGgAIrh4YdzXgAAlMR//vMfDRkyRL/++qsuX74sSapTp4769u2rhQsX5mvfuXNnbd++XZLk6empvn376quvvirROd9++21J0siRI0sZfc5QmIceekipqam65557FBUVladISUpK0vLly/X5558X2k/utdaqVUtJSUlauHBhgQWFtSE3sbGx6tmzZ4H7UXq+vr4aMGCAPvvsM33//ff6wx/+UOxjGfIEAABQTkJCQrRv3z7t2bNH1f43fnbbtm1WiwlJWr9+vXx8fPT666/rzJkzJS4mTp8+rWXLlsnf31/du3cvdfwrV67UmTNn5O3trc8//zzfHY9atWpp5MiRWrFiRYF9ZGVlacmSJZJyih1nZ2etX79ex44dK3V8KFtDhgyRJL3//vslOq5KFxReXl5KTk6Wl5eXo0MBAAA3sFtuucUyzr+wv+ZPmDBBCxcu1KRJk+Tu7l7i8yxdulQZGRkKDw+3FDClcfjwYUlS8+bNVcPGR1p+++23SkhIUL169TRkyBD16tVLxhgtXry41PGVxNGjR/WXv/xFt9xyi9zd3eXl5aVbbrlF9957r6Kjo60ek5mZqfnz56tnz5666aabVL16dd1yyy364x//aHWuwdmzZ/WPf/xDLVq0kIeHh3x9fdWjRw8tXrxYBT0HKXceiCR99dVX6t69u2rVqiUnJyf9+uuveWJ59913FRYWplq1aql69epq2bKlZsyYYXVOii3X269fP7m4uGjZsmVKL8lz8g0AAADK3UcffWQkmTZt2ljdP2PGDDN//vxSnWPYsGFGkpk3b16BbRo1amQkmQULFhTZ31tvvWUkGR8fH3P+/HmbYho6dKiRZCZNmmSMuZqHVq1aFbuPdevWGUnG1q+uR44cMbVr1zaSTI0aNUzbtm1N+/btjZ+fn5Fk2rVrl++Yc+fOma5du1rO26hRI9OpUydTp04dy/a14uPjTWBgoJFk3NzcTMeOHc0tt9xiOX7EiBEmOzs733ly98+ePdtIMnXr1jWdO3c2N998szly5Igxxpjk5GTTvXt3I8lUq1bNNGrUyLRp08a4ublZcnny5MlSXW+uDh06GElm48aNxc4vBQUAAIAdpKammpo1axpJZvv27Xn2zZ8/3zz11FOlPkeTJk2s9n+tkhQUBw4cMNWqVTOSzK233mq+/PJLk5SUVOx4UlNTTY0aNYwks3XrVmOMMSkpKcbDw6PIOK9V2oJi/PjxRpIZOXKkSU1NzbNv//79VguwwYMHG0mmadOmZsuWLXn2xcfHm5deesmynZ2dbTp16mQkmTvvvNMkJiZa9q1atcry7/7OO+/kO0/udbm5uZn33nvPUnRcuXLFXLlyxRhjzJAhQ4wk84c//MH897//tRx77tw5c9999xlJ5v777y/V9eb685//bCSZyMjIAtvku4ZitwQAAECpREREGElm/Pjxls9WrlxpHnvssVL3nZ2dbVxcXIwkc+LEiQLblaSgMMaY559/3vKlV5JxcnIyLVq0MKNGjTLR0dEmLS2twGNz70YEBQXl+fyBBx7Ic9eiKKUtKPr162ckmZ9//rlY7bdu3WokGXd3d3Pw4MEi269du9bSPiEhId/+l156yXJX4/q7FLnXNWHCBKt9//zzz5ZjU1JS8u2/ePGiCQwMNE5OTubXX381xpT8eq81a9YsI8mMGTOm2MdU6TkUAAAAFcmIESMkSUuWLFFGRoZ27Nih+fPna+7cuaXuOykpSZmZmZIkPz+/UveXa/r06fr+++919913y83NTcYYHThwQB999JGGDBmi5s2bKzY21uqxuZPPh167QqykYcOGScrJQ27M5SkwMFCS9OWXXxY4l+FaufMj7r33XjVr1qzI9mvWrJEkPfDAA6pXr16+/WPGjJG7u7uOHj2qAwcOWO0j92fjekuXLpUkPfjgg1bnBdeoUUO9e/eWMUYbN26UVPLrvVbuz87p06eLfQwFBQAAgJ306tVLAQEBOnv2rN5++23NmDFDH3/8sVxdXfO1ve2227RgwYJi952WlmZ57+bmVibx5urZs6dWrFihpKQkbdiwQXPmzFHPnj3l5OSkY8eO6e6779Yvv/yS55gTJ05o3bp1kvIXFOHh4fL19dWpU6csX8bL07hx4+Tq6qr/+7//U5MmTTRmzBgtXrxYv//+u9X2+/fvlyTdfvvtxer/4MGDkqTgAla/9fLysnzJz217vVatWln9fPfu3ZJyCouwsDCrr7Vr10rKyblU8uu9loeHhyRZHnNcHBQUAAAAdlKtWjUNHz5ckvT666/rgw8+KPBplBMmTNCAAQOK3fe1dyWSk5NLF2gBPDw81K1bNz3xxBP6/vvvtWHDBtWsWVOXL1/WK6+8kqft4sWLlZ2drY4dO6pFixZ59rm5uemBBx6QpAIfoVuW2rdvrw0bNqhv3746ceKE5s2bp+HDhysgIED9+vWzFBC5cp+aVKtWrWL1f+HCBUk5a4wUpG7dupKk1NRUq/tr1qxp9fPcf8tDhw5p06ZNVl+//fabpKtFQEmv91rnzp2TJNWuXbuwS86DggIAAMCOfHx8JElTp06Vv79/ge0iIiJK9KXO3d1d3t7ekq5+KSxvYWFhevzxxyVJW7duzbMvt1D46aefLI9Gvfb13nvvScoZXmTtsadl7fbbb9fq1at1/vx5xcTEaOrUqQoICNCaNWvUp08fJSUlWdrmFnnXflYYT09PSdKpU6cKbHPy5Mk8fRdXbt/z58+XyZn/XODr6aeftul6r5X7s3PzzTcXO0YKCgAAADvKnW+Quy5FWWrfvr0kFfoX6LJ2yy23SJIyMjIsn+3cuVN79uyRk5OT6tatW+DLzc1Nly9fLvECfqXh6empfv36afbs2frll1/UtGlTnThxQqtWrbK0ad26tSRpy5YtxeqzefPmkqR9+/ZZ3Z+amqrjx4/naVtcucOo9uzZU6LjchXneq+Vew0dO3Ys9jkoKAAAAOwkMzNTmzZtUp06dSxfWq/33nvvadasWZoxY0aJ+w8LC5Mkbd++vVRx5jpz5kyRk3p/+OEHScozeTn37kT37t2VmJhY4GvKlCl52ttbjRo11LZtW0nKM79g8ODBkqRly5bpv//9b5H99OvXT5L0xRdfKDExMd/+efPmKT09XY0aNco3/Kso9957ryRp0aJFOnv2bImOvV5B13utbdu2SZK6detW7H4pKAAAAOxkx44dunDhgu68806r+3/88Uf5+/urW7du+ve//13i/vv27StJiouLK1WcuRYtWqT27dtr/vz5+b7MJiUl6amnntKiRYskSY888ogkKSsrS0uWLJGUM2yrMLnzSWJjYy1/wS8PY8eO1WeffaZLly7l+XzDhg367rvvJOX9i/ytt96qe++9V2lpaQoPD7d8yc516NAhvfzyy5btXr16qXPnzkpPT9fDDz+cZ+jTmjVr9Mwzz0iSpk2bZlkVu7g6deqkBx98UGfPnlWfPn20c+fOPPuzsrIUGxurYcOGWVa3Lun1XntdJ0+eVMuWLS2TyIulxA+nBQAAgE2eeuopI8m8+eabVvfHxcWZrKws86c//cnMmDGjxP1nZ2eboKAg4+zsnGdxtWvlrkPh6elpbrrppgJfu3fvNq+//nqeNSiaNGlibrvtNtOsWTPLKs2SzBNPPGHpf9WqVUaSqV69erEWwctdmbmwhdRKuw5Fu3btjCTj4uJiWrVqZW677TZLHiSZ4cOH5zvm3LlzJjQ01NKmcePGplOnTqZu3boFrpQdEBBgWY+iY8eOJigoyHJ8REREoStlFyY1NdX06dPH0rZhw4amS5cupm3btpZFAiWZy5cv23y9xhjz3HPPGUl5Fu0rDqf/XQiuYYwpcAY+AAC4cXl5eZX4L8yZmZkKCwvT6dOndeTIERlj5Ovrq4YNGyoyMlLh4eF52l+8eFH+/v7auXOnZX5CSbz00kuaOnWq3njjDU2cODHf/saNG+vo0aNF9rNz5061bt1acXFxWrVqleVpQomJiXJxcVGDBg0UGhqq0aNHW4ZaSTlrTHz66ad64IEH9Pnnnxd5nldffVVTpkxRcHCw9u7da7VNbGysevbsKUklXldBktatW6fly5dr48aNOn78uJKTk1W/fn21bNlS48aNU//+/a3+u165ckXz58/Xp59+qj179ig9PV3169fXrbfeqpEjR6p///552p85c0Yvvviili9frmPHjsnd3V3t27fX6NGjNWzYMKvnyP2sqOvKzs5WdHS0PvnkE+3YsUPJycmqXbu2GjZsqB49euiPf/yjOnfuXKrrbdOmjQ4ePKjffvut0CdW5bsGCor8UlJSLE9gAAAAyJWcnGx5klJ5+fDDD7V48WJ999132rRpk7p27Vqi41NSUtS0aVP5+flp//79qlaNEe4o2rp169SrVy89/vjjioqKKtGxFBRWlOQORefOnfONqyutsu6zrPtLSUlRYGCgjh8/Xqb/Ua3o113WfVaWPJZHn/xMlp3KkMvKcN2VIY9Sxb/u8uizov1M2nKHoqTuvfdehYeH684779TBgwdLtBZFrjfeeEN//etftXjx4nyLygHW9OzZU9u3b9ehQ4csa2YUl0s5xVSpOTk5Ffs/Ws7OzmX+l4qy7rM8YpQkb2/vCh1nZfi3kSp+HsujT34my05lyGVluO7KkEepclx3ZchlecVYVu655x7L8JoJEybY1MfYsWOVkpKi7OzsMo4OVVFKSop69OihiRMnlriYkCgoSm3cuHEVvs/yiLE8VIbrrgy5rAzXXRnyKFWO664MuawM110Z8ihVjuuuDLms6DH+6U9/KnUfbm5umjlzZhlEgxuBt7e3Zs2aZfPxDHlCieXOMbHHONKqjDyWHXJZdshl2SCPZYdcAhWf89PXrtENFJOzs7N69OghFxducpUGeSw75LLskMuyQR7LDrkEKjbuUAAAAACwGc8RAwAAAGAzCgoAAAAANqOgAAAAAGAzCgoAAAAANqOggCTp6aeflpOTU55XvXr18rTZv3+/Bg4cKB8fH3l5een222/XsWPHLPtzF+CpXbu2atasqYEDB+q3336z96U4VFF5vHDhgsaPH6+AgAB5eHioVatWmjt3bp4+yONVJ06c0PDhw3XTTTepRo0aat++vXbs2GHZb4zR008/LX9/f3l4eKhHjx7au3dvnj7Onz+viIgI+fj4yMfHRxEREUpKSrL3pThUYXm8cuWKpk6dqrZt26pmzZry9/fXiBEj9Pvvv+fpgzzmKOpn8lp/+ctf5OTkpNdffz3P5+SyeHnkdw5QeVBQwKJ169ZKSEiwvHbv3m3Z99///ldhYWFq2bKlYmNj9fPPP2vmzJmqXr26pc1f//pXLV26VNHR0YqLi9OFCxfUv39/ZWVlOeJyHKawPE6ePFkxMTFatGiR9u/fr8mTJ2vChAlavny5pQ15zHH+/Hl17dpVrq6uWrVqlfbt26dXXnlFtWrVsrR56aWX9Oqrr+rtt9/Wtm3bVK9ePfXp00epqamWNkOHDtWuXbsUExOjmJgY7dq1SxEREY64JIcoKo+XLl3STz/9pJkzZ+qnn37S119/rYMHD2rgwIF5+rnR8ygV72cy17Jly/Tjjz/K398/374bPZfFySO/c4BKxgDGmFmzZpl27doVuP+hhx4yw4cPL3B/UlKScXV1NdHR0ZbPTpw4YapVq2ZiYmLKNNaKrKg8tm7d2jz77LN5PuvYsaOZMWOGMYY8Xmvq1KkmLCyswP3Z2dmmXr16Zvbs2ZbP0tLSjI+Pj3n33XeNMcbs27fPSDJbtmyxtNm8ebORZH755ZfyC74CKSqP1mzdutVIMkePHjXGkMdcxc3lb7/9Zho0aGD27NljGjVqZF577TXLPnJZvDzyOweoXLhDAYv4+Hj5+/urSZMmGjJkiA4fPixJys7O1ooVK9S8eXP169dPderUUZcuXbRs2TLLsTt27NCVK1fUt29fy2f+/v5q06aNfvjhB7tfiyMVlEdJCgsL0zfffKMTJ07IGKN169bp4MGD6tevnyTyeK1vvvlGnTp10gMPPKA6deqoQ4cOmj9/vmX/kSNHlJiYmCdX7u7uuvPOOy252rx5s3x8fNSlSxdLm9tvv10+Pj43TD6LyqM1ycnJcnJysvzFmDzmKE4us7OzFRERob///e9q3bp1vj7IZdF55HcOUPlQUECS1KVLF33yySdavXq15s+fr8TERN1xxx06e/asTp06pQsXLmj27Nm66667tGbNGt1777267777tH79eklSYmKi3Nzc5Ovrm6ffunXrKjEx0RGX5BCF5VGS3nzzTQUHBysgIEBubm6666679M477ygsLEwSebzW4cOHNXfuXDVr1kyrV6/WmDFjNHHiRH3yySeSZMlH3bp18xx3ba4SExNVp06dfH3XqVPnhslnUXm8XlpamqZNm6ahQ4fK29tbEnnMVZxcvvjii3JxcdHEiROt9kEui84jv3OAyoc17CFJCg8Pt7xv27atQkND1bRpU3388ccaMmSIJGnQoEGaPHmyJKl9+/b64Ycf9O677+rOO+8ssF9jjJycnMo3+AqksDz+7W9/05tvvqktW7bom2++UaNGjbRhwwY9/vjjql+/vnr37l1gvzdaHqWcv1J26tRJL7zwgiSpQ4cO2rt3r+bOnasRI0ZY2l2fl+tzZS1vN1I+i5tHKWeC9pAhQ5Sdna133nknz74bPY9S0bncsWOH3njjDf3000+F5uVGz2VReczOzpbE7xygMuEOBayqWbOm2rZtq/j4eNWuXVsuLi4KDg7O06ZVq1aWJ27Uq1dPGRkZOn/+fJ42p06dyvcX5BvJtXm8fPmypk+frldffVUDBgxQSEiIxo8fr4ceekgvv/yyJPJ4rfr16xf5Mycp318jr81VvXr1dPLkyXx9nz59+obJZ1F5zHXlyhU9+OCDOnLkiNauXWu5OyGRx1xF5XLjxo06deqUGjZsKBcXF7m4uOjo0aOaMmWKGjduLIlcSkXnkd85QOVDQQGr0tPTtX//ftWvX19ubm7q3LmzDhw4kKfNwYMH1ahRI0nSrbfeKldXV61du9ayPyEhQXv27NEdd9xh19grkmvzeOXKFV25ckXVquX9v52zs7PlL3Lk8aquXbsW+jPXpEkT1atXL0+uMjIytH79ekuuQkNDlZycrK1bt1ra/Pjjj0pOTr5h8llUHqWrxUR8fLy+/fZb3XTTTXnak8ccReUyIiJC//nPf7Rr1y7Ly9/fX3//+9+1evVqSeRSKjqP/M4BKiHHzQdHRTJlyhQTGxtrDh8+bLZs2WL69+9vvLy8zK+//mqMMebrr782rq6u5r333jPx8fHmrbfeMs7Ozmbjxo2WPsaMGWMCAgLMt99+a3766SfTq1cv065dO5OZmemoy7K7ovJ45513mtatW5t169aZw4cPmwULFpjq1aubd955x9IHecyxdetW4+LiYp5//nkTHx9vFi9ebGrUqGEWLVpkaTN79mzj4+Njvv76a7N7927z8MMPm/r165uUlBRLm7vuusuEhISYzZs3m82bN5u2bdua/v37O+KSHKKoPF65csUMHDjQBAQEmF27dpmEhATLKz093dLPjZ5HY4r3M3m965/yZAy5LE4e+Z0DVC4UFDDG5Dyir379+sbV1dX4+/ub++67z+zduzdPmw8++MAEBQWZ6tWrm3bt2plly5bl2X/58mUzfvx44+fnZzw8PEz//v3NsWPH7HkZDldUHhMSEsyoUaOMv7+/qV69umnRooV55ZVXTHZ2tqUNebzqX//6l2nTpo1xd3c3LVu2NO+9916e/dnZ2WbWrFmmXr16xt3d3XTv3t3s3r07T5uzZ8+aYcOGGS8vL+Pl5WWGDRtmzp8/b8/LcLjC8njkyBEjyepr3bp1lnbkMUdRP5PXs1ZQkMvi5ZHfOUDl4WSMMY68QwIAAACg8mIOBQAAAACbUVAAAAAAsBkFBQAAAACbUVAAAAAAsBkFBQAAAACbUVAAAAAAsBkFBQAAAACbUVAAAAAAsBkFBQAAAACbUVAAAAAAsBkFBQAAAACbUVAAAAAAsNn/A6S6RcdqFL2wAAAAAElFTkSuQmCC\n",
"text/plain": [
"Graphics object consisting of 1 graphics primitive"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import numpy as np\n",
"LSAT=np.array([576, 635, 558, 578, 666, 580, 555, 661, 651, 605, 653, 575, 545, 572, 594]) # LSAT data\n",
"GPA=np.array([3.39, 3.30, 2.81, 3.03, 3.44, 3.07, 3.00, 3.43, 3.36, 3.13, 3.12, 2.74, 2.76, 2.88, 3.96]) # GPA data\n",
"# use np.vstack to vertically stack a sequence of arrays into a matrix of stacked rows\n",
"# see https://docs.scipy.org/doc/numpy/reference/generated/numpy.vstack.html#numpy.vstack\n",
"LSATGPA_data = np.vstack((LSAT,GPA)) # this is a 2D array of matric now\n",
"p = points(zip(LSATGPA_data[0,:],LSATGPA_data[1,:]))\n",
"p.show(figsize=(8,3),axes_labels=['$Y_i$ (LSAT scores)','$Z_i$ (GPA)'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Plug-in Point Estimate of the Correlation\n",
"\n",
"The law school was interested in the **correlation** between the GPA and LSAT scores:\n",
"\n",
"$$\n",
"\\boxed{\n",
"\\theta^* = \\frac{\\int \\int (y-E(Y))(z-E(Z))dF(y,z)}{\\sqrt{\\int (y-E(Y))^2 dF(y) \\int (z-E(Z))^2 dF(z)}}\n",
"}\n",
"$$\n",
"\n",
"The *plug-in estimate of the population correlation* $\\theta^*$ is the *sample correlation*:\n",
"$$\n",
"\\boxed{\n",
"\\widehat{\\Theta}_n = \\frac{\\sum_{i=1}^n(Y_i-\\overline{Y}_n)(Z_i-\\overline{Z}_n)}{\\sqrt{\\sum_{i=1}^n(Y_i-\\overline{Y}_n)^2 \\sum_{i=1}^n(Z_i-\\overline{Z}_n)^2}}\n",
"}\n",
"$$"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The Sample Correlation Matrix is\n",
"[[1. 0.54591892]\n",
" [0.54591892 1. ]]\n",
"For a pair of variables we can extract the right entry of the symmetric Sample Correlation Matrix:\n",
"0.5459189161795885\n"
]
}
],
"source": [
"# the sample correlation matrix, it's symmetric with diagonal elements as 1\n",
"SampleCorrelationMatrix = np.corrcoef(LSATGPA_data) \n",
"print \"The Sample Correlation Matrix is\"\n",
"print SampleCorrelationMatrix\n",
"print \"For a pair of variables we can extract the right entry of the symmetric Sample Correlation Matrix:\"\n",
"SampleCorrelation_GPA_LSAT = SampleCorrelationMatrix[(0,1)]\n",
"print SampleCorrelation_GPA_LSAT"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The Bootstrapped $1-\\alpha$ Confidence Interval of the Sample Correlation and a Test\n",
"\n",
"In order to obtain the $1-\\alpha$ confidence interval of the plug-in estimate of the population correlation, we can quickly tap into our function thousand bootstrapped data sets.\n",
"\n",
"Then we can reject (or fail to reject) the null hypothesis that the true correlation coefficient is $0$ if $0$ is not contained in the bootstrapped 95% confidence interval (akin to a Wald Test, but with the nonparametric model)."
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"ename": "NameError",
"evalue": "name 'makeBootstrappedConfidenceIntervalOfStatisticT' is not defined",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mB\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mInteger\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1000\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# number of bootstrap samples\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;31m# get bootstrapped 1-alpha confidence Interval for statT99thPercentile\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mlowerCITcor\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mupperCITcor\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mbootValuesTcor\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmakeBootstrappedConfidenceIntervalOfStatisticT\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mLSATGPA_data\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstatTSampleCorrCoeff\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0malpha\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mB\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mInteger\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m>=\u001b[0m\u001b[0mlowerCITcor\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mInteger\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m<=\u001b[0m\u001b[0mupperCITcor\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mNameError\u001b[0m: name 'makeBootstrappedConfidenceIntervalOfStatisticT' is not defined"
]
}
],
"source": [
"# use the right lamba expression for the statistic of interest\n",
"statTSampleCorrCoeff = lambda dataMatrix : np.corrcoef(dataMatrix)[(0,1)] #statistic of interest\n",
"alpha=0.05\n",
"B=1000 # number of bootstrap samples\n",
"# get bootstrapped 1-alpha confidence Interval for statT99thPercentile\n",
"lowerCITcor,upperCITcor,bootValuesTcor = \\\n",
" makeBootstrappedConfidenceIntervalOfStatisticT(LSATGPA_data, statTSampleCorrCoeff, alpha, B) \n",
"\n",
"if 0>=lowerCITcor and 0<=upperCITcor:\n",
" print \"0 is inside the 1-alpha bootstrapped confidence interval for the correlation coefficient:\"\n",
" print \" So, we fail to reject the Null Hypothesis that the population correlation coefficient = 0\"\n",
"else:\n",
" print \"0 is not inside the 1-alpha bootstrapped confidence interval for the correlation coefficient:\"\n",
" print \" So, we reject the Null Hypothesis that the population correlation coefficient = 0\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Why is testing whether correlation = 0 is of interest?\n",
"### Correlation Versus Causation - Proceed with Extreme Caution!\n",
"\n",
"The following image from [https://en.wikipedia.org/wiki/Correlation_and_dependence](https://en.wikipedia.org/wiki/Correlation_and_dependence) shows:\n",
"> Several sets of (x, y) points, with the (Pearson's) correlation coefficient of x and y for each set. Note that the correlation reflects the noisiness and direction of a linear relationship (top row), but not the slope of that relationship (middle), nor many aspects of nonlinear relationships (bottom). N.B.: the figure in the center has a slope of 0 but in that case the correlation coefficient is undefined because the variance of Y is zero.\n",
"\n",
"![\"Correlation](\"https://upload.wikimedia.org/wikipedia/commons/thumb/d/d4/Correlation_examples2.svg/1200px-Correlation_examples2.svg.png\")
\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Nonparametric Hypothesis Testing \n",
"## Are two samples from the same distribution or not?\n",
"\n",
"What if we are not interested in estimating $F^*$ itself, but we are interested in scientificially investigating whether two distributions are the same. For example, perhaps, whether the distribution of earthquake magnitudes was the same in April as it was in March or whether human subjects under a clinical trial are reacting differently two a new drug to treat some disease (typically you have a control group that does not get the treatment and the treated group).\n",
"\n",
"You already have the means to do by using a Wald test of say the difference in sample means of the two samples, or correlation coefficient, etc for parametric or nonparametric experiments.\n",
"\n",
"We will next see a useful nonparametric approach to this problem.\n",
"\n",
"\n",
"## Permutation Testing\n",
"\n",
"A Permuation Test is a **non-parametric exact** method for testing whether two distributions are the same based on samples from each of them. In industry analogs and variants of permutation testing is known as *A/B Testing* in industry today.\n",
"\n",
"What do we mean by \"non-parametric exact\"? It is non-parametric because we do not impose any parametric assumptions. It is exact because it works for any sample size.\n",
"\n",
"Formally, we suppose that: \n",
"$$ X_1,X_2,\\ldots,X_m \\overset{IID}{\\sim} F^* \\quad \\text{and} \\quad X_{m+1}, X_{m+2},\\ldots,X_{m+n} \\overset{IID}{\\sim} G^* \\enspace , $$\n",
"are two sets of independent samples where the possibly unknown DFs \n",
"$F^*,\\,G^* \\in \\{ \\text{all DFs} \\}$.\n",
"\n",
"(Notice that we have written it so that the subscripts on the $X$s run from 1 to $m+n$.)\n",
"\n",
"Now, consider the following hypothesis test: \n",
"$$H_0: F^*=G^* \\quad \\text{versus} \\quad H_1: F^* \\neq G^* \\enspace . $$\n",
"\n",
"Our test statistic uses the observations in both both samples. We want a test statistic that is a sensible one for the test, i.e., will be large when when $F^*$ is 'too different' from $G^*$\n",
"\n",
"So, let our test statistic $T(X_1,\\ldots,X_m,X_{m+1},\\ldots,X_{m+n})$ be say: \n",
"$$\n",
"T:=T(X_1,\\ldots,X_m,X_{m+1},\\ldots,X_{m+n})= \\text{abs} \\left( \\frac{1}{m} \\sum_{i=1}^m X_i - \\frac{1}{n} \\sum_{i=m+1}^n X_i \\right) \\enspace .\n",
"$$\n",
"\n",
"(In words, we have chosen a test statistic that is the absolute value of the difference in the sample means. Note the limitation of this: if $F^*$ and $G^*$ have the same mean but different variances, our test statistic $T$ will not be large.)\n",
"\n",
"Then the idea of a permutation test is as follows:\n",
"\n",
"- Let $N:=m+n$ be the pooled sample size and consider all $N!$ permutations of the observed data $x_{obs}:=(x_1,x_2,\\ldots,x_m,x_{m+1},x_{m+2},\\ldots,x_{m+n})$.\n",
"- For each permutation of the data compute the statistic $T(\\text{permuted data } x)$ and denote these $N!$ values of $T$ by $t_1,t_2,\\ldots,t_{N!}$.\n",
"- Under $H_0: X_1,\\ldots,X_m,X_{m+1},\\ldots,X_{m+n} \\overset{IID}{\\sim}F^*=G^*$, each of the permutations of $x= (x_1,x_2,\\ldots,x_m,x_{m+1},x_{m+2},\\ldots,x_{m+n})$ has the same joint probability $\\prod_{i=1}^{m+n} f(x_i)$, where $f(x_i)$ is the density function corresponding to $F^*=G^*$, $f(x_i)=dF(x_i)=dG(x_i)$. \n",
"- Therefore, the transformation of the data by our statistic $T$ also has the same probability over the values of $T$, namely $\\{t_1,t_2,\\ldots,t_{N!}\\}$. Let $\\mathbf{P}_0$ be this permutation distribution under the null hypothesis. $\\mathbf{P}_0$ is discrete and uniform over $\\{t_1,t_2,\\ldots,t_{N!}\\}$. \n",
"- Let $t_{obs} := T(x_{obs})$ be the observed value of the test statistic.\n",
"- Assuming we reject $H_0$ when $T$ is large, the P-value = $\\mathbf{P}_0 \\left( T \\geq t_{obs} \\right)$\n",
"- Saying that $\\mathbf{P}_0$ is discrete and uniform over $\\{t_1, t_2, \\ldots, t_{N!}\\}$ says that each possible permutation has an equal probabability of occuring (under the null hypothesis). There are $N!$ possible permutations and so the probability of any individual permutation is $\\frac{1}{N!}$\n",
"\n",
"$$\n",
"\\text{P-value} = \\mathbf{P}_0 \\left( T \\geq t_{obs} \\right) = \\frac{1}{N!} \\left( \\sum_{j=1}^{N!} \\mathbf{1} (t_j \\geq t_{obs}) \\right), \\qquad \\mathbf{1} (t_j \\geq t_{obs}) = \\begin{cases} 1 & \\text{if } \\quad t_j \\geq t_{obs} \\\\ 0 & \\text{otherwise} \\end{cases}\n",
"$$\n",
"\n",
"This will make more sense if we look at some real data. \n",
"\n",
"## Permutation Testing with Shell Data\n",
"\n",
"In 2008, Guo Yaozong and Chen Shun collected data on the diameters of coarse venus shells from New Brighton beach for a course project. They recorded the diameters for two samples of shells, one from each side of the New Brighton Pier. The data is given in the following two cells.\n",
"\n",
"*NOTE - Real Data for Really Applied Statistics*: Under my guidance and assistance, students collected the shells on either side of the pier, cleaned the shells with warm soap water, rinsed and and treated with a diluted chlorine solution, used a taxonomy guide to identify the mollusc species and classified them manually. Then they focused on *Dosinia anus* or coarse venus shells. After the diluted chlorine treatment to disinfect microflora/fauna, the shells were washed, dried, placed on a graph paper, etched around the base with a mechanical pencil, and finally measured with a scale for the largest diameter within the outline."
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [],
"source": [
"leftSide = [52, 54, 60, 60, 54, 47, 57, 58, 61, 57, 50, 60, 60, 60, 62, 44, 55, 58, 55,\\\n",
" 60, 59, 65, 59, 63, 51, 61, 62, 61, 60, 61, 65, 43, 59, 58, 67, 56, 64, 47,\\\n",
" 64, 60, 55, 58, 41, 53, 61, 60, 49, 48, 47, 42, 50, 58, 48, 59, 55, 59, 50, \\\n",
" 47, 47, 33, 51, 61, 61, 52, 62, 64, 64, 47, 58, 58, 61, 50, 55, 47, 39, 59,\\\n",
" 64, 63, 63, 62, 64, 61, 50, 62, 61, 65, 62, 66, 60, 59, 58, 58, 60, 59, 61,\\\n",
" 55, 55, 62, 51, 61, 49, 52, 59, 60, 66, 50, 59, 64, 64, 62, 60, 65, 44, 58, 63]"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [],
"source": [
"rightSide = [58, 54, 60, 55, 56, 44, 60, 52, 57, 58, 61, 66, 56, 59, 49, 48, 69, 66, 49,\\\n",
" 72, 49, 50, 59, 59, 59, 66, 62, 44, 49, 40, 59, 55, 61, 51, 62, 52, 63, 39,\\\n",
" 63, 52, 62, 49, 48, 65, 68, 45, 63, 58, 55, 56, 55, 57, 34, 64, 66, 54, 65,\\\n",
" 61, 56, 57, 59, 58, 62, 58, 40, 43, 62, 59, 64, 64, 65, 65, 59, 64, 63, 65,\\\n",
" 62, 61, 47, 59, 63, 44, 43, 59, 67, 64, 60, 62, 64, 65, 59, 55, 38, 57, 61,\\\n",
" 52, 61, 61, 60, 34, 62, 64, 58, 39, 63, 47, 55, 54, 48, 60, 55, 60, 65, 41,\\\n",
" 61, 59, 65, 50, 54, 60, 48, 51, 68, 52, 51, 61, 57, 49, 51, 62, 63, 59, 62,\\\n",
" 54, 59, 46, 64, 49, 61]"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(115, 139)"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(leftSide), len(rightSide)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$(115 + 139)!$ is a very big number. Lets start small, and take a subselection of the shell data to demonstrate the permutation test concept: the first two shells from the left of the pier and the first one from the right:"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[52, 54, 58]"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rightSub = [52, 54]\n",
"leftSub = [58]\n",
"totalSample = rightSub + leftSub\n",
"totalSample"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So now we are testing the hypotheses\n",
"\n",
"$$\\begin{array}{lcl}H_0&:& X_1,X_2,X_3 \\overset{IID}{\\sim} F^*=G^* \\\\H_1&:&X_1, X_2 \\overset{IID}{\\sim} F^*, \\,\\,X_3 \\overset{IID}{\\sim} G^*, F^* \\neq G^*\\end{array}$$ \n",
"\n",
"With the test statistic\n",
"$$\\begin{array}{lcl}T(X_1,X_2,X_3) &=& \\text{abs} \\left(\\displaystyle\\frac{1}{2}\\displaystyle\\sum_{i=1}^2X_i - \\displaystyle\\frac{1}{1}\\displaystyle\\sum_{i=2+1}^3X_i\\right) \\\\ &=&\\text{abs}\\left(\\displaystyle\\frac{X_1+ X_2}{2} - \\displaystyle\\frac{X_3}{1}\\right)\\end{array}$$\n",
"\n",
"Our observed data $x_{obs} = (x_1, x_2, x_3) = (52, 54, 58)$\n",
"\n",
"and the realisation of the test statistic for this data is $t_{obs} = \\text{abs}\\left(\\displaystyle\\frac{52+54}{2} - \\frac{58}{1}\\right) = \\text{abs}\\left(53 - 58\\right) = \\text{abs}(-5) = 5$\n",
"\n",
"Now we need to tabulate the permutations and their probabilities. There are 3! = 6 possible permutataions of three items. For larger samples, you could use the `factorial` function to calculate this:"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"6"
]
},
"execution_count": 56,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"factorial(3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We said that under the null hypotheses (the samples have the same DF) each permutation is equally likely, so each permutation has probability $\\displaystyle\\frac{1}{6}$.\n",
"\n",
"There is a way in Python (the language under the hood in Sage), to get all the permuations of a sequence:"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[[52, 54, 58],\n",
" [52, 58, 54],\n",
" [54, 52, 58],\n",
" [54, 58, 52],\n",
" [58, 52, 54],\n",
" [58, 54, 52]]"
]
},
"execution_count": 57,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"list(Permutations(totalSample))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can tabulate the permuations, their probabilities, and the value of the test statistic that would be associated with that permutation:\n",
"\n",
"\n",
"\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[52, 54, 58] has t = 5\n",
"[52, 58, 54] has t = 1\n",
"[54, 52, 58] has t = 5\n",
"[54, 58, 52] has t = 4\n",
"[58, 52, 54] has t = 1\n",
"[58, 54, 52] has t = 4\n"
]
}
],
"source": [
"allPerms = list(Permutations(totalSample))\n",
"for p in allPerms:\n",
" t = abs((p[0] + p[1])/2 - p[2]/1)\n",
" print p, \" has t = \", t"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To calculate the P-value for our test statistic $t_{obs} = 5$, we need to look at how many permutations would give rise to test statistics that are at least as big, and add up their probabilities.\n",
"\n",
"$$\n",
"\\begin{array}{lcl}\\text{P-value} &=& \\mathbf{P}_0(T \\geq t_{obs}) \\\\&=&\\mathbf{P}_0(T \\geq 5)\\\\&=&\\frac{1}{6} + \\frac {1}{6} \\\\&=&\\frac{2}{6}\\\\ &=&\\frac{1}{3} \\\\ &\\approx & 0.333\\end{array}\n",
"$$\n",
"\n",
"We could write ourselves a little bit of code to do this in SageMath. As you can see, we could easily improve this to make it more flexible so that we could use it for different numbers of samples, but it will do for now."
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1/3"
]
},
"execution_count": 59,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"allPerms = list(Permutations(totalSample))\n",
"pProb = 1/len(allPerms)\n",
"pValue = 0\n",
"tobs = 5\n",
"for p in allPerms:\n",
" t = abs((p[0] + p[1])/2 - p[2]/1)\n",
" if t >= tobs:\n",
" pValue = pValue + pProb\n",
"pValue"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This means that there is little or no evidence against the null hypothesis (that the shell diameter observations are from the same DF).\n",
"\n",
"### Pooled sample size\n",
"\n",
"The lowest possible P-value for a pooled sample of size $N=m+n$ is $\\displaystyle\\frac{1}{N!}$. Can you see why this is? \n",
"\n",
"So with our small sub-samples the smallest possible P-value would be $\\frac{1}{6} \\approx 0.167$. If we are looking for P-value $\\leq 0.01$ to constitute very strong evidence against $H_0$, then we have to have a large enough pooled sample for this to be possible. Since $5! = 5 \\times 4 \\times 3 \\times 2 \\times 1 = 120$, it is good to have $N \\geq 5$\n",
"\n",
"### YouTry in class\n",
"\n",
"Try copying and pasting our code and then adapting it to deal with a sub-sample (52, 54, 60) from the left of the pier and (58, 54) from the right side of the pier. "
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[52, 54, 60, 58, 54]"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rightSub = [52, 54, 60]\n",
"leftSub = [58, 54]\n",
"totalSample = rightSub + leftSub\n",
"totalSample"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### You will have to think about:\n",
"\n",
"- calculating the value of the test statistic for the observed data and for all the permuations of the total sample\n",
"- calculating the probability of each permutation\n",
"- calculating the P-value by adding the probabilities for the permutations with test statistics at least as large as the observed value of the test statistic"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"(add more cells if you need them)\n",
"\n",
"(end of You Try)\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can use the sample function and the Python method for making permutations to experiment with a larger sample, say 5 of each."
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[58, 48, 67, 50, 62, 63, 56, 40, 40, 49]"
]
},
"execution_count": 61,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"n, m = 5, 5\n",
"leftSub = sample(leftSide, n)\n",
"rightSub = sample(rightSide,m)\n",
"totalSample = leftSub + rightSub\n",
"leftSub; rightSub; totalSample"
]
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"37/5"
]
},
"execution_count": 62,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tobs = abs(mean(leftSub) - mean(rightSub))\n",
"tobs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have met sample briefly already: it is part of the Python random module and it does exactly what you would expect from the name: it samples a specified number of elements randomly from a sequence."
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [],
"source": [
"#define a helper function for calculating the tstat from a permutation\n",
"def tForPerm(perm, samplesize1, samplesize2):\n",
" '''Calculates the t statistic for a permutation of data given the sample sizes to split the permuation into.\n",
" \n",
" Param perm is the permutation of data to be split into the two samples.\n",
" Param samplesize1, samplesize2 are the two sample sizes.\n",
" Returns the absolute value of the difference in the means of the two samples split out from perm.'''\n",
" sample1 = [perm[i] for i in range(samplesize1)]\n",
" sample2 = [perm[samplesize1+j] for j in range(samplesize2)]\n",
" return abs(mean(sample1) - mean(sample2))"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"29/126"
]
},
"execution_count": 64,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"allPerms = list(Permutations(totalSample))\n",
"pProb = 1/len(allPerms)\n",
"pValue = 0\n",
"tobs = abs(mean(leftSub) - mean(rightSub))\n",
"for p in allPerms:\n",
" t = tForPerm(p, n, m)\n",
" if t >= tobs:\n",
" pValue = pValue + pProb\n",
"pValue"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"10"
]
},
"execution_count": 65,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"n+m"
]
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"3628800"
]
},
"execution_count": 66,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"factorial(n+m) # how many permutations is it checking"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see from the length of time it takes to do the calculation for $(5+5)! = 10!$ permutations, we will be here a long time if we try to this on all of the two shell data sets. \n",
"\n",
"#### Monte Carlo methods to the rescue! \n",
"We can use Monte Carlo integration to calculate an approximate P-value, and this will be our next topic. \n",
"\n",
" \n",
"\n",
"### You try\n",
"\n",
"Try working out the P-value for a sub-sample (58, 63) from the left of the pier and (61) from the right (the two last values in the left-side data set and the last value in the right-side one). Do it as you would if given a similar question in the exam: you choose how much you want to use Sage to help and how much you do just with pen and paper. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Example: Permutation test for shell diameters using the bootsrapped/permuted confidence interval"
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[52, 54, 60, 60, 54, 47, 57, 58, 61, 57, 50, 60, 60, 60, 62, 44, 55, 58, 55, 60, 59, 65, 59, 63, 51, 61, 62, 61, 60, 61, 65, 43, 59, 58, 67, 56, 64, 47, 64, 60, 55, 58, 41, 53, 61, 60, 49, 48, 47, 42, 50, 58, 48, 59, 55, 59, 50, 47, 47, 33, 51, 61, 61, 52, 62, 64, 64, 47, 58, 58, 61, 50, 55, 47, 39, 59, 64, 63, 63, 62, 64, 61, 50, 62, 61, 65, 62, 66, 60, 59, 58, 58, 60, 59, 61, 55, 55, 62, 51, 61, 49, 52, 59, 60, 66, 50, 59, 64, 64, 62, 60, 65, 44, 58, 63]\n",
"[58, 54, 60, 55, 56, 44, 60, 52, 57, 58, 61, 66, 56, 59, 49, 48, 69, 66, 49, 72, 49, 50, 59, 59, 59, 66, 62, 44, 49, 40, 59, 55, 61, 51, 62, 52, 63, 39, 63, 52, 62, 49, 48, 65, 68, 45, 63, 58, 55, 56, 55, 57, 34, 64, 66, 54, 65, 61, 56, 57, 59, 58, 62, 58, 40, 43, 62, 59, 64, 64, 65, 65, 59, 64, 63, 65, 62, 61, 47, 59, 63, 44, 43, 59, 67, 64, 60, 62, 64, 65, 59, 55, 38, 57, 61, 52, 61, 61, 60, 34, 62, 64, 58, 39, 63, 47, 55, 54, 48, 60, 55, 60, 65, 41, 61, 59, 65, 50, 54, 60, 48, 51, 68, 52, 51, 61, 57, 49, 51, 62, 63, 59, 62, 54, 59, 46, 64, 49, 61]\n",
"115 139\n"
]
},
{
"data": {
"text/plain": [
"array([52, 54, 60, 60, 54, 47, 57, 58, 61, 57, 50, 60, 60, 60, 62, 44, 55,\n",
" 58, 55, 60, 59, 65, 59, 63, 51, 61, 62, 61, 60, 61, 65, 43, 59, 58,\n",
" 67, 56, 64, 47, 64, 60, 55, 58, 41, 53, 61, 60, 49, 48, 47, 42, 50,\n",
" 58, 48, 59, 55, 59, 50, 47, 47, 33, 51, 61, 61, 52, 62, 64, 64, 47,\n",
" 58, 58, 61, 50, 55, 47, 39, 59, 64, 63, 63, 62, 64, 61, 50, 62, 61,\n",
" 65, 62, 66, 60, 59, 58, 58, 60, 59, 61, 55, 55, 62, 51, 61, 49, 52,\n",
" 59, 60, 66, 50, 59, 64, 64, 62, 60, 65, 44, 58, 63, 58, 54, 60, 55,\n",
" 56, 44, 60, 52, 57, 58, 61, 66, 56, 59, 49, 48, 69, 66, 49, 72, 49,\n",
" 50, 59, 59, 59, 66, 62, 44, 49, 40, 59, 55, 61, 51, 62, 52, 63, 39,\n",
" 63, 52, 62, 49, 48, 65, 68, 45, 63, 58, 55, 56, 55, 57, 34, 64, 66,\n",
" 54, 65, 61, 56, 57, 59, 58, 62, 58, 40, 43, 62, 59, 64, 64, 65, 65,\n",
" 59, 64, 63, 65, 62, 61, 47, 59, 63, 44, 43, 59, 67, 64, 60, 62, 64,\n",
" 65, 59, 55, 38, 57, 61, 52, 61, 61, 60, 34, 62, 64, 58, 39, 63, 47,\n",
" 55, 54, 48, 60, 55, 60, 65, 41, 61, 59, 65, 50, 54, 60, 48, 51, 68,\n",
" 52, 51, 61, 57, 49, 51, 62, 63, 59, 62, 54, 59, 46, 64, 49, 61])"
]
},
"execution_count": 67,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"print leftSide # shell diameters on left side of pier\n",
"print rightSide # on right side\n",
"print len(leftSide), len(rightSide)\n",
"shellDiameterDataArray = np.array(leftSide + rightSide) # concatenate as np.array, observed data\n",
"shellDiameterDataArray\n"
]
},
{
"cell_type": "code",
"execution_count": 68,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[52 54 60 60 54 47 57 58 61 57 50 60 60 60 62 44 55 58 55 60 59 65 59 63\n",
" 51 61 62 61 60 61 65 43 59 58 67 56 64 47 64 60 55 58 41 53 61 60 49 48\n",
" 47 42 50 58 48 59 55 59 50 47 47 33 51 61 61 52 62 64 64 47 58 58 61 50\n",
" 55 47 39 59 64 63 63 62 64 61 50 62 61 65 62 66 60 59 58 58 60 59 61 55\n",
" 55 62 51 61 49 52 59 60 66 50 59 64 64 62 60 65 44 58 63]\n",
"56.82608695652174\n",
"[58 54 60 55 56 44 60 52 57 58 61 66 56 59 49 48 69 66 49 72 49 50 59 59\n",
" 59 66 62 44 49 40 59 55 61 51 62 52 63 39 63 52 62 49 48 65 68 45 63 58\n",
" 55 56 55 57 34 64 66 54 65 61 56 57 59 58 62 58 40 43 62 59 64 64 65 65\n",
" 59 64 63 65 62 61 47 59 63 44 43 59 67 64 60 62 64 65 59 55 38 57 61 52\n",
" 61 61 60 34 62 64 58 39 63 47 55 54 48 60 55 60 65 41 61 59 65 50 54 60\n",
" 48 51 68 52 51 61 57 49 51 62 63 59 62 54 59 46 64 49 61]\n",
"56.66187050359712\n"
]
}
],
"source": [
"print shellDiameterDataArray[0:115] # left diameters\n",
"print shellDiameterDataArray[0:115].mean() # mean of left side\n",
"print shellDiameterDataArray[115:115+139] # right diameters\n",
"print shellDiameterDataArray[115:115+139].mean() # mean of right side"
]
},
{
"cell_type": "code",
"execution_count": 69,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.16421645292462017"
]
},
"execution_count": 69,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"observedTestStat = np.abs(shellDiameterDataArray[0:115].mean() - shellDiameterDataArray[115:115+139].mean())\n",
"observedTestStat\n"
]
},
{
"cell_type": "code",
"execution_count": 70,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.16421645292462017"
]
},
"execution_count": 70,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# use the right lamba expression for the statistic of interest\n",
"TestStat = \\\n",
" lambda dataArray : np.abs(dataArray[0:115].mean() - dataArray[115:115+139].mean())\n",
"observedTestStat = TestStat(shellDiameterDataArray)\n",
"observedTestStat "
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([64, 61, 44, 49, 47, 60, 49, 49, 48, 66, 61, 34, 58, 56, 44, 50, 68,\n",
" 59, 54, 54, 60, 41, 51, 40, 65, 61, 62, 64, 61, 65, 55, 57, 61, 51,\n",
" 64, 48, 64, 55, 59, 47, 49, 47, 59, 54, 59, 50, 58, 43, 64, 60, 48,\n",
" 63, 42, 59, 64, 63, 58, 62, 65, 61, 59, 52, 50, 59, 55, 59, 51, 50,\n",
" 49, 65, 59, 59, 60, 58, 69, 58, 57, 56, 63, 64, 43, 47, 58, 59, 61,\n",
" 48, 54, 62, 47, 59, 55, 64, 61, 55, 55, 55, 33, 52, 60, 55, 65, 58,\n",
" 39, 61, 38, 60, 60, 64, 60, 52, 63, 39, 56, 61, 60, 63, 72, 54, 62,\n",
" 64, 66, 60, 58, 62, 61, 58, 47, 56, 66, 47, 59, 65, 59, 58, 55, 47,\n",
" 61, 50, 62, 60, 60, 40, 51, 49, 59, 60, 43, 49, 46, 61, 57, 61, 62,\n",
" 60, 63, 59, 50, 55, 63, 57, 51, 57, 61, 59, 54, 48, 44, 65, 39, 62,\n",
" 64, 41, 64, 62, 63, 61, 64, 59, 58, 57, 58, 59, 54, 61, 60, 60, 66,\n",
" 62, 62, 62, 48, 49, 62, 52, 44, 51, 64, 52, 34, 59, 45, 65, 67, 58,\n",
" 66, 63, 52, 67, 61, 64, 52, 56, 60, 68, 55, 63, 49, 55, 59, 55, 58,\n",
" 55, 44, 62, 59, 50, 65, 60, 59, 65, 63, 66, 47, 61, 62, 58, 62, 64,\n",
" 53, 65, 61, 52, 57, 60, 50, 61, 58, 61, 62, 65, 59, 62, 51, 60])"
]
},
"execution_count": 71,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.array(sample(shellDiameterDataArray,115+139)) \n",
"# resample from concatenated data to bootstrap TestStat under H_0"
]
},
{
"cell_type": "code",
"execution_count": 74,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"observed Test Statistics = 0.16421645292462017\n",
"The inner (1 - 0.0500000000000000 ) percentile based Confidence Interval for the statistic T = \n",
" [ 0.012865186111978313,1.8708820769471317 ]\n"
]
}
],
"source": [
"alpha=0.05\n",
"B=100 # number of bootstrap samples for estimating p-value in a permutation test / or bootstrapped as per 'sample'\n",
"\n",
"bootstrappedTestStats=[]\n",
"for i in range(B):\n",
" # sample from concatenated data to bootstrap or permute as per 'sample(...)' function's behaviour\n",
" permutedBootstrappedData = np.array(sample(shellDiameterDataArray,115+139)) \n",
" #print i, permutedBootstrappedData\n",
" bootstrappedTestStats.append(TestStat(permutedBootstrappedData))\n",
" \n",
"bootstrappedTestStatsArray=np.array(bootstrappedTestStats)\n",
"\n",
"print \"observed Test Statistics = \", observedTestStat\n",
"#print bootstrappedTestStats\n",
"\n",
"alpaAsPercentage=alpha*100.0\n",
"lowerBootstrap1MinusAlphaCIForStatisticT = np.percentile(bootstrappedTestStats,alpaAsPercentage/2)\n",
"upperBootstrap1MinusAlphaCIForStatisticT = np.percentile(bootstrappedTestStats,100-alpaAsPercentage/2)\n",
"\n",
"print \"The inner (1 - \"+str(alpha)+\" ) percentile based Confidence Interval for the statistic T = \"\n",
"print \" [ \"+str(lowerBootstrap1MinusAlphaCIForStatisticT)+\",\" +str(upperBootstrap1MinusAlphaCIForStatisticT)+\" ]\"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### So do we reject the Null Hypothesis that the diameters were the same on either side of the New Brighton Pier?\n",
"\n",
"To answer this question we simply need to see whether the observed Test Statistic of the absolute difference between the sample means between the left and right side of the pier lies:\n",
" - inside the 95% permuted/bootstrapped confidence interval?\n",
" - or outside?\n",
" \n",
"We can conclude that there is no evidence to reject the null hypothesis because $0.164216452925 \\in [0.05, 0.82]$. And therefore the inviduals of *Dosinia anus* have the same distribution on either side of the Pier.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "SageMath 8.7",
"language": "",
"name": "sagemath"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.15"
},
"lx_course_instance": "Summer 2019",
"lx_course_name": "Introduction to Data Science: A Comp-Math-Stat Approach",
"lx_course_number": "YOIYUI001"
},
"nbformat": 4,
"nbformat_minor": 2
}
\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Nonparametric Hypothesis Testing \n",
"## Are two samples from the same distribution or not?\n",
"\n",
"What if we are not interested in estimating $F^*$ itself, but we are interested in scientificially investigating whether two distributions are the same. For example, perhaps, whether the distribution of earthquake magnitudes was the same in April as it was in March or whether human subjects under a clinical trial are reacting differently two a new drug to treat some disease (typically you have a control group that does not get the treatment and the treated group).\n",
"\n",
"You already have the means to do by using a Wald test of say the difference in sample means of the two samples, or correlation coefficient, etc for parametric or nonparametric experiments.\n",
"\n",
"We will next see a useful nonparametric approach to this problem.\n",
"\n",
"\n",
"## Permutation Testing\n",
"\n",
"A Permuation Test is a **non-parametric exact** method for testing whether two distributions are the same based on samples from each of them. In industry analogs and variants of permutation testing is known as *A/B Testing* in industry today.\n",
"\n",
"What do we mean by \"non-parametric exact\"? It is non-parametric because we do not impose any parametric assumptions. It is exact because it works for any sample size.\n",
"\n",
"Formally, we suppose that: \n",
"$$ X_1,X_2,\\ldots,X_m \\overset{IID}{\\sim} F^* \\quad \\text{and} \\quad X_{m+1}, X_{m+2},\\ldots,X_{m+n} \\overset{IID}{\\sim} G^* \\enspace , $$\n",
"are two sets of independent samples where the possibly unknown DFs \n",
"$F^*,\\,G^* \\in \\{ \\text{all DFs} \\}$.\n",
"\n",
"(Notice that we have written it so that the subscripts on the $X$s run from 1 to $m+n$.)\n",
"\n",
"Now, consider the following hypothesis test: \n",
"$$H_0: F^*=G^* \\quad \\text{versus} \\quad H_1: F^* \\neq G^* \\enspace . $$\n",
"\n",
"Our test statistic uses the observations in both both samples. We want a test statistic that is a sensible one for the test, i.e., will be large when when $F^*$ is 'too different' from $G^*$\n",
"\n",
"So, let our test statistic $T(X_1,\\ldots,X_m,X_{m+1},\\ldots,X_{m+n})$ be say: \n",
"$$\n",
"T:=T(X_1,\\ldots,X_m,X_{m+1},\\ldots,X_{m+n})= \\text{abs} \\left( \\frac{1}{m} \\sum_{i=1}^m X_i - \\frac{1}{n} \\sum_{i=m+1}^n X_i \\right) \\enspace .\n",
"$$\n",
"\n",
"(In words, we have chosen a test statistic that is the absolute value of the difference in the sample means. Note the limitation of this: if $F^*$ and $G^*$ have the same mean but different variances, our test statistic $T$ will not be large.)\n",
"\n",
"Then the idea of a permutation test is as follows:\n",
"\n",
"- Let $N:=m+n$ be the pooled sample size and consider all $N!$ permutations of the observed data $x_{obs}:=(x_1,x_2,\\ldots,x_m,x_{m+1},x_{m+2},\\ldots,x_{m+n})$.\n",
"- For each permutation of the data compute the statistic $T(\\text{permuted data } x)$ and denote these $N!$ values of $T$ by $t_1,t_2,\\ldots,t_{N!}$.\n",
"- Under $H_0: X_1,\\ldots,X_m,X_{m+1},\\ldots,X_{m+n} \\overset{IID}{\\sim}F^*=G^*$, each of the permutations of $x= (x_1,x_2,\\ldots,x_m,x_{m+1},x_{m+2},\\ldots,x_{m+n})$ has the same joint probability $\\prod_{i=1}^{m+n} f(x_i)$, where $f(x_i)$ is the density function corresponding to $F^*=G^*$, $f(x_i)=dF(x_i)=dG(x_i)$. \n",
"- Therefore, the transformation of the data by our statistic $T$ also has the same probability over the values of $T$, namely $\\{t_1,t_2,\\ldots,t_{N!}\\}$. Let $\\mathbf{P}_0$ be this permutation distribution under the null hypothesis. $\\mathbf{P}_0$ is discrete and uniform over $\\{t_1,t_2,\\ldots,t_{N!}\\}$. \n",
"- Let $t_{obs} := T(x_{obs})$ be the observed value of the test statistic.\n",
"- Assuming we reject $H_0$ when $T$ is large, the P-value = $\\mathbf{P}_0 \\left( T \\geq t_{obs} \\right)$\n",
"- Saying that $\\mathbf{P}_0$ is discrete and uniform over $\\{t_1, t_2, \\ldots, t_{N!}\\}$ says that each possible permutation has an equal probabability of occuring (under the null hypothesis). There are $N!$ possible permutations and so the probability of any individual permutation is $\\frac{1}{N!}$\n",
"\n",
"$$\n",
"\\text{P-value} = \\mathbf{P}_0 \\left( T \\geq t_{obs} \\right) = \\frac{1}{N!} \\left( \\sum_{j=1}^{N!} \\mathbf{1} (t_j \\geq t_{obs}) \\right), \\qquad \\mathbf{1} (t_j \\geq t_{obs}) = \\begin{cases} 1 & \\text{if } \\quad t_j \\geq t_{obs} \\\\ 0 & \\text{otherwise} \\end{cases}\n",
"$$\n",
"\n",
"This will make more sense if we look at some real data. \n",
"\n",
"## Permutation Testing with Shell Data\n",
"\n",
"In 2008, Guo Yaozong and Chen Shun collected data on the diameters of coarse venus shells from New Brighton beach for a course project. They recorded the diameters for two samples of shells, one from each side of the New Brighton Pier. The data is given in the following two cells.\n",
"\n",
"*NOTE - Real Data for Really Applied Statistics*: Under my guidance and assistance, students collected the shells on either side of the pier, cleaned the shells with warm soap water, rinsed and and treated with a diluted chlorine solution, used a taxonomy guide to identify the mollusc species and classified them manually. Then they focused on *Dosinia anus* or coarse venus shells. After the diluted chlorine treatment to disinfect microflora/fauna, the shells were washed, dried, placed on a graph paper, etched around the base with a mechanical pencil, and finally measured with a scale for the largest diameter within the outline."
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [],
"source": [
"leftSide = [52, 54, 60, 60, 54, 47, 57, 58, 61, 57, 50, 60, 60, 60, 62, 44, 55, 58, 55,\\\n",
" 60, 59, 65, 59, 63, 51, 61, 62, 61, 60, 61, 65, 43, 59, 58, 67, 56, 64, 47,\\\n",
" 64, 60, 55, 58, 41, 53, 61, 60, 49, 48, 47, 42, 50, 58, 48, 59, 55, 59, 50, \\\n",
" 47, 47, 33, 51, 61, 61, 52, 62, 64, 64, 47, 58, 58, 61, 50, 55, 47, 39, 59,\\\n",
" 64, 63, 63, 62, 64, 61, 50, 62, 61, 65, 62, 66, 60, 59, 58, 58, 60, 59, 61,\\\n",
" 55, 55, 62, 51, 61, 49, 52, 59, 60, 66, 50, 59, 64, 64, 62, 60, 65, 44, 58, 63]"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [],
"source": [
"rightSide = [58, 54, 60, 55, 56, 44, 60, 52, 57, 58, 61, 66, 56, 59, 49, 48, 69, 66, 49,\\\n",
" 72, 49, 50, 59, 59, 59, 66, 62, 44, 49, 40, 59, 55, 61, 51, 62, 52, 63, 39,\\\n",
" 63, 52, 62, 49, 48, 65, 68, 45, 63, 58, 55, 56, 55, 57, 34, 64, 66, 54, 65,\\\n",
" 61, 56, 57, 59, 58, 62, 58, 40, 43, 62, 59, 64, 64, 65, 65, 59, 64, 63, 65,\\\n",
" 62, 61, 47, 59, 63, 44, 43, 59, 67, 64, 60, 62, 64, 65, 59, 55, 38, 57, 61,\\\n",
" 52, 61, 61, 60, 34, 62, 64, 58, 39, 63, 47, 55, 54, 48, 60, 55, 60, 65, 41,\\\n",
" 61, 59, 65, 50, 54, 60, 48, 51, 68, 52, 51, 61, 57, 49, 51, 62, 63, 59, 62,\\\n",
" 54, 59, 46, 64, 49, 61]"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(115, 139)"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(leftSide), len(rightSide)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$(115 + 139)!$ is a very big number. Lets start small, and take a subselection of the shell data to demonstrate the permutation test concept: the first two shells from the left of the pier and the first one from the right:"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[52, 54, 58]"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rightSub = [52, 54]\n",
"leftSub = [58]\n",
"totalSample = rightSub + leftSub\n",
"totalSample"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So now we are testing the hypotheses\n",
"\n",
"$$\\begin{array}{lcl}H_0&:& X_1,X_2,X_3 \\overset{IID}{\\sim} F^*=G^* \\\\H_1&:&X_1, X_2 \\overset{IID}{\\sim} F^*, \\,\\,X_3 \\overset{IID}{\\sim} G^*, F^* \\neq G^*\\end{array}$$ \n",
"\n",
"With the test statistic\n",
"$$\\begin{array}{lcl}T(X_1,X_2,X_3) &=& \\text{abs} \\left(\\displaystyle\\frac{1}{2}\\displaystyle\\sum_{i=1}^2X_i - \\displaystyle\\frac{1}{1}\\displaystyle\\sum_{i=2+1}^3X_i\\right) \\\\ &=&\\text{abs}\\left(\\displaystyle\\frac{X_1+ X_2}{2} - \\displaystyle\\frac{X_3}{1}\\right)\\end{array}$$\n",
"\n",
"Our observed data $x_{obs} = (x_1, x_2, x_3) = (52, 54, 58)$\n",
"\n",
"and the realisation of the test statistic for this data is $t_{obs} = \\text{abs}\\left(\\displaystyle\\frac{52+54}{2} - \\frac{58}{1}\\right) = \\text{abs}\\left(53 - 58\\right) = \\text{abs}(-5) = 5$\n",
"\n",
"Now we need to tabulate the permutations and their probabilities. There are 3! = 6 possible permutataions of three items. For larger samples, you could use the `factorial` function to calculate this:"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"6"
]
},
"execution_count": 56,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"factorial(3)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We said that under the null hypotheses (the samples have the same DF) each permutation is equally likely, so each permutation has probability $\\displaystyle\\frac{1}{6}$.\n",
"\n",
"There is a way in Python (the language under the hood in Sage), to get all the permuations of a sequence:"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[[52, 54, 58],\n",
" [52, 58, 54],\n",
" [54, 52, 58],\n",
" [54, 58, 52],\n",
" [58, 52, 54],\n",
" [58, 54, 52]]"
]
},
"execution_count": 57,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"list(Permutations(totalSample))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can tabulate the permuations, their probabilities, and the value of the test statistic that would be associated with that permutation:\n",
"\n",
"| Permutation | \n", "$t$ | \n", "$\\mathbf{P}_0(T=t)$ | \n", "
| \n", " | \n", " | Probability under Null | \n", "
| (52, 54, 58) | \n", "5 | \n", "$\\frac{1}{6}$ | \n", "
| (52, 58, 54) | \n", "1 | \n", "$\\frac{1}{6}$ | \n", "
| (54, 52, 58) | \n", "5 | \n", "$\\frac{1}{6}$ | \n", "
| (54, 58, 52) | \n", "4 | \n", "$\\frac{1}{6}$ | \n", "
| (58, 52, 54) | \n", "1 | \n", "$\\frac{1}{6}$ | \n", "
| (58, 54, 52) | \n", "4 | \n", "$\\frac{1}{6}$ | \n", "