018_LinRegIntro - sds-3.x/ScaDaMaLe

ScaDaMaLe Course site and book

Let us visit an interactive visual cognitive tool for the basics ideas in linear regression:

http://setosa.io/ev/ordinary-least-squares-regression/

Linear Regression by Hastie and Tibshirani

Chapter 3 of https://www.dataschool.io/15-hours-of-expert-machine-learning-videos/

The following video playlist is a very concise and thorough treatment of linear regression for those who have taken the 200-level linear algebra. Others can fully understand it with some effort and revisiting.

Linear Regression PlayList

Ridge regression has a Bayesian interpretation where the weights have a zero-mean Gaussian prior. See 7.5 in Murphy's Machine Learning: A Probabilistic Perspective for details.

Please take notes in mark-down if you want.

For latex math within markdown you can do the following for in-line maths: \(\mathbf{A}_{i,j} \in \mathbb{R}^1\). And to write maths in display mode do the following:

\[\mathbf{A} \in \mathbb{R}^{m \times d} \]

You will need to write such notes for your final project presentation!

Gradient Descent

Refresher: what-is-gradient-descent

Getting our hands dirty with Darren Wilkinson's blog on regression

Let's follow the exposition in:

You need to scroll down the fitst link embedded in iframe below to get to the section on Analysis of quantitative data with Descriptive statistics and Linear regression sub-sections (you can skip the earlier part that shows you how to run everything locally in spark-shell - try this on your own later).

Let's do this live now...

import breeze.stats.distributions._
def x = Gaussian(1.0,2.0).sample(10000)
val xRdd = sc.parallelize(x)

import breeze.stats.distributions._
x: IndexedSeq[Double]
xRdd: org.apache.spark.rdd.RDD[Double] = ParallelCollectionRDD[0] at parallelize at command-2971213210277124:3

println(xRdd.mean)
println(xRdd.sampleVariance)

0.9961211853107911
4.025384520599017

val xStats = xRdd.stats
xStats.mean
xStats.sampleVariance
xStats.sum

xStats: org.apache.spark.util.StatCounter = (count: 10000, mean: 0.996121, stdev: 2.006236, max: 8.326542, min: -6.840659)
res1: Double = 9961.21185310791

val x2 = Gaussian(0.0,1.0).sample(10000) // 10,000 Gaussian samples with mean 0.0 and std dev 1.0
val xx = x zip x2 // creating tuples { (x,x2)_1, ... , (x,x2)_10000}
val lp = xx map {p => 2.0*p._1 + 1.0*p._2 + 1.5} // { lp_i := (2.0*x + 1.0 * x2 + 1.5)_i, i=1,...,10000 }
val eps = Gaussian(0.0,1.0).sample(10000) // standard normal errors
val y = (lp zip eps) map (p => p._1 + p._2)
val yx = (y zip xx) map (p => (p._1, p._2._1, p._2._2))

x2: IndexedSeq[Double] = Vector(0.1510498720110515, -0.21709023201809416, -0.9420189342233366, -0.6050521762158899, 0.48771393128636853, 0.027986752952643124, 0.08206698908429294, -0.7810931162704136, -1.2057806605697985, -0.9176042669882863, 1.1959113751272468, 0.5771223111098949, -0.5030579509564024, 0.05792512611966754, 2.0492117764647917, -1.6192400586380957, -0.28568171141353893, 1.662660432138664, -0.3053683539449641, 0.3879563181609168, -0.7733054017369991, -0.39556455561853454, 1.7717311537878704, 0.33232937623812936, -0.1392031424321576, -0.5587152832826312, -1.7575839385785728, -0.6579581256900572, 2.037190332203657, 0.03462069676057432, -0.3795037160416964, 0.4622268686186779, -1.0897313199516543, -1.6880936202156007, -0.19374927120525648, -0.5509651353004006, -1.7588836379571422, -0.28930357427256465, 0.4020360571731675, -1.6301541413671794, 0.5958281089725639, 0.366330814768569, 0.4073858014158006, 0.16855406345881346, -0.947375008877488, -1.301502215739479, 0.1470275032863848, 1.0019450542117738, -0.5904174670089379, 0.5435130824204192, 0.3257024438513822, 0.7490447645980914, 0.7972538819971793, -3.008935968399453, -0.2458138862820675, 2.2893054863712985, -0.16582870751291443, -0.7663457059098656, -0.02129165082715293, -1.678393618311418, 0.4477301980923963, -1.3761630188037661, 0.8348043252317754, 2.593999513789406, 0.6054528239524932, -0.517937225538966, -0.8704495697165462, 0.7963689735065775, -0.22860472183526676, -0.8133035588543074, 1.1807570202429787, 0.09708549308780165, -0.1255029357381233, -0.23174042403046444, 1.9877912284224935, -1.6162508631493637, -0.1284467343953938, 0.715203575233169, 0.3144454735478882, -0.1848647516127121, -0.5496887866336722, -0.2545756305645813, 0.16750436775298258, -2.4673043543431854, 1.1418851052679557, 0.3748790073641352, -0.817404869860111, -0.8361798593188912, -1.5394267146879406, -0.41572364800501355, -0.1012436049109305, -1.1015184209065036, 1.8240944932710277, -0.028399443888511452, -0.8207026038892242, 0.015061100495787802, 0.18421526981224481, 1.2655676528174074, -1.057100552093993, 0.9714045895202634, 0.7411082212281875, 0.8163373740421433, -0.08103816988370252, 0.5892784701674155, 0.32505574686630534, -0.9962963394557808, 0.484895621648311, 0.9927193853136387, -1.7490403684405513, 1.6202003817264536, -0.2699908354021868, 0.9907548061408172, -0.6242833307164519, 0.4843781855746414, 0.4842034053606869, 0.7600831116212132, -0.6640803417192944, -0.5474019416456044, 0.2700255027168447, -1.3841075813504558, 1.6002816913642322, -0.7092207329917671, -0.08883062902810138, 2.025819363561132, 0.05567161231138376, -1.556067597966872, -0.5832140748557112, 0.31537969362482127, 0.015469743624534625, -1.2740442928737779, 1.3809654728074165, -2.333571571876549, 0.20354188937573037, 1.0247244424109938, -0.25757737278880144, -2.8220962107471803, 0.09738713753633461, 1.89542633873579, -2.6782503894934844, 0.7782540200985733, 0.6414117319660293, 2.264173731050526, -1.5111510345308878, 0.9287758292855764, 0.5055499928507728, 0.16205786373841113, -1.0297014434622302, -1.291377179402119, -0.40663773460337127, -1.2343888551769233, -0.5036902484765656, -0.3721487060577523, -1.987521549256808, 1.175800228965741, -0.00707441417410632, 1.0509376236793728, 0.7494475310558243, -0.38427172332612247, -0.930358252352808, 0.960228212698756, 1.0130821037354474, -0.03770027592004082, -0.9950054698782884, -1.8847718393419073, 1.2260442128311473, 0.8973362086382494, -1.1448274676767252, 1.2547096088120568, -0.9186364033184078, 0.8026832422467616, -0.4758593270796066, -0.830645491686263, -0.5400873564064651, -0.6630169496517384, 0.6381413581264455, -0.7504597023733945, -0.5222934465699843, 1.8847574147176338, 0.6145641448752742, -0.5988311192477164, -0.48271398431423773, 0.5473323912712278, -0.5885123950740869, 1.0174330673374294, 0.9167061097763491, -0.5816363296158883, 0.030959788500977675, -0.5137243911676949, -1.8709170592192632, -1.7472131497273815, -1.1974973880763593, -0.09142445477515819, 0.40624320435362116, -0.40045440389966913, -0.37854644978248697, 0.11830887334108421, 1.4637485338108103, -0.7665087390801844, -0.3760615977493943, -1.5309037473721014, -1.4651567904030773, -0.026798948253209182, 0.2066789574673153, 1.8143254569727947, 2.3406140678357765, 0.18034205783137355, -1.4479109856334291, -1.1294650286779386, 1.2932328016031633, -0.1393663274001316, -0.8468551132460095, 1.2713996676202226, -0.060721411812719187, 0.6928376952914521, 1.0579153298124913, -0.7645892788282653, -1.4263627705259183, 0.5336299350415817, 0.6573179344240204, 0.6654619137191999, -2.123504844254055, 1.4866262970106303, 0.129061989976924, 0.22269040795154907, -0.049559875163616714, -0.9450310180664842, -0.11473631908336672, -0.39245949031895117, 1.5510785871987065, 1.8254864916125728, 2.061530134034693, 0.8418470519782356, 0.06828445075170164, 1.663002872049384, 0.5626537089196723, 1.1924431746135973, 0.37430311472451033, -0.370821637491865, -0.09451085038956872, 1.8818552958789954, -0.5452487096855598, 0.9207216541161212, -0.03840103995077165, 1.3724685915375903, 0.8666584948313616, -1.1444960562771815, 0.2805555937763331, 0.7940128878027313, 0.5091140746762197, 1.5121770072162093, -0.5742888800806798, 2.3581174706724157, 1.7068949917127816, 0.4624349386051622, 0.1225204014240068, 0.9398794428978549, -0.9518050374642764, -1.0522230289227708, 0.8902191043953458, -1.0945928783869934, 0.18036570004934346, -0.8406140101582382, -0.8634555290392517, 0.7792166711410811, 1.289633532883873, -0.4501732830926528, 2.7021303462005424, -0.22262701182155162, 0.6460605038491092, -1.0247372175076013, -1.307280513522457, 1.6404006952919912, -0.0518398187177849, 0.45080433367542755, 0.3719461665317735, -1.4543187272074507, 0.10188821604602745, -0.27296335375143177, 1.2451029860749379, 0.14487479131269917, 0.18201741036674932, 0.03920775130429693, -0.2580100207138237, 1.7047536767840974, 0.29766687892421284, -1.7401156308492993, 0.05515391539227905, -0.27119832477161965, -1.0452547788609865, -1.4984850253735646, -2.079436054773939, 0.44308840473545447, 0.25444206315208995, -0.19417163889547193, -0.5363606084152478, 0.36176129341638513, -0.33671294206693936, -2.2316073845645206, 0.0455416268115695, 1.0401389018577378, -0.3141083536641634, 0.07341971359886727, 1.0284248217766079, 1.278483571961632, -0.7833209611949998, 1.3772836627732903, 0.9700373682532052, -0.8662419500657893, -1.5938783703449753, -0.6174495172524604, -0.7819620808358254, 0.10302666069195707, 1.0028767323240257, -0.10674590158973656, 0.38777338877043965, -1.0509152481698179, -0.06337185419126991, 1.4117521801837538, 0.7613291392405328, 1.0897823589924052, -0.6637558495096709, -1.8157963102001593, 1.2358784007741146, -2.15537358668726, 0.2971613832014052, -0.750050504028126, 0.8329394704642922, -1.071705174325631, -0.9018883675016863, -0.8504391140892998, -1.0078945028740045, -0.44317562320236065, 1.3250813581783045, 0.6196354624287462, -1.278621584064485, 1.9184511848237962, -0.516835702626045, 0.6357213808712543, 0.925579417321024, -0.5545984413834245, -1.3658354424121768, -0.5628777726709475, -0.04769253246241577, -0.6991542495947202, -1.1147192653783664, -0.5166063563392334, -1.7956252514175055, 1.6987454382135425, -0.43821792344869875, 1.022408572973808, 0.3336089209393924, 0.02312505623899069, -0.31233039785382466, -0.824119195846109, 0.13191861079694536, -0.6036888443048509, 0.11668453965551463, -1.2724302084947103, 0.28050518201340424, -1.0824241957346425, -1.1326924749331988, 0.5172393559288399, -0.7220647087293313, -1.2436289736107498, -0.6156240117132562, 1.0406128654122049, -0.6297102775960882, 0.012206672359322898, 0.4525998876038395, -1.0451363444854351, -1.3734168129137967, 0.08679497767122704, -2.04474580239617, -0.19771649973564123, -1.3186299012886857, -0.2761375653147074, 0.5741590875530609, 0.9593237124668771, 1.0961298140479905, 0.4391578691796409, -0.6238036279250703, -0.05430692986877771, -0.11640313781962018, 2.0002028697055154, 0.7350624069633737, -0.17859693817047373, -2.314523014865333, 0.22960402078570183, 0.5724080038056716, -0.07916244301664163, -0.5341143586242104, 1.6460207223980545, 0.3137460006226553, -0.9207927088314583, 0.6967661703270028, 0.5571396449070067, 0.31796421714524625, -0.2228633316357967, -0.5499817903565626, -0.7841758341511148, -0.02401453955427877, -2.3166764321273394, 1.0391140336549198, 0.5498488089121792, 1.2335108771359133, -1.9943985764799559, 0.042171455201203426, -0.7485156690697342, -0.9570963024254994, 1.301620771050323, 0.04031728339000787, -0.6859845801505859, 1.3911313622090693, -1.6605547611122933, 0.6699454108732157, -1.1570517171606756, -0.36244860379594424, -0.09907031133103263, 0.19148521269498425, 0.8267773040085179, -1.3644874947043841, -1.197567110439719, 1.4817262828203837, -0.10843051264695289, 0.08108871807685476, 1.3545724635606313, 0.5100719528278953, 0.057285477365546435, -0.37191016428905793, 0.18281213200485982, -0.8842932261652275, -1.6494429918726925, -0.7764264486783047, -0.3042211935492598, -0.020825036678536985, -0.30368381843937947, 0.7800891562758471, -0.0735325489432372, -0.1679937736476282, -1.79719393467339, 1.0755228066035305, -0.15773836735128086, -0.0783402653348485, -1.719143525921516, 0.8435175129296529, 0.6761343445136715, 0.5244103817082383, 0.8620143952438806, -2.3060336651929445, 1.4304037697137544, -0.7759942408828852, -1.2609787369882242, -1.4478365748347117, 2.0278952751593384, -0.7861955769834937, -0.2479198127035199, -0.6571378828960646, -2.1673373348092366, -1.7499456634223678, 0.19663998656853085, 0.23504549724402013, 0.3638239098400113, -1.3775094864583974, -0.020668102118018273, 0.25664562550595316, 0.28318073789879256, -0.29571052367446293, 0.7681213080895282, 1.5610874049559282, -2.622344889120145, -0.8202101196954132, 0.7748393413408604, 0.508724216091097, -0.3751101664710589, -0.9591972717671285, 0.925165234056263, 1.8459965937810827, 0.27342245677072646, -2.4602979491815096, 0.6963477323668592, 2.167550027845107, -0.19332822247961037, 0.5981380076459031, -0.694964755780304, 0.08919555493105245, -2.0804801088705767, 1.1920101085447832, 1.231750346524178, -1.455653407187848, 1.2328411389316813, -0.39573709387712014, -1.1956194147671937, -0.04302633139042216, 2.0750703846663137, -1.7646202495106422, -0.017144573753570983, 0.1715206676563901, -0.9656202470043257, -1.2648785618794656, -1.3357866485679637, 0.1818276074213889, -0.1742396893961462, -0.6767070415199365, 2.137638188136061, 0.4838102855828845, 0.3075626385768448, -1.3633563499769654, 0.02908534679907131, 2.321245009450074, -0.037890984508530186, -1.3762801283466828, 1.6125793622774343, 0.201560934160273, -0.15731271258242444, 0.6015468376832578, -0.0991480003326356, 0.9603378027154168, 0.5407700164856909, -0.18116162969719404, 1.515039231410728, 1.0516518441747622, -1.108150706054602, -0.0902162047955897, 0.525636626469496, -0.6505154458390053, -0.6385600015638014, -0.03765503579803463, -0.8067465597676813, 0.25298188952694345, -1.2484755253881343, 0.5907884315290602, -0.7407171504354528, 0.09342389392065106, -0.6358631568411891, -0.7990035391904377, 0.7318639477491172, 1.0721139408861649, -0.7709696933298915, 1.4262689270118047, 0.008052693380149429, -0.7675924241217212, -0.875355770610139, 0.3318552144813144, 0.47230162167225975, -0.28845007593399613, 0.33674053005607557, -1.2685836846998517, 0.9973713512813284, -0.14095584542177128, 0.6434211266970435, -1.3166935231718915, 0.5010874593219053, 0.5400322253256449, 0.1131615290866549, 1.1948743843705798, -0.4068064171361299, -0.2145531753251343, -0.8414414845064078, -0.17448661922809763, 0.015039956542559319, -2.00225473143416, 0.7768533690748148, 0.7280781053306761, -0.6814357844622568, 2.4398143015234925, -1.9715384636473667, -1.089089720589046, -1.0771030789727833, -1.1187887124853761, 0.26304747184316635, 0.6104069218328998, 0.814212432670669, -1.1311163095796626, -0.715657837824097, 0.11337791560485909, 1.1903117127474667, 0.8655118045637775, -0.4786300318354617, 1.1374241148097717, 1.3574248394663742, -0.59796895966422, 0.5787074373907313, -1.4837192620796162, 0.15009771866713956, -1.0785198355458125, 0.43654174540174034, 1.367374206941478, 1.1624833124548457, 1.4378845237950593, -0.4752367281036082, -0.6295231876858187, 0.13609988490474167, 1.451172421854568, -0.9690521636819287, -0.4657652028114279, -0.3272245138051527, -0.3254730682226242, -1.1968451698216205, -2.374605096166881, 1.4231065460066927, 0.3339627050188421, -0.37529586334821136, 1.7121288810552737, 0.7740341080806812, -0.08101997848273264, 0.8171553719668059, 1.004037394973044, 0.22996069331921015, -0.135350519278533, -0.4727403129550938, -0.1182151974084787, -1.9468945045910213, 0.4579087802822954, 1.3836409234546432, 0.02325932498935564, 0.475495388508411, -0.37698432805620236, -0.8219848732487407, -1.3951694203634668, 0.0484490852454009, 0.9222291689334502, 1.5679464498188418, 0.810147143385619, 1.9186942393210789, -0.2176761309409645, 1.497682278641026, -0.6745280698091511, -0.14287398549911423, -0.6123313834457077, 0.5533479535312111, 0.17525285583058028, 0.26417401560092896, 0.7989612854266647, -0.6531854416899909, 0.3890137678570137, 1.5942398929174109, 0.29843502966443813, 0.28270288452527464, -1.030296515196584, 0.29004278605641326, 1.1839066433022867, 0.5617980143776449, 0.3177318190024553, -0.17078152902760071, -0.4129031279462113, 1.6509715706618235, -0.27367425727559463, 1.2450208243939591, 1.4063428988232145, 1.6639488103943056, 0.4109699563209272, -2.1509670233572784, 0.2639495116581482, -0.9708523309574577, 1.2588123216998128, -0.6395148239066109, -2.3731627585033115, 0.43524872907522616, 0.6328073362083871, -0.2381862432808314, 0.40492896291344244, -1.0527840343286374, 1.0689442908821987, 0.789255581937264, 1.1724902795494772, 1.2893390188349818, 0.8926054903089587, 1.4246842661952515, 0.21910907895561132, 0.34689263334132264, -0.5771627038699307, -0.04827308490054824, -0.8361201294738408, 0.2776346366703466, -2.583216199463966, -0.5754336242709439, -0.4740298729504418, 0.45528050967309863, -0.937001723779216, 0.12442585999261868, -1.037597020742512, 0.6656497732259127, 0.09718535082910094, 0.472044663569549, 0.01365815985041524, 0.21443946294874028, 0.38540036725618015, 0.7807770362857364, 0.12940125757456059, -0.9556204076037894, -1.0007190699745476, -1.4238233782457508, 0.09834207753278863, -2.5447136959911605, 0.1997331013963162, 1.2806574080505437, 0.9315316175761184, -0.6394107818742375, -1.4099227738716835, -0.21682467720141205, -0.2142482857139384, 1.1299353191945833, 0.26409499420187055, -0.6837618328677663, 1.748176871520828, -0.45137750181529596, 0.2545934586572956, 0.6078542443352412, -0.21436291927129, -0.01807798719673732, 0.9995581200333848, -1.9002339898589569, -0.2693745536442081, 0.27738326061401086, 0.9588088083048624, -2.2809380920777813, 0.38515662164152703, 0.14815918760114763, 0.21892066892130446, 1.2291414798572955, -0.7027138126400151, 0.001101516410189726, -0.19878057656460413, -1.3484820015025956, -1.3475429219400465, -2.0210923601428465, -0.09144126594108812, 0.4059005816276273, 0.9335476898909668, -0.17037852812849555, 1.1797576294046501, -0.9814138435948395, -0.59577026888633, -0.946118594015114, 0.31024225601384786, -0.36715612264555036, -0.9129082906750721, 0.24118225941129226, 1.6953067460054017, -0.44575419487658474, 0.8440499307509399, -0.6987818896793766, -1.6504422204189202, 0.9076290994167842, -0.5720216124887916, -0.3907622872528834, -0.5344435450123501, 0.2434291400674394, -1.3828080206005129, -0.40522439460185156, -0.9329831282017179, 1.5016477371243226, -0.4782955772741999, -0.35933071932718297, -0.013071727079844811, 0.9457197761151667, 0.88547572536491, -0.7284350484475177, 3.34597265464979, -0.6565486096719748, 2.0071882494870747, 1.0595021575719632, -0.6347598710491432, 1.7780817104271445, -0.7474659036256496, 2.274681811945376, -0.16661804033866273, 1.229846682825196, -0.24202493293949087, 1.3043974875572906, 0.6542233003162868, 1.1297955530475419, -0.6421340260679929, 0.6218989933034942, 0.41508966549094883, 0.19856384065980512, -0.9782335050516806, 1.5826472247462247, 0.17887593907764898, -1.3498741915231427, -1.9667878392676454, 3.305663883668763, -0.4350440334262082, -0.4097682527976977, 0.9434336763573686, -0.5436220786767807, -0.018197193698519923, -0.8423313162472719, -0.5537406141818406, -1.3445810021139153, 0.5739597321896511, 0.9027226090601936, 0.9962807601235416, -0.3944869331359751, -0.49948645872557595, -1.592644252527489, -0.8963075632556161, -0.281124928867554, -0.38290172562814223, -0.08623070871608174, 0.8689176122759706, -0.6022076534932924, 0.5023469816085081, -0.696239800321511, 0.9822179345376554, 0.9843867592827164, 1.253737442052706, -0.21561036112503276, 0.4763897291185735, -0.3816621678778997, -1.2726727941938998, 0.7808462914630361, -1.279321770813609, -1.249085767708575, -0.5262007219347724, 2.0375551711803523, 0.35565466738690976, 1.729140884257809, -0.2134341316049298, 1.0661506262996305, 0.14405639612841104, 1.5283790176593852, 1.083286397653919, 0.08705656468813867, 0.12354297653909227, -0.8885387288040731, 0.868850539965121, 0.38468690199897226, 0.10529276620539245, -1.5467019468693028, 0.11134848558121198, 0.5242581436594406, -1.4612754791339186, 0.6170671476298276, 2.404874088438192, 0.4536536350335747, 0.5590316578013409, 1.4722471502737542, -0.7826074583703301, 0.03864057928741479, 0.5409052205287742, -0.91764558020482, 0.2907284432003232, 0.0926246944836777, -0.6715464756911873, 1.6188813415483425, 1.937486011066117, -1.2872213802790935, 0.05028238793779849, 0.34278782978677325, -0.9317921756615996, 0.847157773949496, -1.7309498009428164, 1.131234684816195, 1.42559703424871, -1.1042663027995043, 0.5482757432406922, 0.49093316226965256, 1.4299765283373023, 1.2699610859231685, -0.6885892269356135, -0.6834163635059881, 0.09970998500665039, -0.05252250656608228, -0.5738061460548269, 0.04684493779144785, -2.0403431803564103, -2.3426040850283334, 1.3263987617975324, -1.6066813971687723, -1.3712077945231729, 0.023194145470648, 2.0233789284442305, 0.0552515096510362, -0.1848702031009112, 0.8678035694514004, 2.584641397416784, -1.374802793486839, -0.41275499089901185, 1.757248960282745, -0.9174957372175073, 0.8454219752897207, -0.48424938225642095, -0.9452258787410663, -0.4063011961676097, -0.14610880161061304, -1.191656301348928, -0.05534876749708337, -1.0824577801299335, 0.5065919371099356, -0.7234007146985317, -0.4393685486065214, -0.28713403102892043, 0.4023446222345482, 0.7408646840398053, 1.1735145587625548, -1.4959369385460655, 0.9569777537825604, -1.5309823669985632, -0.32832346606728907, -1.9702457066985795, 0.4915363188477118, 0.6267936755653063, 0.5264923978293093, 0.2793517169291047, 0.699630827386399, -0.48194010545572347, -1.9163915803321179, 0.4917274635159007, -0.8918023989162509, 0.054382578076211927, -0.13063575946183192, -2.963221181278889, -1.930107473646217, 0.27434769083662114, -2.5862229219736177, 0.12021993055087092, -0.6499875101530838, 1.2841140434979073, 1.795232260277804, 0.5983161934190495, 0.14748215452570676, 0.012732674940682266, 1.0596508622852832, 0.2612567028003964, -1.048184716657201, 1.2910337044306555, -0.40112015879164387, -2.474368538289194, 0.3607932548010173, -0.39277746077266656, 0.43117281807232705, -0.780462370732576, 0.004212935254872857, 1.1794924506297337, -0.11782146771505461, -0.9169817156798284, 0.6436607218522247, 1.009270297159163, 1.1191953425233845, -1.3185250855422, -0.06390167874291222, 1.0043732007738526, 0.8901659700461146, -0.09710628953572964, -1.7985358888300125, -0.4017184943845876, -0.49366231492607787, 0.47104211065537627, -0.8011138987145442, -0.5644297030299493, -1.2117395349774904, 1.7752801442541137, -0.06040866197475744, 0.5856200577072064, -2.3049530233040403, -1.2233947361735646, 1.789286715947381, -0.5678735891299862, 0.2884538364881288, 0.596466411038459, 0.1257010162863289, 0.08710239368087719, -0.3976608617928727, 0.03973466759058689, 1.2929183222053455, 0.2995297455403365, -0.4712635910013849, -1.404574175177837, 1.6770901706769825, 2.0017133748539893, 0.1274278689589807, 1.3587051209065415, 0.24268281140561118, -1.4595561645069262, 0.6652637699287584, -1.0625231499800574, -0.5489552263804611, 0.011217276797261928, -0.4537248029565897, 0.5114095357826577, 1.110198956335098, 0.417931247837064, -0.5197879191519639, 0.8908423822038136, -0.05428396733277234, -1.6971217632848712, -0.27867393472168944, 0.9007900540298039, -0.20140571160773263, -0.05687271801962116, -0.3743529665542792, -0.5108114271524771, -0.227006519061247, 0.12787523499272155, -0.303674313770039, 0.21916688186617306, 1.5129638774909924, 0.5287700042827598, -0.7887173081259768)
xx: IndexedSeq[(Double, Double)] = Vector((1.6983876233282715,0.1510498720110515), (1.9306937118935266,-0.21709023201809416), (-3.613579039093624,-0.9420189342233366), (1.4774128627063305,-0.6050521762158899), (0.49536920319327216,0.48771393128636853), (2.619266385545484,0.027986752952643124), (-1.6216138344745752,0.08206698908429294), (1.2511380715559544,-0.7810931162704136), (-0.6891022233658615,-1.2057806605697985), (2.7890847095863527,-0.9176042669882863), (-0.8384317817611013,1.1959113751272468), (-0.39702300426389736,0.5771223111098949), (-0.20341459413042506,-0.5030579509564024), (1.940354690386933,0.05792512611966754), (3.1686700336304794,2.0492117764647917), (-0.015718251511384507,-1.6192400586380957), (3.5222633033548227,-0.28568171141353893), (-1.7700416261712641,1.662660432138664), (-1.276616299487233,-0.3053683539449641), (2.847017612404211,0.3879563181609168), (-2.2095588075327264,-0.7733054017369991), (0.12398381308330297,-0.39556455561853454), (-0.889038019080473,1.7717311537878704), (1.2682927214864042,0.33232937623812936), (-1.5242980017423502,-0.1392031424321576), (-1.1047188228499718,-0.5587152832826312), (1.7722204452187196,-1.7575839385785728), (0.04169431488122677,-0.6579581256900572), (0.008442861000104274,2.037190332203657), (2.129861454185579,0.03462069676057432), (-1.3075615522200312,-0.3795037160416964), (2.640226790983577,0.4622268686186779), (1.8115120733853414,-1.0897313199516543), (-2.4269557944745266,-1.6880936202156007), (0.25161494380013494,-0.19374927120525648), (0.6972213441690793,-0.5509651353004006), (-1.4249662489137846,-1.7588836379571422), (-2.8058490335385198,-0.28930357427256465), (3.8366909938243237,0.4020360571731675), (3.110632056495419,-1.6301541413671794), (3.7294439220903084,0.5958281089725639), (-0.003590308691806099,0.366330814768569), (0.6670448231750639,0.4073858014158006), (0.7077886272690928,0.16855406345881346), (4.810852253296299,-0.947375008877488), (1.537026192682279,-1.301502215739479), (1.5778035760893996,0.1470275032863848), (0.8034872067976389,1.0019450542117738), (2.1781007920189275,-0.5904174670089379), (2.1582500596745224,0.5435130824204192), (1.5702056724507172,0.3257024438513822), (0.6273909939411992,0.7490447645980914), (2.9737351998521557,0.7972538819971793), (0.19226769228505514,-3.008935968399453), (0.7040906058297841,-0.2458138862820675), (1.7748793746647895,2.2893054863712985), (0.817755138729894,-0.16582870751291443), (5.098321911797599,-0.7663457059098656), (3.72224257935207,-0.02129165082715293), (2.819825663241845,-1.678393618311418), (-1.0541007736252888,0.4477301980923963), (-1.9931816211815754,-1.3761630188037661), (2.053200357360071,0.8348043252317754), (2.6504828464040466,2.593999513789406), (0.7635959419715312,0.6054528239524932), (-0.6985217947431726,-0.517937225538966), (1.0200787394354123,-0.8704495697165462), (0.9900523144691149,0.7963689735065775), (-2.636058289151889,-0.22860472183526676), (-0.006865612686181466,-0.8133035588543074), (-1.9369303943736362,1.1807570202429787), (0.8207830907828872,0.09708549308780165), (2.9960365323274516,-0.1255029357381233), (-1.4598715511713443,-0.23174042403046444), (-1.6850847827294002,1.9877912284224935), (1.8660345919208075,-1.6162508631493637), (3.4665378645024973,-0.1284467343953938), (-0.5413207856162352,0.715203575233169), (2.0812024732066883,0.3144454735478882), (1.0478641099166173,-0.1848647516127121), (0.09764242478884722,-0.5496887866336722), (2.455169809100939,-0.2545756305645813), (3.929354758370917,0.16750436775298258), (3.050688866390247,-2.4673043543431854), (5.053919984936024,1.1418851052679557), (0.6709281732489789,0.3748790073641352), (0.4881210472750994,-0.817404869860111), (3.6510567194061876,-0.8361798593188912), (1.146503253972402,-1.5394267146879406), (-1.255812790663824,-0.41572364800501355), (0.5596948430990623,-0.1012436049109305), (0.6787285726710425,-1.1015184209065036), (1.5295781767138816,1.8240944932710277), (0.9792294583717878,-0.028399443888511452), (2.5772503164411127,-0.8207026038892242), (3.5067014361379223,0.015061100495787802), (1.0530086719680942,0.18421526981224481), (3.1546436554171695,1.2655676528174074), (2.5666446000164393,-1.057100552093993), (-0.25408644443838413,0.9714045895202634), (5.094415707144548,0.7411082212281875), (-0.5842186502529905,0.8163373740421433), (0.27046894035259084,-0.08103816988370252), (-0.1900912387832303,0.5892784701674155), (0.19119860102738795,0.32505574686630534), (1.342435075277992,-0.9962963394557808), (2.01576225946382,0.484895621648311), (-1.3315732630965176,0.9927193853136387), (4.651801823771725,-1.7490403684405513), (0.40875990335626267,1.6202003817264536), (-0.6688861808540578,-0.2699908354021868), (-1.2112344110158406,0.9907548061408172), (-0.273739139868751,-0.6242833307164519), (0.008955401325448697,0.4843781855746414), (-0.4364876411254188,0.4842034053606869), (-0.5943099209874005,0.7600831116212132), (1.211946845667774,-0.6640803417192944), (1.5083366446138549,-0.5474019416456044), (0.31043238086970926,0.2700255027168447), (-0.026444982031816222,-1.3841075813504558), (5.594258068053623,1.6002816913642322), (1.7386180400754943,-0.7092207329917671), (3.234629000567968,-0.08883062902810138), (0.07317713975488005,2.025819363561132), (-1.7615858067511803,0.05567161231138376), (2.903921478619658,-1.556067597966872), (2.1095663416536077,-0.5832140748557112), (0.8984342190126537,0.31537969362482127), (-1.2562548556289594,0.015469743624534625), (0.70371421222917,-1.2740442928737779), (6.682663984938959,1.3809654728074165), (2.8144107440089066,-2.333571571876549), (-0.8992433388942302,0.20354188937573037), (3.472636066658491,1.0247244424109938), (1.6891867366656383,-0.25757737278880144), (2.5374071736508563,-2.8220962107471803), (4.690232337846138,0.09738713753633461), (0.12067267908807089,1.89542633873579), (2.7169593848024887,-2.6782503894934844), (1.8062659512087649,0.7782540200985733), (-0.01845293657257696,0.6414117319660293), (-0.4462823816579564,2.264173731050526), (1.7946315433961588,-1.5111510345308878), (-0.4798514257428741,0.9287758292855764), (-1.2555272864460045,0.5055499928507728), (0.4590668877631503,0.16205786373841113), (1.3155977358341158,-1.0297014434622302), (2.9623034849030443,-1.291377179402119), (0.7177622753575823,-0.40663773460337127), (4.726242809142292,-1.2343888551769233), (0.40501124104084163,-0.5036902484765656), (0.671193674673483,-0.3721487060577523), (0.07291534205751815,-1.987521549256808), (3.0132477799937103,1.175800228965741), (1.5800876523474532,-0.00707441417410632), (0.7578218048199775,1.0509376236793728), (-1.0013372637909774,0.7494475310558243), (2.4973399203372537,-0.38427172332612247), (2.7430757646823727,-0.930358252352808), (0.9420574052414756,0.960228212698756), (1.6907242105347866,1.0130821037354474), (0.5383478390341173,-0.03770027592004082), (-0.8776876240450338,-0.9950054698782884), (-2.176789802349519,-1.8847718393419073), (1.438183108689277,1.2260442128311473), (4.2808350483606805,0.8973362086382494), (1.4736191295860737,-1.1448274676767252), (-1.0596070355961702,1.2547096088120568), (1.2806057245712048,-0.9186364033184078), (1.699538265086482,0.8026832422467616), (2.4523729028460886,-0.4758593270796066), (-2.076551861641827,-0.830645491686263), (0.3829213050954927,-0.5400873564064651), (2.526475840066497,-0.6630169496517384), (-2.0435565279682697,0.6381413581264455), (1.0949799154339852,-0.7504597023733945), (2.9295011827611033,-0.5222934465699843), (-1.423740168713263,1.8847574147176338), (0.017436888071446166,0.6145641448752742), (-3.5813526640199997,-0.5988311192477164), (0.41775620625790433,-0.48271398431423773), (3.85232854616207,0.5473323912712278), (0.8559247574537103,-0.5885123950740869), (-1.004205312524907,1.0174330673374294), (-0.3466207150003797,0.9167061097763491), (2.6655511325842545,-0.5816363296158883), (-1.3576480411223657,0.030959788500977675), (0.6570353947095654,-0.5137243911676949), (0.4292434709946844,-1.8709170592192632), (1.0774815292368056,-1.7472131497273815), (1.1870324868533702,-1.1974973880763593), (0.2095199581899425,-0.09142445477515819), (0.010174654351863599,0.40624320435362116), (2.2901710235045103,-0.40045440389966913), (2.863966697515529,-0.37854644978248697), (3.688198186591751,0.11830887334108421), (2.0169458626980523,1.4637485338108103), (-1.505319168869295,-0.7665087390801844), (1.8372698368751088,-0.3760615977493943), (1.8008952595300867,-1.5309037473721014), (-1.546114129410773,-1.4651567904030773), (-0.841091525148669,-0.026798948253209182), (3.3919309207490786,0.2066789574673153), (2.088811112176163,1.8143254569727947), (-2.8769838245667856,2.3406140678357765), (0.4322984726888067,0.18034205783137355), (1.303043972931992,-1.4479109856334291), (-0.8423009866321116,-1.1294650286779386), (2.3991460284110797,1.2932328016031633), (-1.2931723513142943,-0.1393663274001316), (2.0843200968397437,-0.8468551132460095), (1.588879714937239,1.2713996676202226), (0.16628786148190622,-0.060721411812719187), (3.381916527851547,0.6928376952914521), (-1.4139426566326065,1.0579153298124913), (-1.9182803080402397,-0.7645892788282653), (3.4030209272918843,-1.4263627705259183), (2.348169386628692,0.5336299350415817), (2.7108694994929197,0.6573179344240204), (3.526692155778424,0.6654619137191999), (1.9363458607956388,-2.123504844254055), (1.2676248599666975,1.4866262970106303), (0.6250180966831009,0.129061989976924), (0.5133381875585852,0.22269040795154907), (4.062846560502843,-0.049559875163616714), (-1.0584402793210144,-0.9450310180664842), (-1.3944570089945305,-0.11473631908336672), (5.159311184253761,-0.39245949031895117), (1.05473696332479,1.5510785871987065), (0.15029838928005657,1.8254864916125728), (-4.3831809540558195,2.061530134034693), (-0.5446931918197788,0.8418470519782356), (1.7932009992060465,0.06828445075170164), (2.873840491469598,1.663002872049384), (-2.392423493455181,0.5626537089196723), (-2.246490718314186,1.1924431746135973), (3.051289545082127,0.37430311472451033), (2.921558256207268,-0.370821637491865), (3.9054719498051775,-0.09451085038956872), (0.30648948283890143,1.8818552958789954), (0.28062132301179654,-0.5452487096855598), (-0.6636405711277971,0.9207216541161212), (-0.23995842176230187,-0.03840103995077165), (-3.091235716784537,1.3724685915375903), (-1.4940558643511443,0.8666584948313616), (1.9003814806931487,-1.1444960562771815), (2.8178532836819334,0.2805555937763331), (0.37928684873689544,0.7940128878027313), (3.586810316002774,0.5091140746762197), (3.4620802514921007,1.5121770072162093), (0.0576382324555591,-0.5742888800806798), (2.3444623732611887,2.3581174706724157), (4.446712694084447,1.7068949917127816), (0.11914364217820061,0.4624349386051622), (1.0378949464259808,0.1225204014240068), (3.1306376315369757,0.9398794428978549), (3.2980072200625696,-0.9518050374642764), (-0.2975742780145032,-1.0522230289227708), (2.4125155391923037,0.8902191043953458), (-0.009450198942695609,-1.0945928783869934), (-0.4114802634876489,0.18036570004934346), (-1.039046520029077,-0.8406140101582382), (0.03977909071980956,-0.8634555290392517), (3.84686114196863,0.7792166711410811), (-0.4466827723217275,1.289633532883873), (1.1955794181366715,-0.4501732830926528), (3.305267656565898,2.7021303462005424), (0.5920837478263631,-0.22262701182155162), (1.8041012458041257,0.6460605038491092), (1.0663497873352485,-1.0247372175076013), (0.46091153333898405,-1.307280513522457), (0.7938864878059918,1.6404006952919912), (1.1094190243389264,-0.0518398187177849), (-0.641191315654986,0.45080433367542755), (-2.2044702809494647,0.3719461665317735), (1.0625161889548376,-1.4543187272074507), (3.9153759785005255,0.10188821604602745), (-0.31526297573578144,-0.27296335375143177), (6.7096374220372565,1.2451029860749379), (1.0132384508094343,0.14487479131269917), (2.1110257417957357,0.18201741036674932), (1.7567294506702076,0.03920775130429693), (2.2859862705949117,-0.2580100207138237), (3.0561313373519234,1.7047536767840974), (1.676678150633316,0.29766687892421284), (1.937200165306161,-1.7401156308492993), (1.7306597418522667,0.05515391539227905), (-0.7687330306969606,-0.27119832477161965), (1.6133041783818296,-1.0452547788609865), (-0.45709897470234107,-1.4984850253735646), (-0.11760611625946105,-2.079436054773939), (1.6931561226134122,0.44308840473545447), (-3.120132757148525,0.25444206315208995), (2.4874329799865276,-0.19417163889547193), (-2.071286380736543,-0.5363606084152478), (1.2245794297058403,0.36176129341638513), (1.5561427304696545,-0.33671294206693936), (-2.2132167533079596,-2.2316073845645206), (-0.1806519997812015,0.0455416268115695), (-1.8531519070689244,1.0401389018577378), (-1.4251005245889679,-0.3141083536641634), (-0.12116861831588355,0.07341971359886727), (-0.16311082359616202,1.0284248217766079), (1.1452304838866687,1.278483571961632), (0.16439959613142852,-0.7833209611949998), (2.372898903650935,1.3772836627732903), (-1.620865721699925,0.9700373682532052), (0.36995107938907257,-0.8662419500657893), (1.3699125246227284,-1.5938783703449753), (2.48540066183223,-0.6174495172524604), (1.7700634670756181,-0.7819620808358254), (-1.4895838800008887,0.10302666069195707), (-2.2924674968350516,1.0028767323240257), (1.7961909211171703,-0.10674590158973656), (5.272881180606573,0.38777338877043965), (-0.25483551061368104,-1.0509152481698179), (-0.6889539664067525,-0.06337185419126991), (4.8051377193993385,1.4117521801837538), (2.9847454768322432,0.7613291392405328), (-0.24298275058801688,1.0897823589924052), (0.4450211079339286,-0.6637558495096709), (3.9757632460215255,-1.8157963102001593), (3.1943058250078,1.2358784007741146), (1.6029441920839047,-2.15537358668726), (0.34496068898614773,0.2971613832014052), (-0.701365313565937,-0.750050504028126), (2.8797174467601363,0.8329394704642922), (-0.13811245773318692,-1.071705174325631), (-0.4071920336779187,-0.9018883675016863), (-0.980614720236086,-0.8504391140892998), (2.049070837825484,-1.0078945028740045), (-0.6656569729544988,-0.44317562320236065), (1.6714981157456288,1.3250813581783045), (3.3730612370932245,0.6196354624287462), (3.0056576612532875,-1.278621584064485), (1.5115584507080988,1.9184511848237962), (1.7299939794801924,-0.516835702626045), (2.578275842879262,0.6357213808712543), (2.6052371080790184,0.925579417321024), (-2.250831271025072,-0.5545984413834245), (2.6102577747003695,-1.3658354424121768), (-0.6857732802886782,-0.5628777726709475), (2.68127965494721,-0.04769253246241577), (-0.23756253168905261,-0.6991542495947202), (2.933572585641892,-1.1147192653783664), (0.5614839385686299,-0.5166063563392334), (3.225346367977653,-1.7956252514175055), (-0.004972800365140584,1.6987454382135425), (2.3788301806207484,-0.43821792344869875), (-2.039370915599196,1.022408572973808), (2.0463606707908997,0.3336089209393924), (-0.4145554379817833,0.02312505623899069), (0.08116934207464666,-0.31233039785382466), (-0.5477934703182952,-0.824119195846109), (-2.1370441042978534,0.13191861079694536), (0.25420113532525435,-0.6036888443048509), (1.0360988579414563,0.11668453965551463), (1.8638704589133164,-1.2724302084947103), (4.753630548842833,0.28050518201340424), (-3.3515189565291017,-1.0824241957346425), (1.5594601410586848,-1.1326924749331988), (3.1996282995078142,0.5172393559288399), (-2.898934616162431,-0.7220647087293313), (0.954273115625876,-1.2436289736107498), (0.28240108595091895,-0.6156240117132562), (0.7716522151389817,1.0406128654122049), (3.5094764570608667,-0.6297102775960882), (-2.7297328544947694,0.012206672359322898), (3.1669548682515365,0.4525998876038395), (-1.4851752825309625,-1.0451363444854351), (1.737419298693093,-1.3734168129137967), (1.7310067105728377,0.08679497767122704), (2.5247654346540447,-2.04474580239617), (1.7264929904926656,-0.19771649973564123), (3.4257461180577016,-1.3186299012886857), (-2.5422136030597478,-0.2761375653147074), (0.4542310289448179,0.5741590875530609), (-3.195921668005856,0.9593237124668771), (0.7587040179222494,1.0961298140479905), (1.4084546036945158,0.4391578691796409), (-1.0057668147395509,-0.6238036279250703), (-0.49303951955178493,-0.05430692986877771), (-1.1278958812038873,-0.11640313781962018), (-0.5550385126005621,2.0002028697055154), (-1.327368897717546,0.7350624069633737), (0.6693045410982479,-0.17859693817047373), (0.9549634686422886,-2.314523014865333), (1.500290442382846,0.22960402078570183), (-1.062055316386933,0.5724080038056716), (4.17217815778416,-0.07916244301664163), (2.678008750316924,-0.5341143586242104), (-0.8407983093249223,1.6460207223980545), (0.1875458897850375,0.3137460006226553), (-1.4179172686232406,-0.9207927088314583), (-0.49483409094368813,0.6967661703270028), (3.0009788765530763,0.5571396449070067), (2.0247746940707065,0.31796421714524625), (-0.2532041121084554,-0.2228633316357967), (3.652347645598589,-0.5499817903565626), (0.7886575879739799,-0.7841758341511148), (1.8099012750189574,-0.02401453955427877), (2.6230258299791482,-2.3166764321273394), (-0.717877434770809,1.0391140336549198), (-0.061241279683518,0.5498488089121792), (5.075089912736638,1.2335108771359133), (-0.8661311149639208,-1.9943985764799559), (-0.4069254685974877,0.042171455201203426), (4.929354815737875,-0.7485156690697342), (1.5066515864634513,-0.9570963024254994), (-1.509352007807196,1.301620771050323), (-0.17664963489668062,0.04031728339000787), (-0.4992194165794621,-0.6859845801505859), (-1.113404970974071,1.3911313622090693), (0.09081039944017699,-1.6605547611122933), (-1.109106884499735,0.6699454108732157), (0.6485386309736403,-1.1570517171606756), (7.436914934152387,-0.36244860379594424), (0.4579796383680059,-0.09907031133103263), (0.43490753215458244,0.19148521269498425), (1.5924776792705433,0.8267773040085179), (1.5577372417716786,-1.3644874947043841), (-0.1383846240201141,-1.197567110439719), (1.102472332306647,1.4817262828203837), (-0.5007687156746798,-0.10843051264695289), (2.746605165951965,0.08108871807685476), (1.1354991842453341,1.3545724635606313), (0.5494189005690517,0.5100719528278953), (-0.06026129274218861,0.057285477365546435), (4.089663482193665,-0.37191016428905793), (0.726791148499311,0.18281213200485982), (2.4464995994435466,-0.8842932261652275), (0.15700154458188786,-1.6494429918726925), (1.272472165346263,-0.7764264486783047), (-3.195988699113852,-0.3042211935492598), (3.520246085692976,-0.020825036678536985), (1.3375848973775626,-0.30368381843937947), (0.15355688719653504,0.7800891562758471), (0.3017504926614425,-0.0735325489432372), (1.210549152793178,-0.1679937736476282), (2.722259220076117,-1.79719393467339), (2.145916964295476,1.0755228066035305), (1.2838598063485724,-0.15773836735128086), (-0.47498086044457755,-0.0783402653348485), (-1.5584264763520719,-1.719143525921516), (2.0902157221330726,0.8435175129296529), (0.14776584386725267,0.6761343445136715), (3.954333236916723,0.5244103817082383), (0.5686484060482982,0.8620143952438806), (0.5252372312948147,-2.3060336651929445), (1.0577965808033563,1.4304037697137544), (0.9412615679730147,-0.7759942408828852), (3.5989366916799774,-1.2609787369882242), (2.919020206007784,-1.4478365748347117), (0.7075735919201673,2.0278952751593384), (-1.7988006346520633,-0.7861955769834937), (-0.1702629410006551,-0.2479198127035199), (-1.8327114024097777,-0.6571378828960646), (-2.4320883864687493,-2.1673373348092366), (1.5506981243634088,-1.7499456634223678), (-1.6486363495756167,0.19663998656853085), (-0.07210328541259714,0.23504549724402013), (3.3359309494368614,0.3638239098400113), (-1.9661523473680842,-1.3775094864583974), (-1.5234912819808697,-0.020668102118018273), (3.6096747834567844,0.25664562550595316), (3.792600951806,0.28318073789879256), (5.445353546258513,-0.29571052367446293), (3.5440273579163764,0.7681213080895282), (0.9197396037638609,1.5610874049559282), (3.811849778098893,-2.622344889120145), (2.7332345459673975,-0.8202101196954132), (-0.5818046844472109,0.7748393413408604), (-0.5281115636440643,0.508724216091097), (1.5044235562058499,-0.3751101664710589), (2.873650376809124,-0.9591972717671285), (1.8129440785275448,0.925165234056263), (2.0179287591118342,1.8459965937810827), (0.8322142722701716,0.27342245677072646), (-0.4285731224378031,-2.4602979491815096), (2.4399636602901773,0.6963477323668592), (-0.9106687544606735,2.167550027845107), (2.2303985755555713,-0.19332822247961037), (1.3790259597005396,0.5981380076459031), (3.7996185186834577,-0.694964755780304), (1.2956010771846629,0.08919555493105245), (1.538146993149411,-2.0804801088705767), (0.34956131459239337,1.1920101085447832), (2.324890219256198,1.231750346524178), (-0.007461769742674651,-1.455653407187848), (-0.25687852714004933,1.2328411389316813), (-0.10083857130927787,-0.39573709387712014), (-0.4576304833103986,-1.1956194147671937), (3.5460429684560597,-0.04302633139042216), (2.3895059706818245,2.0750703846663137), (0.6141642991105689,-1.7646202495106422), (2.7461944701594874,-0.017144573753570983), (-0.9026983960663617,0.1715206676563901), (0.7378437996873302,-0.9656202470043257), (1.66416214705831,-1.2648785618794656), (0.27509031436823117,-1.3357866485679637), (-1.1687533676873079,0.1818276074213889), (-0.3868893758816525,-0.1742396893961462), (4.421141775966374,-0.6767070415199365), (2.3562673999312493,2.137638188136061), (-1.4191111968198054,0.4838102855828845), (1.2100382115048613,0.3075626385768448), (3.9433787679288734,-1.3633563499769654), (3.4523083260238985,0.02908534679907131), (0.5931106981807055,2.321245009450074), (0.5323635042903148,-0.037890984508530186), (0.6432548496942007,-1.3762801283466828), (1.7830049160784451,1.6125793622774343), (3.343550972317695,0.201560934160273), (0.3058331091080799,-0.15731271258242444), (3.5132026769406743,0.6015468376832578), (-2.971225740597245,-0.0991480003326356), (1.3624546339392878,0.9603378027154168), (-0.8015108509863962,0.5407700164856909), (-2.1839032564232803,-0.18116162969719404), (-0.304110507108742,1.515039231410728), (3.3539330766126354,1.0516518441747622), (1.3933519433178922,-1.108150706054602), (2.734409447480007,-0.0902162047955897), (1.4442418533781687,0.525636626469496), (4.870497710752748,-0.6505154458390053), (0.27599486504010007,-0.6385600015638014), (1.249763671657228,-0.03765503579803463), (2.8900482496931597,-0.8067465597676813), (0.35997271322967783,0.25298188952694345), (1.6204223282918582,-1.2484755253881343), (0.17015908552188153,0.5907884315290602), (-0.17656545289416115,-0.7407171504354528), (4.73815324418716,0.09342389392065106), (-2.4335528485721105,-0.6358631568411891), (3.4290423053083563,-0.7990035391904377), (-2.204366994061862,0.7318639477491172), (-2.9824782737280255,1.0721139408861649), (1.559154053243073,-0.7709696933298915), (1.7355575197273223,1.4262689270118047), (1.11638805377925,0.008052693380149429), (4.6951872993892,-0.7675924241217212), (2.737373852455214,-0.875355770610139), (-1.151456200240173,0.3318552144813144), (3.0152094819448543,0.47230162167225975), (2.0959843639444466,-0.28845007593399613), (0.33596546387727444,0.33674053005607557), (1.040331737541483,-1.2685836846998517), (2.431166696911837,0.9973713512813284), (2.230055598607015,-0.14095584542177128), (1.3700666149073755,0.6434211266970435), (-0.4327077332358864,-1.3166935231718915), (-1.6077594316454586,0.5010874593219053), (-2.3718964441684305,0.5400322253256449), (2.6760922672748455,0.1131615290866549), (1.460158508062642,1.1948743843705798), (4.289535800486267,-0.4068064171361299), (0.7168408537080622,-0.2145531753251343), (-0.354296028077995,-0.8414414845064078), (0.6348900562033981,-0.17448661922809763), (4.50175610250241,0.015039956542559319), (4.007289452214197,-2.00225473143416), (0.5029484622378626,0.7768533690748148), (1.6705288530684772,0.7280781053306761), (-1.9266086884168034,-0.6814357844622568), (0.6563298284821315,2.4398143015234925), (3.8936524199780114,-1.9715384636473667), (-0.5323653052192214,-1.089089720589046), (-2.187378645730113,-1.0771030789727833), (1.4490595169584695,-1.1187887124853761), (-4.570489383235827,0.26304747184316635), (0.15125945754562187,0.6104069218328998), (3.502660482621176,0.814212432670669), (-0.7485242949055997,-1.1311163095796626), (3.5175847260522777,-0.715657837824097), (1.292722602559689,0.11337791560485909), (-1.0688535529107246,1.1903117127474667), (2.610918918455355,0.8655118045637775), (2.165592336786331,-0.4786300318354617), (3.2180039450399307,1.1374241148097717), (2.3110620837831366,1.3574248394663742), (-1.2620670074181475,-0.59796895966422), (-2.022658500226865,0.5787074373907313), (0.4225942665282606,-1.4837192620796162), (-1.8746483642338414,0.15009771866713956), (1.3304121456690439,-1.0785198355458125), (2.528499056420297,0.43654174540174034), (-2.5920035883882417,1.367374206941478), (1.8434672245723738,1.1624833124548457), (2.099308858628433,1.4378845237950593), (-0.8157539314263724,-0.4752367281036082), (-0.5844201825108297,-0.6295231876858187), (3.6001251647036625,0.13609988490474167), (0.9750494830689591,1.451172421854568), (-1.591208358857827,-0.9690521636819287), (2.272688328123594,-0.4657652028114279), (1.088984838199552,-0.3272245138051527), (2.700650297482846,-0.3254730682226242), (0.29059088340060446,-1.1968451698216205), (-0.8790370032378447,-2.374605096166881), (4.604279856177609,1.4231065460066927), (2.722449227993615,0.3339627050188421), (3.587573251254368,-0.37529586334821136), (-1.4412229896551314,1.7121288810552737), (2.0753371457353946,0.7740341080806812), (1.1111154091929833,-0.08101997848273264), (1.7671887536692417,0.8171553719668059), (1.9408459883070575,1.004037394973044), (0.7804144409892885,0.22996069331921015), (-1.8543998116282898,-0.135350519278533), (6.042636547872292,-0.4727403129550938), (2.1280007672692847,-0.1182151974084787), (0.4471612263594835,-1.9468945045910213), (1.2505523478947802,0.4579087802822954), (0.6984945299477181,1.3836409234546432), (-0.8471775601094396,0.02325932498935564), (4.322477962324404,0.475495388508411), (1.4341612180258405,-0.37698432805620236), (6.239615258142803,-0.8219848732487407), (-3.1535278213087903,-1.3951694203634668), (-0.463940144432212,0.0484490852454009), (-0.14771318640324793,0.9222291689334502), (0.20282785933721803,1.5679464498188418), (1.8482316809367179,0.810147143385619), (1.464243976509526,1.9186942393210789), (-2.344895033686407,-0.2176761309409645), (-0.6222382474654642,1.497682278641026), (-2.028256489351115,-0.6745280698091511), (-1.8185091579263823,-0.14287398549911423), (0.7775757995485517,-0.6123313834457077), (2.208419884574348,0.5533479535312111), (-1.510931362400885,0.17525285583058028), (0.15702274793503757,0.26417401560092896), (-0.460045243107128,0.7989612854266647), (3.543882315893044,-0.6531854416899909), (1.1358671646312586,0.3890137678570137), (1.0004036793674747,1.5942398929174109), (-2.229985885404819,0.29843502966443813), (0.27723413696937793,0.28270288452527464), (-1.0586523471937253,-1.030296515196584), (-0.6053699726992356,0.29004278605641326), (-0.9059032244427896,1.1839066433022867), (0.30610586096155024,0.5617980143776449), (-1.1035109557285785,0.3177318190024553), (0.818395734988232,-0.17078152902760071), (1.2887245591289829,-0.4129031279462113), (3.228020724069559,1.6509715706618235), (0.6610048448754513,-0.27367425727559463), (1.5679633903647399,1.2450208243939591), (1.5877868046262307,1.4063428988232145), (1.9799774803293921,1.6639488103943056), (1.8297536930120843,0.4109699563209272), (-0.21778571324236595,-2.1509670233572784), (0.6433445252381844,0.2639495116581482), (-1.261874773876885,-0.9708523309574577), (-1.020712141743926,1.2588123216998128), (-0.942167556946186,-0.6395148239066109), (2.7320211668089156,-2.3731627585033115), (2.812558906168454,0.43524872907522616), (-0.056923496414450714,0.6328073362083871), (-0.6014131312820246,-0.2381862432808314), (-3.5196269653783325,0.40492896291344244), (-1.1067214502928073,-1.0527840343286374), (-0.4736116098154268,1.0689442908821987), (2.6403094933932953,0.789255581937264), (1.5982824524769197,1.1724902795494772), (-0.09123150857004858,1.2893390188349818), (0.3777640975590455,0.8926054903089587), (1.5757168418364564,1.4246842661952515), (-0.24202485980852528,0.21910907895561132), (4.589933486557021,0.34689263334132264), (-2.331742795642637,-0.5771627038699307), (1.0060513913917988,-0.04827308490054824), (-0.4295021770429881,-0.8361201294738408), (-0.21561647198666445,0.2776346366703466), (1.4993086132360396,-2.583216199463966), (1.0627822777545926,-0.5754336242709439), (-2.4095942089088482,-0.4740298729504418), (1.5975461609152366,0.45528050967309863), (2.1565799558529477,-0.937001723779216), (0.0962289023611611,0.12442585999261868), (2.7095641996945012,-1.037597020742512), (-0.9219208363686942,0.6656497732259127), (-1.0338058740520197,0.09718535082910094), (1.6364174103842437,0.472044663569549), (1.2274838964756027,0.01365815985041524), (-0.7303973452135653,0.21443946294874028), (1.6471043795729972,0.38540036725618015), (0.13612088488197083,0.7807770362857364), (-4.201610226957669,0.12940125757456059), (0.8225909001565422,-0.9556204076037894), (-0.9482388322224697,-1.0007190699745476), (-1.5523717842885172,-1.4238233782457508), (-1.4048827543540963,0.09834207753278863), (1.5338217310158921,-2.5447136959911605), (2.2987508128497063,0.1997331013963162), (-1.9069233483255275,1.2806574080505437), (-0.48017389975436475,0.9315316175761184), (3.708171200382932,-0.6394107818742375), (1.646362242992558,-1.4099227738716835), (5.344457888650813,-0.21682467720141205), (4.0401870066710455,-0.214248285713938...

val rddLR = sc.parallelize(yx)
rddLR.take(5)

rddLR: org.apache.spark.rdd.RDD[(Double, Double, Double)] = ParallelCollectionRDD[4] at parallelize at command-2971213210277128:1
res2: Array[(Double, Double, Double)] = Array((5.677461671611596,1.6983876233282715,0.1510498720110515), (6.5963106438672625,1.9306937118935266,-0.21709023201809416), (-6.457080480280965,-3.613579039093624,-0.9420189342233366), (5.725199404014689,1.4774128627063305,-0.6050521762158899), (5.2485124653727215,0.49536920319327216,0.48771393128636853))

val dfLR = rddLR.toDF("y","x1","x2")
dfLR.show
dfLR.show(5)

+-------------------+--------------------+--------------------+
|                  y|                  x1|                  x2|
+-------------------+--------------------+--------------------+
|  5.677461671611596|  1.6983876233282715|  0.1510498720110515|
| 6.5963106438672625|  1.9306937118935266|-0.21709023201809416|
| -6.457080480280965|  -3.613579039093624| -0.9420189342233366|
|  5.725199404014689|  1.4774128627063305| -0.6050521762158899|
| 5.2485124653727215| 0.49536920319327216| 0.48771393128636853|
|  6.147100844408568|   2.619266385545484|0.027986752952643124|
|-0.7268042607303868| -1.6216138344745752| 0.08206698908429294|
| 5.5363351738375695|  1.2511380715559544| -0.7810931162704136|
|-0.1828379004896784| -0.6891022233658615| -1.2057806605697985|
|  6.488199218865876|  2.7890847095863527| -0.9176042669882863|
| 1.3969322930903634| -0.8384317817611013|  1.1959113751272468|
|  1.468434023004454|-0.39702300426389736|  0.5771223111098949|
|-0.6234341966751746|-0.20341459413042506| -0.5030579509564024|
|  6.452061108539848|   1.940354690386933| 0.05792512611966754|
|  9.199371261060582|  3.1686700336304794|  2.0492117764647917|
|0.07256979343692907|-0.01571825151138...| -1.6192400586380957|
|  9.589528428542355|  3.5222633033548227|-0.28568171141353893|
| 1.1696257215909478| -1.7700416261712641|   1.662660432138664|
|-0.8138719762638745|  -1.276616299487233| -0.3053683539449641|
|  9.309926019422797|   2.847017612404211|  0.3879563181609168|
+-------------------+--------------------+--------------------+
only showing top 20 rows

+------------------+-------------------+--------------------+
|                 y|                 x1|                  x2|
+------------------+-------------------+--------------------+
| 5.677461671611596| 1.6983876233282715|  0.1510498720110515|
|6.5963106438672625| 1.9306937118935266|-0.21709023201809416|
|-6.457080480280965| -3.613579039093624| -0.9420189342233366|
| 5.725199404014689| 1.4774128627063305| -0.6050521762158899|
|5.2485124653727215|0.49536920319327216| 0.48771393128636853|
+------------------+-------------------+--------------------+
only showing top 5 rows

dfLR: org.apache.spark.sql.DataFrame = [y: double, x1: double ... 1 more field]

// importing for regression
import org.apache.spark.ml.regression.LinearRegression
import org.apache.spark.ml.linalg._

val lm = new LinearRegression
lm.explainParams
lm.getStandardization
lm.setStandardization(false)
lm.getStandardization
lm.explainParams

import org.apache.spark.ml.regression.LinearRegression
import org.apache.spark.ml.linalg._
lm: org.apache.spark.ml.regression.LinearRegression = linReg_fd18c4adb9af
res5: String =
aggregationDepth: suggested depth for treeAggregate (>= 2) (default: 2)
elasticNetParam: the ElasticNet mixing parameter, in range [0, 1]. For alpha = 0, the penalty is an L2 penalty. For alpha = 1, it is an L1 penalty (default: 0.0)
epsilon: The shape parameter to control the amount of robustness. Must be > 1.0. (default: 1.35)
featuresCol: features column name (default: features)
fitIntercept: whether to fit an intercept term (default: true)
labelCol: label column name (default: label)
loss: The loss function to be optimized. Supported options: squaredError, huber. (Default squaredError) (default: squaredError)
maxBlockSizeInMB: Maximum memory in MB for stacking input data into blocks. Data is stacked within partitions. If more than remaining data size in a partition then it is adjusted to the data size. Default 0.0 represents choosing optimal value, depends on specific algorithm. Must be >= 0. (default: 0.0)
maxIter: maximum number of iterations (>= 0) (default: 100)
predictionCol: prediction column name (default: prediction)
regParam: regularization parameter (>= 0) (default: 0.0)
solver: The solver algorithm for optimization. Supported options: auto, normal, l-bfgs. (Default auto) (default: auto)
standardization: whether to standardize the training features before fitting the model (default: true, current: false)
tol: the convergence tolerance for iterative algorithms (>= 0) (default: 1.0E-6)
weightCol: weight column name. If this is not set or empty, we treat all instance weights as 1.0 (undefined)

// Transform data frame to required format
val dflr = (dfLR map {row => (row.getDouble(0), 
           Vectors.dense(row.getDouble(1),row.getDouble(2)))}).
           toDF("label","features")
dflr.show(5)

+------------------+--------------------+
|             label|            features|
+------------------+--------------------+
| 5.677461671611596|[1.69838762332827...|
|6.5963106438672625|[1.93069371189352...|
|-6.457080480280965|[-3.6135790390936...|
| 5.725199404014689|[1.47741286270633...|
|5.2485124653727215|[0.49536920319327...|
+------------------+--------------------+
only showing top 5 rows

dflr: org.apache.spark.sql.DataFrame = [label: double, features: vector]

// Fit model
val fit = lm.fit(dflr)
fit.intercept

fit: org.apache.spark.ml.regression.LinearRegressionModel = LinearRegressionModel: uid=linReg_fd18c4adb9af, numFeatures=2
res8: Double = 1.4942320912612896

fit.coefficients

res9: org.apache.spark.ml.linalg.Vector = [1.9996093571014764,1.0144075351786204]

val summ = fit.summary

summ: org.apache.spark.ml.regression.LinearRegressionTrainingSummary = org.apache.spark.ml.regression.LinearRegressionTrainingSummary@b34e84f

summ.r2

res10: Double = 0.9436414660418528

summ.rootMeanSquaredError

res11: Double = 1.0026971017612039

summ.coefficientStandardErrors

res12: Array[Double] = Array(0.005018897596822849, 0.009968316877946075, 0.01123197045538135)

summ.pValues

res13: Array[Double] = Array(0.0, 0.0, 0.0)

summ.tValues

res14: Array[Double] = Array(398.4160502432455, 101.76317101464718, 133.03383384038267)

summ.predictions.show(5)

+------------------+--------------------+------------------+
|             label|            features|        prediction|
+------------------+--------------------+------------------+
| 5.677461671611596|[1.69838762332827...| 5.043570003209616|
|6.5963106438672625|[1.93069371189352...| 5.134647336087737|
|-6.457080480280965|[-3.6135790390936...|-6.687105473093168|
| 5.725199404014689|[1.47741286270633...| 3.834711189101326|
|5.2485124653727215|[0.49536920319327...|2.9795176720949392|
+------------------+--------------------+------------------+
only showing top 5 rows

summ.residuals.show(5)

+-------------------+
|          residuals|
+-------------------+
| 0.6338916684019802|
| 1.4616633077795251|
|0.23002499281220334|
|  1.890488214913363|
| 2.2689947932777823|
+-------------------+
only showing top 5 rows

This gives you more on doing generalised linear modelling in Scala. But let's go back to out pipeline using the power-plant data.

Exercise: Writing your own Spark program for the least squares fit

Your task is to use the syntactic sugar below to write your own linear regression function using `reduce` and `broadcast` operations

How would you write your own Spark program to find the least squares fit on the following 1000 data points in RDD rddLR, where the variable y is the response variable, and X1 and X2 are independent variables?

More precisely, find \(w_1, w_2\), such that,

\(\sum_{i=1}^{10} (w_1 X1_i + w_2 X2_i - y_i)^2\) is minimized.

Report \(w_1\), \(w_2\), and the Root Mean Square Error and submit code in Spark. Analyze the resulting algorithm in terms of all-to-all, one-to-all, and all-to-one communication patterns.

rddLR.count

res17: Long = 10000

rddLR.take(10)

res18: Array[(Double, Double, Double)] = Array((5.677461671611596,1.6983876233282715,0.1510498720110515), (6.5963106438672625,1.9306937118935266,-0.21709023201809416), (-6.457080480280965,-3.613579039093624,-0.9420189342233366), (5.725199404014689,1.4774128627063305,-0.6050521762158899), (5.2485124653727215,0.49536920319327216,0.48771393128636853), (6.147100844408568,2.619266385545484,0.027986752952643124), (-0.7268042607303868,-1.6216138344745752,0.08206698908429294), (5.5363351738375695,1.2511380715559544,-0.7810931162704136), (-0.1828379004896784,-0.6891022233658615,-1.2057806605697985), (6.488199218865876,2.7890847095863527,-0.9176042669882863))

val dfLR = rddLR.toDF("y","x1","x2")
dfLR.show(10)

+-------------------+-------------------+--------------------+
|                  y|                 x1|                  x2|
+-------------------+-------------------+--------------------+
|  5.677461671611596| 1.6983876233282715|  0.1510498720110515|
| 6.5963106438672625| 1.9306937118935266|-0.21709023201809416|
| -6.457080480280965| -3.613579039093624| -0.9420189342233366|
|  5.725199404014689| 1.4774128627063305| -0.6050521762158899|
| 5.2485124653727215|0.49536920319327216| 0.48771393128636853|
|  6.147100844408568|  2.619266385545484|0.027986752952643124|
|-0.7268042607303868|-1.6216138344745752| 0.08206698908429294|
| 5.5363351738375695| 1.2511380715559544| -0.7810931162704136|
|-0.1828379004896784|-0.6891022233658615| -1.2057806605697985|
|  6.488199218865876| 2.7890847095863527| -0.9176042669882863|
+-------------------+-------------------+--------------------+
only showing top 10 rows

dfLR: org.apache.spark.sql.DataFrame = [y: double, x1: double ... 1 more field]

rddLR.getNumPartitions

res21: Int = 2

rddLR.map( yx1x2 => (yx1x2._2, yx1x2._3, yx1x2._1) ).take(10)

res22: Array[(Double, Double, Double)] = Array((1.6983876233282715,0.1510498720110515,5.677461671611596), (1.9306937118935266,-0.21709023201809416,6.5963106438672625), (-3.613579039093624,-0.9420189342233366,-6.457080480280965), (1.4774128627063305,-0.6050521762158899,5.725199404014689), (0.49536920319327216,0.48771393128636853,5.2485124653727215), (2.619266385545484,0.027986752952643124,6.147100844408568), (-1.6216138344745752,0.08206698908429294,-0.7268042607303868), (1.2511380715559544,-0.7810931162704136,5.5363351738375695), (-0.6891022233658615,-1.2057806605697985,-0.1828379004896784), (2.7890847095863527,-0.9176042669882863,6.488199218865876))

import breeze.linalg.DenseVector
    val pts = rddLR.map( yx1x2 => DenseVector(yx1x2._2, yx1x2._3, yx1x2._1) ).cache

import breeze.linalg.DenseVector
pts: org.apache.spark.rdd.RDD[breeze.linalg.DenseVector[Double]] = MapPartitionsRDD[40] at map at command-2971213210277150:2

Now all we need to do is one-to-all and all-to-one broadcast of the gradient...

val w = DenseVector(0.0, 0.0, 0.0)
val w_bc = sc.broadcast(w)
val step = 0.1
val max_iter = 100

/*
YouTry: Fix the expressions for grad_w0, grad_w1 and grad_w2 to have w_bc.value be the same as that from ml lib's fit.coefficients
*/    
for (i <- 1 to max_iter) {
        val grad_w0 = pts.map(x => 2*(w_bc.value(0)*1.0 + w_bc.value(1)*x(0) + w_bc.value(2)*x(1) - x(2))*1.0).reduce(_+_)
        val grad_w1 = pts.map(x => 2*(w_bc.value(0)*1.0 + w_bc.value(1)*x(0) + w_bc.value(2)*x(1) - x(2))*x(0)).reduce(_+_)
        val grad_w2 = pts.map(x => 2*(w_bc.value(0)*1.0 + w_bc.value(1)*x(0) + w_bc.value(2)*x(1) - x(2))*x(1)).reduce(_+_)
        w_bc.value(0) = w_bc.value(0) - step*grad_w0
        w_bc.value(1) = w_bc.value(1) - step*grad_w1
        w_bc.value(2) = w_bc.value(2) - step*grad_w2
}

w: breeze.linalg.DenseVector[Double] = DenseVector(703602.5501372493, 2297216.693185762, 194396.78259181185)
w_bc: org.apache.spark.broadcast.Broadcast[breeze.linalg.DenseVector[Double]] = Broadcast(16)
step: Double = 0.1
max_iter: Int = 100

w_bc.value

res24: breeze.linalg.DenseVector[Double] = DenseVector(703602.5501372493, 2297216.693185762, 194396.78259181185)

fit.intercept

res25: Double = 1.4942320912612896

fit.coefficients

res26: org.apache.spark.ml.linalg.Vector = [1.9996093571014764,1.0144075351786204]

Computing each of the gradients requires an all-to-one communication (due to the .reduce(_+_)). There are two of these per iteration. Broadcasting the updated w_bc requires one to all communication.

sds-3.x/ScaDaMaLe

Linear Regression by Hastie and Tibshirani

Gradient Descent

Getting our hands dirty with Darren Wilkinson's blog on regression

Exercise: Writing your own Spark program for the least squares fit

Your task is to use the syntactic sugar below to write your own linear regression function using reduce and broadcast operations

Your task is to use the syntactic sugar below to write your own linear regression function using `reduce` and `broadcast` operations