Applied Statistics

1MS926, Spring 2019, Uppsala University

©2019 Raazesh Sainudiin. Attribution 4.0 International (CC BY 4.0)

02. Numbers, Strings, Booleans and Sets

Applied Statistics

©2019 Raazesh Sainudiin. Attribution 4.0 International (CC BY 4.0)

Numbers and Arithmetic Operations

We will start by showing you some of the basic numeric capabilities of SageMath.

A worksheet cell is the area enclosed by a gray rectangle.
You may type any expression you want to evaluate into a worksheet cell. We have already put some expressions into this worksheet.

When you are in a cell you can evaluate the expression in it by pressing or just by clicking the evaluate button below the cell.

To start with, we are going to be using SAGE like a hand-held calculator. Let's perform the basic arithmetic operations of addition, subtraction, multiplication, division, exponentiation, and remainder over the three standard number systems: Integers denoted by $\mathbb{Z}$, Rational Numbers denoted by $\mathbb{Q}$ and Real Numbers denoted by $\mathbb{R}$. Let us recall the real number line and the basics of number systems next.

Number sets within Complex Numbers

In [102]:
def showURL(url, ht=500):
    """Return an IFrame of the url to show in notebook with height ht """
    from IPython.display import IFrame
    return IFrame(url, width='95%', height=ht) 
showURL('https://en.wikipedia.org/wiki/Number',400)
Out[102]:

The most basic numbers are called natural numbers and they are denoted by $\mathbb{N} :=\{0, 1,2,3,\ldots\}$. See https://en.wikipedia.org/wiki/Natural_number.

The natural numbers are the basis from which many other number sets may be built by extension: the integers, by including (if not yet in) the neutral element 0 and an additive inverse (−n) for each nonzero natural number n; the rational numbers, by including a multiplicative inverse (1/n) for each nonzero integer n (and also the product of these inverses by integers); the real numbers by including with the rationals the limits of (converging) Cauchy sequences of rationals; the complex numbers, by including with the real numbers the unresolved square root of minus one (and also the sums and products thereof); and so on. These chains of extensions make the natural numbers canonically embedded (identified) in the other number systems.

In [103]:
showURL("https://en.wikipedia.org/wiki/Natural_number#Notation",300)
Out[103]:

Let us get our fingers dirty with some numerical operations in SageMath.

Note that anything after a '#' symbol is a comment - comments are ignored by SAGE but help programmers to know what's going on.

Example 1: Integer Arithmetic

Try evaluating the cell containing 1+2 below by placing the cursor in the cell and pressing .

In [3]:
1+2 # one is being added to 2
Out[3]:
3

Now, modify the above expression and evaluate it again. Try 3+4, for instance.

In [4]:
3-4 # subtracting 4 from 3
Out[4]:
-1

The multiplication operator is *, the division operator is /.

In [5]:
2*6 # multiplying 2 by 6
Out[5]:
12
In [6]:
15/5 # dividing 15 by 5
Out[6]:
3
In [7]:
type(1)
Out[7]:
<type 'sage.rings.integer.Integer'>

The exponentiation operator is ^.

In [8]:
2^3 # exponentiating 2 by 3, i.e., raising 2 to the third power
Out[8]:
8

However, Python's exponentiation operator ** also works.

In [9]:
2**3
Out[9]:
8

Being able to finding the remainder after a division is surprisingly useful in computer programming.

In [10]:
11%3 # remainder after 11 is divided by 3; i.e., 11=3*3+2
Out[10]:
2

Another way of referring to this is 11 modulus 3, which evaluates to 2. Here % is the modulus operator.

You try

Try typing in and evaluating some expressions of your own. You can get new cells above or below an existing cell by clicking 'Insert' in the menu above and 'Insert Cell Above' or 'Insert Cell below'. You can also place the cursor at an existing cell and click + icon above to get a new cell below.

What happens if you put space between the characters in your expression, like: 1 + 2 instead of 1+2?.

Example 2: Operator Precedence for Evaluating Arithmetic Expressions

Sometimes we want to perform more than one arithmetic operation with some given integers.
Suppose, we want to

  • "divide 12 by 4 then add the product of 2 and 3 and finally subtract 1."

Perhaps this can be achieved by evaluating the expression "12/4+2*3-1"?

But could that also be interpreted as

  • "divide 12 by the sum of 4 and 2 and multiply the result by the difference of 3 and 1"?

In programming, there are rules for the order in which arithmetic operations are carried out. This is called the order of precedence.

The basic arithmetic operations are: +, -, *, %, /, ^.

The order in which operations are evaluated are as follows:

  • ^ Exponents are evaluated right to left
  • *, %, / Then multiplication, remainder and division operations are evaluated left to right
  • +, - Finally, addition and subtraction are evaluated left to right

When operators are at the same level in the list above, what matters is the evaluation order (right to left, or left to right).

Operator precedence can be forced using parenthesis.

In [109]:
showURL("https://en.wikipedia.org/wiki/Order_of_operations", 300)
Out[109]:
In [12]:
(12/4) + (2*3) - 1 # divide 12 by 4 then add the product of 2 and 3 and finally subtract 1
Out[12]:
8
In [13]:
12/4+2*3-1 # due to operator precedence this expression evaluates identically to the parenthesized expression above
Out[13]:
8

Operator precedence can be forced using nested parentheses. When our expression has nested parenthesis, i.e., one pair of parentheses inside another pair, the expression inside the inner-most pair of parentheses is evaluated first.

The following cell evaluates the mathematical expression:

$$\frac{12}{4+2} (3-1)$$

In [14]:
(12/(4+2)) * (3-1)  # divide 12 by the sum of 4 and 2 and multiply the result by the difference of 3 and 1
Out[14]:
4

You try

Try writing an expression which will subtract 3 from 5 and then raise the result to the power of 3.

Find out for yourself what we mean by the precedence for exponentiation (^) being from right to left:

  • What do you think the expression 3^3^2 would evaluate to?
    • Is it the same as (3^3)^2, i.e., 27 squared, or
    • 3^(3^2), i.e., 3 raised to the power 9?

Try typing in the different expressions to find out:

Find an expression which will add the squares of four numbers together and then divide that sum of squares by 4.

Find what the precedence is for the modulus operator % that we discussed above: try looking at the difference between the results for 10%2^2 and 10%2*2 (or 10^2+2). Can you see how SageMath is interpreting your expressions?

Note that when you have two operators at the same precedence level (like % and *), then what matters is the order - left to right or right to left. You will see this when you evaluate 10%2*2.

Does putting spaces in your expression make any difference?

Using parenthesis or white spaces can improve readability a lot! So be generous with them to evaluate the following expression:

$$10^2 + 2^8 -4$$

In [15]:
10^2+2^8-4
Out[15]:
352
In [16]:
10^2 + 2^8 -4
Out[16]:
352
In [17]:
(((10^2) + (2^8)) - 4)  # this may be overkill!
Out[17]:
352

The lesson to learn is that it is always good to use the parentheses: you will make it clear to someone reading your code what you mean to happen as well as making sure that the computer actually does what you mean it to!

Try this 10 minutes-long videos to get some practice if you are really rusty with order of operations:

Example 3: Rational Arithmetic

So far we have been dealing with integers. Integers are a type in SAGE. Algebraically speaking, integers, rational numbers and real numbers form a ring. This is something you will learn in detail in a maths course in Group Theory or Abstract Algebra, but let's take a quick peek at the definition of a ring.

In [110]:
showURL("https://en.wikipedia.org/wiki/Ring_(mathematics)#Definition_and_illustration",400)
Out[110]:
In [19]:
type(1) # find the data type of 1
Out[19]:
<type 'sage.rings.integer.Integer'>

The output above tells us that 1 is of type sage.rings.integer.Integer.

In [111]:
showURL("https://en.wikipedia.org/wiki/Integer",400)
Out[111]:

However, life with only integers denoted by $\mathbb{Z} := \{\ldots,-3,-2,-1,0,1,2,3,\ldots\}$ is a bit limited. What about values like $1/2$ or $\frac{1}{2}$?

This brings us to the rational numbers denoted by $\mathbb{Q}$.

In [112]:
showURL("https://en.wikipedia.org/wiki/Rational_number",400)
Out[112]:
In [22]:
type(1/2) # data type of 1/2 is a sage.rings.rational.Rational
Out[22]:
<type 'sage.rings.rational.Rational'>

Try evaluating the cell containing 1/2 + 2 below.

In [23]:
1/2 + 2 # add one half to 2 or four halves to obtain the rational number 5/2 or five halves
Out[23]:
5/2

SageMath seems to have done rational arithmetic for us when evaluating the above expression.

Next, modify the expression in the cell below and evaluate it again. Try 1/3+2/4, for instance.

In [24]:
1/2 + 1/3
Out[24]:
5/6

You can do arithmetic with rationals just as we did with integers.

In [25]:
3/4 - 1/4 # subtracting 3/4 from 1/4
Out[25]:
1/2
In [26]:
1/2 * 1/2 # multiplying 1/2 by 1/2
Out[26]:
1/4
In [27]:
(2/5) / (1/5) # dividing 2/5 by 1/5
Out[27]:
2
In [28]:
(1/2)^3 # exponentiating 1/2 by 3, i.e., raising 1/2 to the third power
Out[28]:
1/8

You try

Write an expression which evaluates to 1 using the rationals 1/3 and 1/12, some integers, and some of the arithmetical operators - there are lots of different expressions you can choose, just try a few.

What does SageMath do with something like 1/1/5? Can you see how this is being interpreted? What should we do if we really want to evaluate 1 divided by 1/5?

Try adding some rationals and some integers together - what type is the result?

Example 4: Real Arithmetic (multi-precision floating-point arithmetic)

Recall that real numbers denoted by $\mathbb{R}$ include natural numbers ($\mathbb{N}$), integers ($\mathbb{Z}$), rational numbers ($\mathbb{Q}$) and various types of irrational numbers like:

Real numbers can be thought of as all the numbers in the real line between negative infinity and positive infinity. Real numbers are represented in decimal format, for e.g. 234.4677878.

In [104]:
showURL("https://en.wikipedia.org/wiki/Real_number#Definition",400)
Out[104]:

We cannot do exact real arithmetic (but do it aproximately) in a computer with http://www.mpfr.org/'s multiprecision floating-point numbers, and can combine them with integer and rational types in SageMath.

Technical note: Computers can be made to exactly compute in integer and rational arithmetic. But, because computers with finite memory (all computers today!) cannot represent the uncountably infinitely many real numbers, they can only mimic or approximate arithmetic over real numbers using finitely many computer-representable floating-point numbers.

Let's take a peak at von Neumann architecture of any typical computer today.

Von_Neumann_Architecture.svg

The punch-line is that the Memory unit or Random Access Memory (RAM) inside the Central Processing Unit as well as Input and Output Devices are physical with finite memory. Therefore we cannot exactly represent all the uncontably infinitely many real numbers in $\mathbb{R}$ and need to resort to their approximation using floating point numbers.

In [106]:
showURL("https://en.wikipedia.org/wiki/Von_Neumann_architecture",400)
Out[106]:

See SageMath Quick Start on Numerical Analysis to understand SageMath's multiprecision real arithmetic.

For now, let's compare the results of evaluating the expressions below to the equivalent expressions using rational numbers above.

In [30]:
type(0.5) # data type of 0.5 is a sage.rings.real_mpfr.RealLiteral
Out[30]:
<type 'sage.rings.real_mpfr.RealLiteral'>
In [7]:
RR # Real Field with the default 53 bits of precision
Out[7]:
Real Field with 53 bits of precision
In [32]:
RR(0.5)  # RR(0.5) is the same as 0.5 in SageMath
Out[32]:
0.500000000000000
In [8]:
type(RR(0.5)) # RR(0.5) is the same type as 0.5 from before
Out[8]:
<type 'sage.rings.real_mpfr.RealLiteral'>
In [22]:
??RR
In [33]:
0.5 + 2 # one half as 0.5 is being added to 2 to obtain the real number 2.500..0 in SageMath
Out[33]:
2.50000000000000
In [34]:
0.75 - 0.25 # subtracting 0.75 from 0.25 is the same as subtracting 0.75 from 1/4
Out[34]:
0.500000000000000
In [35]:
0.5 * 0.5 # multiplying 0.5 by 0.5 is the same as 1/2 * 1/2
Out[35]:
0.250000000000000
In [36]:
(2 / 5.0) / 0.2 # dividing 2/5. by 0.2 is the same as (2/5) / (1/5)
Out[36]:
2.00000000000000
In [37]:
0.5^3.0 # exponentiating 0.5 by 3.0 is the same as (1/2)^3
Out[37]:
0.125000000000000

You try

Find the type of 1/2.

Try a few different ways of getting the same result as typing ((((1/5) / (1/10)) * (0.1 * 2/5) + 4/100))*5/(3/5) - this exact expression has already been put in for you in the cell below you could try something just using floating point numbers. Then see how important the parentheses are around rationals when you have an expression like this - try taking some of the parenthesis out and just play with complex expressions like these to get familiar.

In [38]:
((((1/5) / (1/10)) * (0.1 * 2/5) + 4/100))*5/(3/5)
Out[38]:
1.00000000000000
In [39]:
((((1/5) / (1/10)) * (1/10 * 2/5) + 4/100))*5/(3/5)
Out[39]:
1

Example 5: Variables and assignments of numbers and expressions

Loosely speaking one can think of a variable as a way of referring to a memory location used by a computer program. A variable is a symbolic name for this physical location. This memory location contains values, like numbers, text or more complicated types and crucially what is contained in a variable can change based on operations we do to it.

In SageMath, the symbol = is the assignment operator. You can assign a numerical value to a variable in SageMath using the assignment operator. This is a good way to store values you want to use or modify later.

(If you have programmed before using a a language like C or C++ or Java, you'll see that SageMath is a bit different because in SageMath you don't have to say what type of value is going to be assigned to the variable.)

Feel free to take a deeper dive into the computer science concept of assignment.

In [40]:
a = 1    # assign 1 to a variable named a
In [41]:
a        # disclose a  - you need to explicitly do this!
Out[41]:
1

Just typing the name of a variable to get the value works in the SageMath Notebook, but if you are writing a program and you want to output the value of a variable, you'll probably want to use something like the print command.

In [42]:
print(a)

1
In [43]:
b = 2
c = 3
print a, b, c  # print out the values of a and b and c

1 2 3
In [44]:
x=2^(1/2)
x
Out[44]:
sqrt(2)
In [45]:
type(x)   # x is a sage symbolic expression
Out[45]:
<type 'sage.symbolic.expression.Expression'>

Many of the commands in SageMath/Python are "methods" of objects.

That is, we access them by typing:

  • the name of the mathematical object,
  • a dot/period,
  • the name of the method, and
  • parentheses (possibly with an argument).

This is a huge advantage, once you get familiar with it, because it allows you to do only the things that are possible, and all such things. See SageMath programming guide for more details on this.

Let's try to hit the Tab button after the . following x below to view all available methods for x which is currently sqrt(2).

In [ ]:
x.  #  hit the Tab button after the '.' following 'x'
In [ ]:
help(x.n)
# we can use ? after a method to get breif help
In [46]:
x.n(digits=10) # this gives a numerical approximation for x
Out[46]:
1.414213562
In [47]:
s = 1; t = 2; u = 3;
print s + t + u

6
In [48]:
f=(5-3)^(6/2)+3*(7-2) # assign the expression to f
f # disclose f
Out[48]:
23
In [49]:
type(f)
Out[49]:
<type 'sage.rings.integer.Integer'>

You try

Try assigning some values to some variables - you choose what values and you choose what variable names to use. See if you can print out the values you have assigned.

You can reassign different values to variable names. Using SageMath you can also change the type of the values assigned to the variable (not all programming languages allow you to do this).

In [50]:
a = 1
print "Right now, a =", a, "and is of type", type(a) # using , and strings in double quotes print can be more flexible

a = 1/3 # reassign 1/3 to the variable a
print "Now, a =", a, "and is of type", type(a) # note the change in type

Right now, a = 1 and is of type <type 'sage.rings.integer.Integer'>
Now, a = 1/3 and is of type <type 'sage.rings.rational.Rational'>

You try

Assign the value 2 to a variable named x.

On the next line down in the same cell, assign the value 3 to a variable named y.

Then (on a third line) put in an expression which will evaluate x + y

Now try reassigning a different value to x and re-evaluating x + y

Example 6: Strings

Variables can be strings (an not just numbers). Anything you put inside quote marks will be treated as a string by SageMath/Python.

Strings as str and unicode are built-in sequence types for storing strings of bytes and unicode-encoded characters and and operating over them.

In [53]:
myStr = "this is a string"   # assign a string to the variable myStr
myStr                        # disclose myStr
Out[53]:
'this is a string'
In [54]:
type(myStr)                  # check the type for myStr
Out[54]:
<type 'str'>

You can also create a string by enclosing them in single quotes or three consecutive single quotes. In SageMath/Python a character (represented by the char type in languages like C/C++/Scala) is just a string made up of one character.

In [55]:
myStr = 'this is a string'   # assign a string to the variable myStr using single quotes
myStr                        # disclose myStr
Out[55]:
'this is a string'

You can assign values to more than one variable on the same line, by separating the assignment expressions with a semicolon ;. However, it is usually best not to do this because it will make your code easier to read (it is hard to spot the other assignments on a single line after the first one).

In [56]:
myStr = '''this is a string'''   # assign a string to the variable myStr using three consecutive single quotes
myStr                            # disclose myStr
Out[56]:
'this is a string'

Using triple single quotes is especially useful if your string has single or double quotes within it. Triple quotes are often used to create DocString to document code in Pyhton/SageMath.

In [57]:
myStrContainingQuotes = '''this string has "a double quoted sub-string" and some escaped characters: \,', - all OK!'''
myStrContainingQuotes
Out[57]:
'this string has "a double quoted sub-string" and some escaped characters: \\,\', - all OK!'

Str and unicode Strings

In Python/SageMath, we need to be extremely careful with strings. The type 'str' is actually a sequence of bytes while the unicode string of type unicode is a sequence of unicode characters (some of which can be more than a byte in size). See this for an nice clarification of ASCII and unicode (utf-8) encoded strings. So, it is a good habit to convert strings from natural languages that are meant for processing into unicode strings using the decode(utf-8) method right away.

In [58]:
x = 'hi猫' # this is hi (each letter is encoded by one byte) followed by the Chinese character for cat (3 bytes)
type(x)  # x is of type str = sequence of bytes in Python2 / SageMath
Out[58]:
<type 'str'>
In [59]:
len(x) # this is a sequence of five hexadecimal numbers each requiring a byte to represent
Out[59]:
5

Disclosing x below only shows the hexa-decimal numbers 68 69 e7 8c ab, but only h for 68 and i for 69 from ASCII table, are displayed as characters here, while \xe7\x8c\xab are shown as hexadecimal numbers with prefix \x instead of the Chinese character for cat: 猫

In [60]:
x
Out[60]:
'hi\xe7\x8c\xab'
In [61]:
print(x) # printing a string displays the desired if the display is unicode-compatible

hi猫

Generally it is safe to convert strings from natural languages to unicode in Python/SageMath.

In [62]:
y = x.decode('utf-8') # this decodes or converts the sequence of bytes to a sequence of unicode characters
type(y)               # the type of y now is unicode
Out[62]:
<type 'unicode'>
In [63]:
len(y)  # now we have a sequence of just 3 unicode characters as we want
Out[63]:
3

Disclosing y shows the two ASCII character h and i and the Chinese cat character 猫 is specified by the corresponding entry in utf-8 table.

In [64]:
y   # output prepended by u shows it is a unicode sequence as opposed to a str which is a byte sequence
Out[64]:
u'hi\u732b'
In [65]:
print y

hi猫

When programmatically processing sequences of unicode characters it is much safer to work with repr for the canonical string representation of the object.

In [66]:
?repr # gives the canonical string representation of the object
In [67]:
print repr(y)

u'hi\u732b'
In [68]:
print repr(y).decode('unicode_escape')

u'hi猫'

Pride and Prejudice as unicode

We will explore frequencies of strings for the most downloaded book at Project Gutenberg that publishes public domain books online. Currently, books published before 1923 are in the public domain - meaning anyone has the right to copy or use the text in any way.

Pride and Prejudice by Jane Austin had the most number of downloads and it's available from

A quick exploration allows us to see the utf-encoded text here.

For now, we will just show how to download the most popular book from the project and display it's contents for processing down the road.

In [23]:
# this downloads the unicode text of the book from the right url we found at the Gutenberg Project
# and assigns it to a variable named prideAndPrejudiceRaw
from urllib import * 
prideAndPrejudiceRaw = urlopen('http://www.gutenberg.org/files/1342/1342-0.txt').read().decode('utf-8')
prideAndPrejudiceRaw[0:1000] # just showing the first 1000 raw characters of the downloaded book as unicode
Out[23]:
u'\ufeffThe Project Gutenberg EBook of Pride and Prejudice, by Jane Austen\r\n\r\nThis eBook is for the use of anyone anywhere at no cost and with\r\nalmost no restrictions whatsoever.  You may copy it, give it away or\r\nre-use it under the terms of the Project Gutenberg License included\r\nwith this eBook or online at www.gutenberg.org\r\n\r\n\r\nTitle: Pride and Prejudice\r\n\r\nAuthor: Jane Austen\r\n\r\nPosting Date: August 26, 2008 [EBook #1342]\r\nRelease Date: June, 1998\r\nLast Updated: March 10, 2018\r\n\r\nLanguage: English\r\n\r\nCharacter set encoding: UTF-8\r\n\r\n*** START OF THIS PROJECT GUTENBERG EBOOK PRIDE AND PREJUDICE ***\r\n\r\n\r\n\r\n\r\nProduced by Anonymous Volunteers\r\n\r\n\r\n\r\n\r\n\r\nPRIDE AND PREJUDICE\r\n\r\nBy Jane Austen\r\n\r\n\r\n\r\nChapter 1\r\n\r\n\r\nIt is a truth universally acknowledged, that a single man in possession\r\nof a good fortune, must be in want of a wife.\r\n\r\nHowever little known the feelings or views of such a man may be on his\r\nfirst entering a neighbourhood, this truth is so well fixed in the minds\r\nof the surround'
In [24]:
type(prideAndPrejudiceRaw) # this is a sequence of utf-8-encoded characters
Out[24]:
<type 'unicode'>
In [25]:
len(prideAndPrejudiceRaw) 
# the length of the unicode string is about 700 thousand unicode characters
Out[25]:
717617

Next we will show how trivial it is to "read" all the chapters into SageMath/Python using these steps:

  • we use regular expressions via the re library to substitue all occurences of white-space characters like one or more consecutive end-of-line, tabs, white space characters, etc with a single white space,
  • we split by 'Chapter ' into multiple chapters in a list
  • print the first 100 character in each of the first 10 Chapters

(don't worry about the details now - we will revist these in detail later)

In [72]:
myString = "strBlah"
myString
Out[72]:
'strBlah'
In [73]:
myString.split?
In [54]:
import re 
# make a list of chapters
chapterList = re.sub('\\s+', ' ',prideAndPrejudiceRaw).split('Chapter ')[1:10]
for chapter in chapterList:
    print repr(chapter[0:100]).decode('unicode_escape'), '\n';

u'1 It is a truth universally acknowledged, that a single man in possession of a good fortune, must be' 

u'2 Mr. Bennet was among the earliest of those who waited on Mr. Bingley. He had always intended to vi' 

u'3 Not all that Mrs. Bennet, however, with the assistance of her five daughters, could ask on the sub' 

u'4 When Jane and Elizabeth were alone, the former, who had been cautious in her praise of Mr. Bingley' 

u'5 Within a short walk of Longbourn lived a family with whom the Bennets were particularly intimate. ' 

u'6 The ladies of Longbourn soon waited on those of Netherfield. The visit was soon returned in due fo' 

u"7 Mr. Bennet's property consisted almost entirely in an estate of two thousand a year, which, unfort" 

u"8 At five o'clock the two ladies retired to dress, and at half-past six Elizabeth was summoned to di" 

u"9 Elizabeth passed the chief of the night in her sister's room, and in the morning had the pleasure "

As we learn more we will return to this popular book's unicode which is stored in our data directory as data\pride_and_prejudice.txt.

Let us motivate the Python methods we will see soon by using them below to plot the number of occurences of he and she in each of the 61 chapters of the book.

In [51]:
import re 
heList = []
sheList = []
i = 0
# make a list of chapters
chapterList = re.sub('\\s+', ' ',prideAndPrejudiceRaw).split('Chapter ')[1:]
for chapter in chapterList:
    i = i+1 # increment chanpter count
    content = (repr(chapter[0:]).decode('unicode_escape')).lower() # get content as lower-case
    heCount = content.count('he') # count number of 'he' occurrences in the chapter
    sheCount = content.count('she') # count number of 'she' occurrences in the chapter
    heList.append((i,heCount)) # append to heList
    sheList.append((i,sheCount)) # append to sheList
In [68]:
p = points(heList, color='blue',legend_label='he')
p += line(heList, color='blue')
p += points(sheList, color='red',legend_label='she')
p += line(sheList, color='red')
p.axes_labels(['Chapter No.','Frequency'])
p.axes_labels_size(1.0)
p.show(figsize=[8,3])
In [63]:
p. # place the cursor after . and hit Tab button to see options for plotting
In [70]:
type(p) # what we did above is superimposing primitive geometric objects: points and lines
Out[70]:
<class 'sage.plot.graphics.Graphics'>

Assignment Gotcha!

Let's examine the three assignments in the cell below.

The first assignment of x=3 is standard: Python/SageMath chooses a memory location for x and saves the integer value 3 in it.

The second assignment of y=x is more interesting and Pythonic: Instead of finding another location for the variable y and copying the value of 3 in it, Python/SageMath differs from the ways of C/C++. Since both variables will have the same value after the assignment, Python/SageMath lets y point to the memory location of x.

Finally, after the third assignment of y=2, x will be NOT be changed to 2 as because the behavior is not that of a C-pointer. Since x and y will not share the same value anymore, y gets its own memory location, containing 2 and x sticks to the originally assigned value 3.

In [75]:
x=3
print(x) # x is 3
y=x
print(x,y) # x is 3 and y is 
y=2
print(x,y)

3
(3, 3)
(3, 2)

As every instance (object or variable) has an identity or id(), i.e. an integer which is unique within the script or program, we can use id() to understand the above behavior of Python/SageMath assignments.

So, let's have a look at our previous example and see how the identities change with the assignments.

In [76]:
x = 3
print('x and its id() are:')
print(x,id(x))

y = x
print('\ny and its id() are:')
print(y,id(y))

y = 2
print('\nx, y and their id()s are:')
print(x,y,id(x),id(y))

x and its id() are:
(3, 140085314697904)

y and its id() are:
(3, 140085314697904)

x, y and their id()s are:
(3, 2, 140085314697904, 140085314531216)

Example 6: Truth statements and Boolean values

Consider statements like "Today is Friday" or "2 is greater than 1" or " 1 equals 1": statements which are either true or not true (i.e., false). SageMath has two values, True and False which you'll meet in this situation. These value are called Booleans values, or values of the type Boolean.

In SageMath, we can express statements like "2 is greater than 1" or " 1 equals 1" with relational operators, also known as value comparison operators. Have a look at the list below.

  • < Less than
  • > Greater than
  • <= Less than or equal to
  • >= Greater than or equal to
  • == Equal to.
  • != Not equal to

Lets try some really simple truth statements.

In [77]:
1 < 1          # 1 is less than 1
Out[77]:
False

Let us evaluate the following statement.

In [78]:
1 <= 1  # 1 is less than or equal to 1
Out[78]:
True

We can use these operators on variables as well as on values. Again, try assigning different values to x and y, or try using different operators, if you want to.

In [79]:
x = 1          # assign the value 1 to x             
y = 2          # assign the value 2 to y 
x == y         # evaluate the truth statement "x is equal to y"
Out[79]:
False

Note that when we check if something equals something else, we use ==, a double equals sign. This is because =, a single equals sign, is the assignment operator we talked about above. Therefore, to test if x equals y we can't write x = y because this would assign y to x, instead we use the equality operator == and write x == y.

We can also assign a Boolean value to a variable.

In [80]:
# Using the same x and y as above
myBoolean = (x == y)   # assign the result of x == y to the variable myBoolean
myBoolean              # disclose myBoolean
Out[80]:
False
In [81]:
type(myBoolean)        # check the type of myBoolean
Out[81]:
<type 'bool'>

If we want to check if two things are not equal we use !=. As we would expect, it gives us the opposite of testing for equality:

In [82]:
x != y                 # evaluate the truth statement "x is not equal to y"
Out[82]:
True
In [83]:
print(x,y)             # Let's print x and y to make sure the above statement makes sense 

(1, 2)

You try

Try assigning some values to two variables - you choose what values and you choose what variable names to use. Try some truth statements to check if they are equal, or one is less than the other.

You try

Try some strings (we looked at strings briefly in Example 5 above). Can you check if two strings are equal? Can you check if one string is less than (<) another string. How do you think that Sage is ordering strings (try comparing "fred" and "freddy", for example)?

In [84]:
'raazb' <= 'raaza'
Out[84]:
False
In [85]:
x = [1]
y = x
y[0] = 5
print x
print(x,id(x),y,id(y))

[5]
([5], 140085306496640, [5], 140085306496640)

Example 7: Mathematical constants

Sage has reserved words that are defined as common mathematical constants. For example, pi and e behave as you expect. Numerical approximations can be obtained using the .n() method, as before.

In [86]:
print pi, "~", pi.n()   # print a numerical approximation of the mathematical constant pi
print e, "~", e.n()     # print a numerical approximation of the mathematical constant e
print I, "~", I.n()     # print a numerical approximation of the imaginary number sqrt(-1)

pi ~ 3.14159265358979
e ~ 2.71828182845905
I ~ 1.00000000000000*I
In [89]:
(pi/e).n(digits=100)   # print the first 100 digits of pi/e
Out[89]:
1.155727349790921717910093183312696299120851023164415820499706535327288631840916939440188434235673559
In [90]:
e^(i*pi)+1          # Euler's identity symbolically - see https://en.wikipedia.org/wiki/Euler%27s_identity
Out[90]:
0

Example 8: SageMath number types and Python number types

We showed how you can find the type of a number value and we demonstrated that by default, SageMath makes 'real' numbers like 3.1 into Sage real literals (sage.rings.real_mpfr.RealLiteral).

If you were just using Python (the programming language underlying most of SageMath) then a value like 3.1 would be a floating point number or float type. Python has some interesting extra operators that you can use with Python floating point numbers, which also work with the Sage rings integer type but not with Sage real literals.

In [91]:
X = 3.1 # convert to default Sage real literal 3.1 
type(X)
Out[91]:
<type 'sage.rings.real_mpfr.RealLiteral'>
In [92]:
X = float(3.1) # convert the default Sage real literal 3.1 to a float 3.1
type(X)
Out[92]:
<type 'float'>

Floor Division (//) - The division of operands where the result is the quotient in which the digits after the decimal point are removed - the result is floored, i.e., rounded towards negative infinity: examples: 9//2 = 4 and 9.0//2.0 = 4.0, -11//3 = -4, -11.0//3 = -4.0

In [93]:
3 // 2 # floor division
Out[93]:
1
In [94]:
3.3 // 2.0 # this will give error - floor division is undefined for Sage real literals

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-94-cfec9a575a73> in <module>()
----> 1 RealNumber('3.3') // RealNumber('2.0') # this will give error - floor division is undefined for Sage real literals

/home/raazesh/all/software/sage/SageMath/src/sage/structure/element.pyx in sage.structure.element.Element.__floordiv__ (build/cythonized/sage/structure/element.c:13318)()
   1814         cdef int cl = classify_elements(left, right)
   1815         if HAVE_SAME_PARENT(cl):
-> 1816             return (<Element>left)._floordiv_(right)
   1817         if BOTH_ARE_ELEMENT(cl):
   1818             return coercion_model.bin_op(left, right, floordiv)

/home/raazesh/all/software/sage/SageMath/src/sage/structure/element.pyx in sage.structure.element.Element._floordiv_ (build/cythonized/sage/structure/element.c:13639)()
   1849             python_op = (<object>self)._floordiv_
   1850         except AttributeError:
-> 1851             raise bin_op_exception('//', self, other)
   1852         else:
   1853             return python_op(other)

TypeError: unsupported operand parent(s) for //: 'Real Field with 53 bits of precision' and 'Real Field with 53 bits of precision'
In [95]:
float(3.5) // float(2.0)
Out[95]:
1.0

Similarly, we have the light-weight Python integer type int that we may want instead of SageMath integer type for non-mathematical operations.

In [96]:
type(3) # the default Sage rings integer type
Out[96]:
<type 'sage.rings.integer.Integer'>
In [97]:
X = int(3) # conversion to a plain Python integer type
type(X)
Out[97]:
<type 'int'>
In [98]:
3/2 # see the result you get when dividing one default Sage rings integer type by another
Out[98]:
3/2

One of the differences of SageMath rings integers to plain Python integers is that result of dividing one SageMath rings integer by another is a rational. This probably seems very sensible, but it is not what happens at the moment with Python integers.

In [99]:
int(7)/int(2) # division using python integers is "floor division"
Out[99]:
3

We showed the .n() method. If X is some Sage real literal and we use X.n(20) we will be asking for 20 bits of precision, which is about how many bits in the computer's memory will be allocated to hold the number. If we ask for X.n(digits=20) will be asking for 20 digits of precision, which is not the same thing. Also note that 20 digits of precision does not mean showing the number to 20 decimal places, it means all the digits including those in front of the decimal point.

In [ ]:
help(n) # always ask for help when you need it - or lookup in help menu above
In [100]:
X=3.55555555
X.n(digits = 3)
Out[100]:
3.56
In [101]:
X.n(3) # this will use 3 bits of precision
Out[101]:
3.5
In [102]:
round(X,3)
Out[102]:
3.556
In [96]:
?round # this opens a window with help information that can be closed

If you want to actually round a number to a specific number of decimal places, you can also use the round(...) function.

For deeper dive see documents on Python Numeric Types and SageMath Numeric Types

Sets

Set theory is at the very foundation in modern mathematics and is necessary to understand the mathematical notions of probability and statistics. We will take a practical mathemtical tour of the essential concepts from set theory that a data scientist needs to understand and build probabilistic models from the data using statistical principles.

In [107]:
showURL("https://en.wikipedia.org/wiki/Set_(mathematics)",500)
Out[107]:

Essentials of Set Theory for Probability and Statistics

These are black-board lectures typeset here.

Let us learn or recall elementary set theory. Sets are perhaps the most fundamental concept in mathematics.

Definitions

Set is a collection of distinct elements.

We write a set by enclosing its elements with curly brackets. Let us see some example next.

  • The collection of $\star$ and $\circ$ is $\{\star,\circ\}$.
  • We can name the set $\{\star,\circ\}$ by the letter $A$ and write $$A=\{\star,\circ\}.$$
  • Question: Is $\{\star,\star,\circ\}$ a set?
  • A set of letters and numbers that I like is $\{b,d,6,p,q,9\}$.
  • The set of first five Greek alphabets is $\{\alpha,\beta,\gamma,\delta,\epsilon\}$.

The set that contains no elements is the empty set. It is denoted by $$\boxed{\emptyset = \{\}} \ .$$

We say an element belongs to or does not belong to a set with the binary operators $$\boxed{\in \ \text{or} \ \notin} \ .$$ For example,

  • $\star \in \{\star,\circ\}$ but the element $\otimes \notin \{\star,\circ\}$
  • $b \in \{b,d,6,p,q,9\}$ but $8 \notin \{b,d,6,p,q,9\}$
  • Question: Is $9 \in \{3,4,1,5,2,8,6,7\}$?

We say a set $C$ is a subset of a set $D$ and write

$$\boxed{C \subset D}$$

if every element of $C$ is also an element of $D$. For example,

  • $\{\star\} \subset \{\star,\circ\}$
  • Question: Is $\{6,9\}\subset \{b,d,6,p,q,9\}$?

Set Operations

We can add distinct new elements to an existing set by union operation denoted by $\cup$ symbol.

For example

  • $\{\circ, \bullet\} \cup \{\star\} = \{\circ,\bullet,\star\}$
  • Question: $\{\circ, \bullet\} \cup \{\bullet\} = \quad$?

More formally, we write the union of two sets $A$ and $B$ as $$\boxed{A \cup B = \{x: x \in A \ \text{or} \ x \in B \}} \ .$$

The symbols above are read as $A$ union $B$ is equal to the set of all $x$ such that $x$ belongs to $A$ or $x$ belongs to $B$ and simply means that $A$ union $B$ or $A \cup B$ is the set of elements that belong to $A$ or $B$.

Similarly, the intersection of two sets $A$ and $B$ written as $$\boxed{A \cap B = \{x: x \in A \ \text{and} \ x \in B \}} $$ means $A$ intersection $B$ is the set of elements that belong to both $A$ and $B$.

For example

  • $\{\circ, \bullet\} \cap \{\circ\} = \{\circ\}$
  • $\{\circ, \bullet\} \cap \{\bullet\} = \{\bullet\}$
  • $\{\circ\} \cap \{a,b,c,d\}=\emptyset$

The set difference of two sets $A$ and $B$ written as $$\boxed{A \setminus B = \{x: x \in A \ \text{and} \ x \notin B \}} $$ means $A \setminus B$ is the set of elements that belong to $A$ and not belong to $B$.

For example

  • $\{\circ, \bullet\} \setminus \{\circ\} = \{\bullet\}$
  • $\{\circ, \bullet\} \setminus \{\bullet\} = \{\circ\}$
  • $\{a,b,c,d\} \setminus \{a,b,c,d\}=\emptyset$

The equality of two sets $A$ and $B$ is defined in terms of subsets as follows: $$\boxed{A = B \quad \text{if and only if} \quad A \subset B \ \text{and} \ B \subset A} \ .$$

Two sets $A$ anb $B$ are said to be disjoint if $$\boxed{ A \cap B = \emptyset} \ .$$

Given a universal set $\Omega$, we define the complement of a subset $A$ of the universal set by $$\boxed{A^c = \Omega \setminus A = \{x: x \in \Omega \ \text{and} \ x \notin A\}} \ .$$

An Interactive Venn Diagram

Let us gain more intuition by seeing the unions and intersections of sets interactively. The following interact is from interact/misc page of Sage Wiki.

In [108]:
# ignore this code for now and focus on the interact in the output cell
def f(s, braces=True): 
    t = ', '.join(sorted(list(s)))
    if braces: return '{' + t + '}'
    return t
def g(s): return set(str(s).replace(',',' ').split())

@interact
def _(X='1,2,3,a', Y='2,a,3,4,apple', Z='a,b,10,apple'):
    S = [g(X), g(Y), g(Z)]
    X,Y,Z = S
    XY = X & Y
    XZ = X & Z
    YZ = Y & Z
    XYZ = XY & Z
    pretty_print(html('<center>'))
    pretty_print(html("$X \cap Y$ = %s"%f(XY)))
    pretty_print(html("$X \cap Z$ = %s"%f(XZ)))
    pretty_print(html("$Y \cap Z$ = %s"%f(YZ)))
    pretty_print(html("$X \cap Y \cap Z$ = %s"%f(XYZ)))
    pretty_print(html('</center>'))
    centers = [(cos(n*2*pi/3), sin(n*2*pi/3)) for n in [0,1,2]]
    scale = 1.7
    clr = ['yellow', 'blue', 'green']
    G = Graphics()
    for i in range(len(S)):
        G += circle(centers[i], scale, rgbcolor=clr[i], 
             fill=True, alpha=0.3)
    for i in range(len(S)):
        G += circle(centers[i], scale, rgbcolor='black')

    # Plot what is in one but neither other
    for i in range(len(S)):
        Z = set(S[i])
        for j in range(1,len(S)):
            Z = Z.difference(S[(i+j)%3])
        G += text(f(Z,braces=False), (1.5*centers[i][0],1.7*centers[i][1]), rgbcolor='black')


    # Plot pairs of intersections
    for i in range(len(S)):
        Z = (set(S[i]) & S[(i+1)%3]) - set(XYZ)
        C = (1.3*cos(i*2*pi/3 + pi/3), 1.3*sin(i*2*pi/3 + pi/3))
        G += text(f(Z,braces=False), C, rgbcolor='black')

    # Plot intersection of all three
    G += text(f(XYZ,braces=False), (0,0), rgbcolor='black')

    # Show it
    G.show(aspect_ratio=1, axes=False)

Create and manipulate sets in SageMath.

Example 0: Lists before Sets

A list is a sequential collection that we will revisit in detail soon. For now, we just need to know that we can create a list by using delimiter , between items and by wrapping with left and right square brackets: [ and ]. For example, the following is a list of 4 integers:

In [105]:
[1,2,3,4]
Out[105]:
[1, 2, 3, 4]
In [106]:
myList = [1,2,3,4]    # we can assign the list to a variable myList
print(myList)         # print myList 
type(myList)          # and ask for its type 

[1, 2, 3, 4]
Out[106]:
<type 'list'>

List is one of the most primitive data structures and has a long history in a popular computer programming language called LISP - originally created as a practical mathematical notation for computer programs.

For now, we just use lists to create sets.

Example 1: Making sets

In SageMath, you do have to specifically say that you want a set when you make it.

Do the following in SageMath/Python to create the mathematical set:

$$X = \{1,2,3,4\}$$

In [107]:
X = set([1, 2, 3, 4])  # make the set X={1,2,3,4} from the List [1,2,3,4]
X                      # disclose X
Out[107]:
{1, 2, 3, 4}
In [108]:
type(X)                # what is the type of X
Out[108]:
<type 'set'>

This is a specialized datatype in Python and more details can be found in Python docs: https://docs.python.org/2/library/datatypes.html

Do the following in SageMath/Python to find out if:

$$4 \in X$$

i.e., if $4$ is an element of $X$ or if $4$ is an element of $X$ or $4$ is in $X$.

In [109]:
4 in X                 # 'is 4 in X?'
Out[109]:
True
In [110]:
5 in X                 # 'is 5 in X?'
Out[110]:
False
In [111]:
Y = set([1, 2])        # make the set Y={1,2}
Y                      # disclose Y
Out[111]:
{1, 2}
In [112]:
4 not in Y             # 'is 4 not in Y?'
Out[112]:
True
In [113]:
1 not in Y             # 'is 1 not in Y?'
Out[113]:
False

We can add new elements to a set.

In [114]:
X.add(5)     # add 5 to the set X
X
Out[114]:
{1, 2, 3, 4, 5}

But remember from the mathematical exposition above that sets contain distinct elements.

In [115]:
X.add(1)     # try adding another 1 to the set X
X
Out[115]:
{1, 2, 3, 4, 5}

You try

Try making the set $Z=\{4,5,6,7\}$ next. The instructions are in the two cells below.

In [116]:
# Write in the expression to make set Z ={4, 5, 6, 7} 
# (press ENTER at the end of this line to get a new line)
Z = set([4,5,6,7]) #([4],[5],[6,7]))
In [117]:
# Check if 4 is in Z 
4 in Z
# (press ENTER at the end of this line to get a new line)
Out[117]:
True

Make a set with the value 2/5 (as a rational) in it. Try adding 0.4 (as a floating point number) to the set.

Does SageMath do what you expect?

Example 2: Subsets

In lectures we talked about subsets of sets.

Recall that Y is a subset of X if each element in Y is also in X.

In [118]:
print "X is", X
print "Y is", Y
print "Is Y a subset of X?"
Y <= X                       # 'is Y a subset of X?'

X is set([1, 2, 3, 4, 5])
Y is set([1, 2])
Is Y a subset of X?
Out[118]:
True

If you have time: We say Y is a proper subset of X if all the elements in Y are also in X but there is at least one element in X that it is not in Y. If X is a (proper) subset of Y, then we also say that Y is a (proper) superset of X.

In [119]:
X < X     # 'is X a proper subset of itself?'
Out[119]:
False
In [120]:
X > Y     # 'is X a proper superset of Y?'
Out[120]:
True
In [121]:
X > X     # 'is X a proper superset of itself?'
Out[121]:
False
In [122]:
X >= Y     # 'is X a superset of Y?' is the same as 'is Y a subset of X?'
Out[122]:
True

Example 3: More set operations

Now let's have a look at the other set operations we talked about above: intersection, union, and difference.

Recall that the intersection of X and Y is the set of elements that are in both X and Y.

In [123]:
X & Y    # '&' is the intersection operator
Out[123]:
{1, 2}

The union of X and Y is the set of elements that are in either X or Y.

In [124]:
X | Y    # '|' is the union operator
Out[124]:
{1, 2, 3, 4, 5}

The set difference between X and Y is the set of elements in X that are not in Y.

In [125]:
X - Y    # '-' is the set difference operator
Out[125]:
{3, 4, 5}

You try

Try some more work with sets of strings below.

In [126]:
fruit = set(['orange', 'banana', 'apple'])
fruit
Out[126]:
{'apple', 'banana', 'orange'}
In [127]:
colours = set(['red', 'green', 'blue', 'orange'])
colours
Out[127]:
{'blue', 'green', 'orange', 'red'}

Fruit and colours are different to us as people, but to the computer, the string 'orange' is just the string 'orange' whether it is in a set called fruit or a set called colours.

In [128]:
print "fruit intersection colours is", fruit & colours    
print "fruit union colours is", fruit | colours    
print "fruit - colours is", fruit - colours    
print "colours - fruit is", colours - fruit

fruit intersection colours is set(['orange'])
fruit union colours is set(['blue', 'green', 'apple', 'orange', 'banana', 'red'])
fruit - colours is set(['banana', 'apple'])
colours - fruit is set(['blue', 'green', 'red'])

Try a few other simple subset examples - make up your own sets and try some intersections, unions, and set difference operations. The best way to try new possible operations on a set such as X we just created is to type a period after X and hit <TAB> key. THis will bring up all the possible methods you can call on the set X.

In [129]:
mySet = set([1,2,3,4,5,6,7,8,9])
In [ ]:
mySet.         # try placing the cursor after the dot and hit <TAB> key
In [ ]:
?mySet.add     # you can get help on a method by prepending a question mark

Infact, there are two ways to make sets in SageMath. We have so far used the python set to make a set.

However we can use the SageMath Set to maka sets too. SageMath Set is more mathematically consisitent. If you are interested in the SageMath Set go to the source and work through the SageMath reference on Sets.

But, first let us appreciate the difference between Python set and SageMath Set!

In [130]:
X = set([1, 2, 3, 4])  # make the set X={1,2,3,4} with python set
X                      # disclose X
Out[130]:
{1, 2, 3, 4}
In [131]:
type(X)                # this is the set in python
Out[131]:
<type 'set'>
In [132]:
anotherX = Set([1, 2, 3, 4])   # make the set anotherX={1,2,3,4} in SAGE Set
anotherX #disclose it
Out[132]:
{1, 2, 3, 4}
In [133]:
type(anotherX)                 # this is the set in SAGE and is more mathy
Out[133]:
<class 'sage.sets.set.Set_object_enumerated_with_category'>
In [ ]:
anotherX. # see what methods are available in SageMath Set

Example 4

Python also provides something called a frozenset, which you can't change like an ordinary set.

In [134]:
aFrozenSet = frozenset([2/5, 0.2, 1/7, 0.1])
aFrozenSet
Out[134]:
frozenset({0.100000000000000, 1/7, 0.200000000000000, 2/5})
In [135]:
aFrozenSet.add(0.3) # This should give an error

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-135-a9833ac872d1> in <module>()
----> 1 aFrozenSet.add(RealNumber('0.3')) # This should give an error

AttributeError: 'frozenset' object has no attribute 'add'

Under the Set's Hood

The key is to remember that sets are unordered: a set {1, 2, 3} is the same as the set {2, 1, 3} is the same as the set {3, 1, 2} ... (soon we will talk about ordered collections like lists where order is important - but for the moment just remember that a set is about collections of unique values or objects). Remember also that the (lower case s) set is a python type (in contrast to the 'more mathy' SageMath Set with a capital S). For many 'non-mathy' computing purposes, what matters about a set is the speed of being able to find things in the set without making it expensive (in computer power) to add and remove things, and the best way of doing this (invisible to the programmer actually using the set) is to base the set on a hash table. In practical terms what this means is that the ordering that Sage uses to actually display a set is related to the hash values it has given to the elements in the set which may be totally unrelated to what you or I would think of as any obvious ordering based on the magnitude of the values in the set, or the order in which we put them in, etc. Remember, you don't need to know anything about hash values and the 'under-the-hood' construction of sets for this course - this explanation is just to give a bit more background.

Using sets to find the words with 15 or more characters in P&P

In [99]:
# This time we are loading the file from our data directory
with open('data/pride_and_prejudice.txt', 'r') as myfile:
  prideAndPrejudiceRaw = myfile.read().decode('utf-8')

import re 
bigWordsSet = set([]) # start an empty set
# make a list of chapters
chapterList = re.sub('\\s+', ' ',prideAndPrejudiceRaw).split('Chapter ')[1:]
for chapter in chapterList:
    content = (repr(chapter[0:]).decode('unicode_escape')).lower() # get content as lower-case
    bigWordsSet = bigWordsSet | \
                    set([x for x in content.split(' ') if (len(x)>14 and x.isalpha())])
    # union with list-comprehended set - soon you will get this!
In [101]:
bigWordsSet # set of big words with > 14 characters in any chapter of P&P
Out[101]:
{u'accomplishments',
 u'acknowledgments',
 u'communicativeness',
 u'condescendingly',
 u'congratulations',
 u'conscientiously',
 u'disappointments',
 u'discontentedness',
 u'disinterestedness',
 u'inconsistencies',
 u'merchantibility',
 u'misrepresentation',
 u'recommendations',
 u'representations',
 u'superciliousness',
 u'thoughtlessness',
 u'uncompanionable',
 u'unenforceability'}

Note that there are much more convenient methods to read text files that have been structured into a certain form. You are purposely being keept much closer to the raw data here. Starting directly with convenient high-level methods is not a good way to appreciate wrangling data closer to the source. Furthermore, it is easier to pick up the high-level methods later on.