Lab02

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 1/29 Lab 2: Table Operations and Arrays Welcome to Lab 2! This week, we'll learn how to import a module and practice table operations! we'll also see how to work with arrays of data, such as all the numbers between 0 and 100 or all the words in the chapter of a book. Lastly, we'll create tables and practice analyzing them with our knowledge of table operations. First, set up the imports by running the cell below. 1. Review: The building blocks of Python code The two building blocks of Python code are expressions and statements . An expression is a piece of code that is self-contained, meaning it would make sense to write it on a line by itself, and usually evaluates to a value. Here are two expressions that both evaluate to 3: 3 5 - 2 One important type of expression is the call expression . A call expression begins with the name of a function and is followed by the argument(s) of that function in parentheses. The function returns some value, based on its arguments. Some important mathematical functions are listed below. Function Description abs Returns the absolute value of its argument max Returns the maximum of all its arguments min Returns the minimum of all its arguments pow Raises its first argument to the power of its second argument round Rounds its argument to the nearest integer Here are two call expressions that both evaluate to 3: In [5]: # Just run this cell import numpy as np from datascience import *

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 2/29 abs(2 - 5) max(round(2.8), min(pow(2, 10), -1 * pow(2, 10))) The expression 2 - 5 and the two call expressions given above are examples of compound expressions , meaning that they are actually combinations of several smaller expressions. 2 - 5 combines the expressions 2 and 5 by subtraction. In this case, 2 and 5 are called subexpressions because they're expressions that are part of a larger expression. A statement is a whole line of code. Some statements are just expressions. The expressions listed above are examples. Other statements make something happen rather than having a value . For example, an assignment statement assigns a value to a name. A good way to think about this is that we're evaluating the right-hand side of the equals sign and assigning it to the left-hand side . Here are some assignment statements: height = 1.3 the_number_five = abs(-5) absolute_height_difference = abs(height - 1.688) An important idea in programming is that large, interesting things can be built by combining many simple, uninteresting things. The key to understanding a complicated piece of code is breaking it down into its simple components. For example, a lot is going on in the last statement above, but it's really just a combination of a few things. This picture describes what's going on. **Question 1.1.** In the next cell, assign the name new_year to the larger number among the following two numbers: 1. the absolute value of , and 2. . Try to use just one statement (one line of code). 2020 In [7]: new_year = ( 5 * 13 * 31 ) + 5 new_year Out[7]: In [8]: # TEST new_year == 2020

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 4/29 Rather than copy-pasting, Python allows us to import modules . A module is a file with Python code that has defined variables and functions. By importing a module, we are able to use its code in our own notebook. Python includes many useful modules that are just an import away. We'll look at the math module as a first example. The math module is extremely useful in computing mathematical expressions in Python. Suppose we want to very accurately compute the area of a circle with a radius of 5 meters. For that, we need the constant , which is roughly 3.14. Conveniently, the math module has pi defined for us: 78.53981633974483 In the code above, the line import math imports the math module. This statement creates a module and then assigns the name math to that module. We are now able to access any variables or functions defined within math by typing the name of the module followed by a dot, then followed by the name of the variable or function we want. <module name>.<name> **Question 2.1.** The module math also provides the name e for the base of the natural logarithm, which is roughly 2.71. Compute , giving it the name near_twenty . Remember: You can access pi from the math module as well! 19.99909997918947 True In [6]: import math radius = 5 area_of_circle = radius ** 2 * math . pi area_of_circle Out[6]: In [9]: near_twenty = ( math . e ** math . pi ) - math . pi near_twenty Out[9]: In [10]: # TEST round ( near_twenty , 8 ) == 19.99909998 Out[10]:

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 5/29 Source Explaination 2.1. Accessing functions In the question above, you accessed variables within the math module. Modules also define functions . For example, math provides the name sin for the sine function. Having imported math already, we can write math.sin(3) to compute the sine of 3. (Note that this sine function considers its argument to be in radians , not degrees. 180 degrees are equivalent to radians.) **Question 2.1.1.** A -radian (45-degree) angle forms a right triangle with equal base and height, pictured below. If the hypotenuse (the radius of the circle in the picture) is 1, then the height is . Compute that value using sin and pi from the math module. Give the result the name sine_of_pi_over_four . Source In [11]: sine_of_pi_over_four = math . sin ( math . pi / 4 ) sine_of_pi_over_four

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 7/29 Run the next cell to load the farmers_markets table. Let's examine our table to see what data it contains. **Question 3.1.** Use the method show to display the first 5 rows of farmers_markets . Note: The terms "method" and "function" are technically not the same thing, but for the purposes of this course, we will use them interchangeably. Hint: tbl.show(3) will show the first 3 rows of tbl . Additionally, make sure not to call .show() without an argument, as this will crash your kernel! FMID MarketName street city County State zip x y 1012063 Caledonia Farmers Market Association - Danville nan Danville Caledonia Vermont 05828 -72.1403 44.411 https://site 1011871 Stearns Homestead Farmers' Market 6975 Ridge Road Parma Cuyahoga Ohio 44130 -81.7286 41.3751 1011878 100 Mile Market 507 Harrison St Kalamazoo Kalamazoo Michigan 49007 -85.5749 42.296 1009364 106 S. Main Street Farmers Market 106 S. Main Street Six Mile nan South Carolina 29682 -82.8187 34.8042 1010691 10th Steet Community Farmers Market 10th Street and Poplar Lamar Barton Missouri 64759 -94.2746 37.4956 ... (8541 rows omitted) Notice that some of the values in this table are missing, as denoted by "nan." This means either that the value is not available (e.g. if we don’t know the market’s street address) or not applicable (e.g. if the market doesn’t have a street address). You'll also notice that the table has a large number of columns in it! num_columns In [15]: # Just run this cell farmers_markets = Table . read_table ( 'farmers_markets.csv' ) In [16]: farmers_markets . show ( 5 )

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 8/29 The table property num_columns returns the number of columns in a table. (A "property" is just a method that doesn't need to be called by adding parentheses.) Example call: <tbl>.num_columns **Question 3.2.** Use num_columns to find the number of columns in our farmers' markets dataset. Assign the number of columns to num_farmers_markets_columns . The table has 59 columns in it! True num_rows Similarly, the property num_rows tells you how many rows are in a table. The table has 8546 rows in it! select Most of the columns are about particular products -- whether the market sells tofu, pet food, etc. If we're not interested in that information, it just makes the table difficult to read. This comes up more than you might think, because people who collect and publish data may not know ahead of time what people will want to do with it. In such situations, we can use the table method select to choose only the columns that we want in a particular table. It takes any number of arguments. Each should be the name of a column in the table. It returns a new table with only those columns in it. The columns are in the order in which they were listed as arguments . For example, the value of farmers_markets.select("MarketName", "State") is a table with only the name and the state of each farmers' market in farmers_markets . **Question 3.3.** Use select to create a table with only the name, city, state, latitude ( y ), and longitude ( x ) of each market. Call that new table farmers_markets_locations . In [17]: num_farmers_markets_columns = farmers_markets . num_columns print ( "The table has" , num_farmers_markets_columns , "columns in it!" ) In [18]: # TEST num_farmers_markets_columns == 59 Out[18]: In [19]: # Just run this cell num_farmers_markets_rows = farmers_markets . num_rows print ( "The table has" , num_farmers_markets_rows , "rows in it!" )

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 10/29 MarketName street city County State zip x y Caledonia Farmers Market Association - Danville nan Danville Caledonia Vermont 05828 -72.1403 44.411 https://sites.goo Stearns Homestead Farmers' Market 6975 Ridge Road Parma Cuyahoga Ohio 44130 -81.7286 41.3751 100 Mile Market 507 Harrison St Kalamazoo Kalamazoo Michigan 49007 -85.5749 42.296 106 S. Main Street Farmers Market 106 S. Main Street Six Mile nan South Carolina 29682 -82.8187 34.8042 ht 10th Steet Community Farmers Market 10th Street and Poplar Lamar Barton Missouri 64759 -94.2746 37.4956 112st Madison Avenue 112th Madison Avenue New York New York New York 10029 -73.9493 40.7939 12 South Farmers Market 3000 Granny White Pike Nashville Davidson Tennessee 37204 -86.7907 36.1184 h 125th Street Fresh Connect Farmers' Market 163 West 125th Street and Adam Clayton Powell, Jr. Blvd. New York New York New York 10027 -73.9482 40.809 http:/ 12th & Brandywine Urban Farm Market 12th & Brandywine Streets Wilmington New Castle Delaware 19801 -75.5345 39.7421 14&U Farmers' Market 1400 U Street NW Washington District of Columbia District of Columbia 20009 -77.0321 38.917 ... (8536 rows omitted) Out[23]: In [24]: # TEST farmers_markets_without_fmid . num_columns == 57

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 11/29 True ['Bakedgoods', 'Beans', 'Cheese', 'Coffee', 'County', 'Crafts', 'Credit', 'Eggs', 'Fa cebook', 'Flowers', 'Fruits', 'Grains', 'Herbs', 'Honey', 'Jams', 'Juices', 'Locatio n', 'Maple', 'MarketName', 'Meat', 'Mushrooms', 'Nursery', 'Nuts', 'Organic', 'OtherM edia', 'PetFood', 'Plants', 'Poultry', 'Prepared', 'SFMNP', 'SNAP', 'Seafood', 'Seaso n1Date', 'Season1Time', 'Season2Date', 'Season2Time', 'Season3Date', 'Season3Time', 'Season4Date', 'Season4Time', 'Soap', 'State', 'Tofu', 'Trees', 'Twitter', 'Vegetable s', 'WIC', 'WICcash', 'Website', 'WildHarvested', 'Wine', 'Youtube', 'city', 'stree t', 'x', 'y', 'zip'] Now, suppose we want to answer some questions about farmers' markets in the US. For example, which market(s) have the largest longitude (given by the x column)? To answer this, we'll sort farmers_markets_locations by longitude. MarketName city State x y Trapper Creek Farmers Market Trapper Creek Alaska -166.54 53.8748 Kekaha Neighborhood Center (Sunshine Markets) Kekaha Hawaii -159.718 21.9704 Hanapepe Park (Sunshine Markets) Hanapepe Hawaii -159.588 21.9101 Kalaheo Neighborhood Center (Sunshine Markets) Kalaheo Hawaii -159.527 21.9251 Hawaiian Farmers of Hanalei Hanalei Hawaii -159.514 22.2033 Hanalei Saturday Farmers Market Hanalei Hawaii -159.492 22.2042 Kauai Culinary Market Koloa Hawaii -159.469 21.9067 Koloa Ball Park (Knudsen) (Sunshine Markets) Koloa Hawaii -159.465 21.9081 West Kauai Agricultural Association Poipu Hawaii -159.435 21.8815 Kilauea Neighborhood Center (Sunshine Markets) Kilauea Hawaii -159.406 22.2112 ... (8536 rows omitted) Oops, that didn't answer our question because we sorted from smallest to largest longitude. To look at the largest longitudes, we'll have to sort in reverse order. Out[24]: In [25]: # TEST print ( sorted ( farmers_markets_without_fmid . labels )) In [26]: farmers_markets_locations . sort ( 'x' ) Out[26]: In [27]: farmers_markets_locations . sort ( 'x' , descending = True )

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 13/29 MarketName city State x y Tanana Valley Farmers Market Fairbanks Alaska -147.781 64.8628 Ester Community Market Ester Alaska -148.01 64.8459 Fairbanks Downtown Market Fairbanks Alaska -147.72 64.8444 Nenana Open Air Market Nenana Alaska -149.096 64.5566 Highway's End Farmers' Market Delta Junction Alaska -145.733 64.0385 MountainTraders Talkeetna Alaska -150.118 62.3231 Talkeetna Farmers Market Talkeetna Alaska -150.118 62.3228 Denali Farmers Market Anchorage Alaska -150.234 62.3163 Kenny Lake Harvest II Valdez Alaska -145.476 62.1079 Copper Valley Community Market Copper Valley Alaska -145.444 62.0879 ... (8536 rows omitted) True True Now let's say we want a table of all farmers' markets in California. Sorting won't help us much here because California is closer to the middle of the dataset. Instead, we use the table method where . Out[28]: In [29]: # TEST type ( farmers_markets_locations_by_latitude ) == tables . Table Out[29]: In [30]: # TEST list ( farmers_markets_locations_by_latitude . column ( 'y' ) . take ( range ( 3 ))) == [ 64.86275 , 6 Out[30]: In [31]: california_farmers_markets = farmers_markets_locations . where ( 'State' , are . equal_to ( 'Ca california_farmers_markets

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 14/29 MarketName city State x y Adelanto Stadium Farmers Market Victorville California -117.405 34.5593 Alameda Farmers' Market Alameda California -122.277 37.7742 Alisal Certified Farmers' Market Salinas California -121.634 36.6733 Altadena Farmers' Market Altadena California -118.158 34.2004 Alum Rock Village Farmers' Market San Jose California -121.833 37.3678 Amador Farmers' Market-- Jackson Jackson California -120.774 38.3488 Amador Farmers' Market-- Pine Grove Pine Grove California -120.774 38.3488 Amador Farmers' Market-- Sutter Creek Sutter Creek California -120.774 38.3488 Anderson Happy Valley Farmers Market Anderson California -122.408 40.4487 Angels Camp Farmers Market-Fresh Fridays Angels Camp California -120.543 38.0722 ... (745 rows omitted) Ignore the syntax for the moment. Instead, try to read that line like this: Assign the name california_farmers_markets to a table whose rows are the rows in the farmers_markets_locations table where the 'State' s are equal to California . where Now let's dive into the details a bit more. where takes 2 arguments: 1. The name of a column. where finds rows where that column's values meet some criterion. 2. A predicate that describes the criterion that the column needs to meet. The predicate in the example above called the function are.equal_to with the value we wanted, 'California'. We'll see other predicates soon. where returns a table that's a copy of the original table, but with only the rows that meet the given predicate . **Question 3.6.** Use california_farmers_markets to create a table called berkeley_markets containing farmers' markets in Berkeley, California. Out[31]: In [32]: berkeley_markets = california_farmers_markets . where ( 'city' , are . equal_to ( 'Berkeley' )) berkeley_markets

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 16/29 gives a new array of numbers that contains the result of multiplying each number in billions_of_numbers by .18. Arrays are not limited to numbers; we can also put all the words in a book into an array of strings. Concretely, an array is a collection of values of the same type . 4.1. Making arrays First, let's learn how to manually input values into an array. This typically isn't how programs work. Normally, we create arrays by loading them from an external source, like a data file. To create an array by hand, call the function make_array . Each argument you pass to make_array will be in the array it returns. Run this cell to see an example: array([ 0.125, 4.75 , -1.3 ]) Each value in an array (in the above case, the numbers 0.125, 4.75, and -1.3) is called an element of that array. Arrays themselves are also values, just like numbers and strings. That means you can assign them to names or use them as arguments to functions. For example, len(<some_array>) returns the number of elements in some_array . **Question 4.1.1.** Make an array containing the numbers 0, 1, -1, , and , in that order. Name it interesting_numbers . Hint: How did you get the values and in lab 2? You can refer to them in exactly the same way here. array([ 0. , 1. , -1. , 3.14159265, 2.71828183]) True True In [35]: make_array ( 0.125 , 4.75 , - 1.3 ) Out[35]: In [52]: interesting_numbers = make_array ( 0 , 1 , - 1 , math . pi , math . e ) interesting_numbers Out[52]: In [53]: # TEST type ( interesting_numbers ) == np . ndarray Out[53]: In [54]: # TEST len ( interesting_numbers ) == 5 Out[54]: In [55]: # TEST all ( interesting_numbers == np . array ([ 0 , 1 , - 1 , math . pi , math . e ]))

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 17/29 True **Question 4.1.2.** Make an array containing the five strings "Hello" , "," , " " , "world" , and "!" . (The third one is a single space inside quotes.) Name it hello_world_components . Note: If you evaluate hello_world_components , you'll notice some extra information in addition to its contents: dtype='<U5' . That's just NumPy's extremely cryptic way of saying that the data types in the array are strings. array(['Hello', ',', ' ', 'world', '!'], dtype='<U5') True True True np.arange Arrays are provided by a package called NumPy (pronounced "NUM-pie"). The package is called numpy , but it's standard to rename it np for brevity. You can do that with: import numpy as np Very often in data science, we want to work with many numbers that are evenly spaced within some range. NumPy provides a special function for this called arange . The line of code np.arange(start, stop, step) evaluates to an array with all the numbers starting at start and counting up by step , stopping before stop is reached. Run the following cells to see some examples! array([1, 3, 5]) Out[55]: In [56]: hello_world_components = make_array ( "Hello" , "," , " " , "world" , "!" ) hello_world_components Out[56]: In [57]: # TEST type ( hello_world_components ) == np . ndarray Out[57]: In [58]: # TEST len ( interesting_numbers ) == 5 Out[58]: In [59]: # TEST all ( hello_world_components == np . array ([ "Hello" , "," , " " , "world" , "!" ])) Out[59]: In [60]: # This array starts at 1 and counts up by 2 # and then stops before 6 np . arange ( 1 , 6 , 2 ) Out[60]:

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 19/29 array([2557628654, 2594939877, 2636772306, 2682053389, 2730228104, 2782098943, 2835299673, 2891349717, 2948137248, 3000716593, 3043001508, 3083966929, 3140093217, 3209827882, 3281201306, 3350425793, 3420677923, 3490333715, 3562313822, 3637159050, 3712697742, 3790326948, 3866568653, 3942096442, 4016608813, 4089083233, 4160185010, 4232084578, 4304105753, 4379013942, 4451362735, 4534410125, 4614566561, 4695736743, 4774569391, 4856462699, 4940571232, 5027200492, 5114557167, 5201440110, 5288955934, 5371585922, 5456136278, 5538268316, 5618682132, 5699202985, 5779440593, 5857972543, 5935213248, 6012074922, 6088571383, 6165219247, 6242016348, 6318590956, 6395699509, 6473044732, 6551263534, 6629913759, 6709049780, 6788214394, 6866332358, 6944055583, 7022349283, 7101027895, 7178722893, 7256490011], dtype=int64) Here's how we get the first element of population_amounts , which is the world population in the first year in the dataset, 1950. 2557628654 The value of that expression is the number 2557628654 (around 2.5 billion), because that's the first thing in the array population_amounts . Notice that we wrote .item(0) , not .item(1) , to get the first element. This is a weird convention in computer science. 0 is called the index of the first item. It's the number of elements that appear before that item. So 3 is the index of the 4th item. Here are some more examples. In the examples, we've given names to the things we get out of population_amounts . Read and run each cell. 3140093217 7256490011 Out[66]: In [67]: population_amounts . item ( 0 ) Out[67]: In [68]: # The 13th element in the array is the population # in 1962 (which is 1950 + 12). population_1962 = population_amounts . item ( 12 ) population_1962 Out[68]: In [69]: # The 66th element is the population in 2015. population_2015 = population_amounts . item ( 65 ) population_2015 Out[69]: In [70]: # The array has only 66 elements, so this doesn't work. # (There's no element with 66 other elements before it.) population_2016 = population_amounts . item ( 66 ) population_2016

9/5/23, 11:20 PM Lab02 localhost:8946/nbconvert/html/Downloads/Lab02.ipynb?download=false 20/29 --------------------------------------------------------------------------- IndexError Traceback (most recent call last) Cell In[70], line 3 1 # The array has only 66 elements, so this doesn't work. 2 # (There's no element with 66 other elements before it.) ----> 3 population_2016 = population_amounts . item( 66 ) 4 population_2016 IndexError : index 66 is out of bounds for axis 0 with size 66 Since make_array returns an array, we can call .item(3) on its output to get its 4th element, just like we "chained" together calls to the method replace earlier. -2 **Question 4.2.1.** Set population_1973 to the world population in 1973, by getting the appropriate element from population_amounts using item . 3942096442 True 4.3. Doing something to every element of an array Arrays are primarily useful for doing the same operation many times, so we don't often have to use .item and work with single elements. Logarithms Here is one simple question we might ask about world population: How big was the population in orders of magnitude in each year? Orders of magnitude quantify how big a number is by representing it as the power of another number (for example, representing 104 as ). One way to do this is by using the logarithm function. The logarithm (base 10) of a number increases by 1 every time we multiply the number by 10. It's like a measure of how many decimal digits the number has, or how big it is in orders of magnitude. We could try to answer our question like this, using the log10 function from the math module and the item method you just saw: In [71]: make_array ( - 1 , - 3 , 4 , - 2 ) . item ( 3 ) Out[71]: In [73]: population_1973 = population_amounts . item ( 23 ) population_1973 Out[73]: In [74]: # TEST population_1973 == 3942096442 Out[74]: In [77]: population_1950_magnitude = math . log10 ( population_amounts . item ( 0 )) population_1951_magnitude = math . log10 ( population_amounts . item ( 1 ))