hw02_revised

hw02_revised February 4, 2024 1 Homework 2: Arrays, Table Manipulation, and Visualization [1]: # Don't change this cell; just run it. # When you log-in please hit return (not shift + return) after typing in your ␣ , → email import numpy as np from datascience import * # These lines do some fancy plotting magic.\n", import matplotlib % matplotlib inline import matplotlib.pyplot as plots plots . style . use( 'fivethirtyeight' ) Recommended Reading : * Data Types * Sequences * Tables Please complete this notebook by filling in the cells provided. Throughout this homework and all future ones, please be sure to not re-assign variables throughout the notebook! For example, if you use max_temperature in your answer to one question, do not reassign it later on. Before continuing the assignment, select “Save and Checkpoint” in the File menu. 1.1 1. Creating Arrays Question 1. Make an array called weird_numbers containing the following numbers (in the given order): 1. -2 2. the sine of 1.2 3. 3 4. 5 to the power of the cosine of 1.2 Hint: sin and cos are functions in the math module. Note: Python lists are different/behave differently than numpy arrays. In Data 8, we use numpy arrays, so please make an array , not a python list. [2]: # Our solution involved one extra line of code before creating # weird_numbers. import math 1

weird_numbers = make_array( -2 ,math . sin( 1.2 ), 3 , 5** math . cos( 1.2 )) weird_numbers [2]: array([-2. , 0.93203909, 3. , 1.79174913]) Question 2. Make an array called numbers_in_order using the np.sort function. [3]: numbers_in_order = make_array(np . sort(weird_numbers)) numbers_in_order [3]: array([[-2. , 0.93203909, 1.79174913, 3. ]]) Question 3. Find the mean and median of weird_numbers using the np.mean and np.median functions. [4]: weird_mean = (np . mean(weird_numbers)) weird_median = (np . median(weird_numbers)) # These lines are provided just to print out your answers. print ( 'weird_mean:' , weird_mean) print ( 'weird_median:' , weird_median) weird_mean: 0.930947052910613 weird_median: 1.361894105821226 1.2 2. Indexing Arrays These exercises give you practice accessing individual elements of arrays. In Python (and in many programming languages), elements are accessed by index , so the first element is the element at index 0. Note: Please don’t use bracket notation when indexing (i.e. arr[0] ), as this can yield different data type outputs than what we will be expecting. Question 1. The cell below creates an array of some numbers. Set third_element to the third element of some_numbers . [5]: some_numbers = make_array( -1 , -3 , -6 , -10 , -15 ) third_element = some_numbers[ 2 ] third_element [5]: -6 Question 2. The next cell creates a table that displays some information about the elements of some_numbers and their order. Run the cell to see the partially-completed table, then fill in the missing information (the cells that say “Ellipsis”) by assigning blank_a , blank_b , blank_c , and blank_d to the correct elements in the table. 2

1.3 3. Basic Array Arithmetic Question 1. Multiply the numbers 42, 4224, 42422424, and -250 by 157. Assign each variable below such that first_product is assigned to the result of 42 ∗ 157 , second_product is assigned to the result of 4224 ∗ 157 , and so on. For this question, don’t use arrays. [10]: first_product = 42*157 second_product = 4224*157 third_product = 42422424*157 fourth_product = -250*157 print (first_product, second_product, third_product, fourth_product) 6594 663168 6660320568 -39250 Question 2. Now, do the same calculation, but using an array called numbers and only a single multiplication ( * ) operator. Store the 4 results in an array named products . [11]: numbers = make_array( 42 , 4224 , 42422424 , -250 ) products = numbers *157 products [11]: array([ 6594, 663168, 6660320568, -39250]) Question 3. Oops, we made a typo! Instead of 157, we wanted to multiply each number by 1577. Compute the correct products in the cell below using array arithmetic. Notice that your job is really easy if you previously defined an array containing the 4 numbers. [12]: correct_products = numbers *1577 correct_products [12]: array([ 66234, 6661248, 66900162648, -394250]) Question 4. We’ve loaded an array of temperatures in the next cell. Each number is the highest temperature observed on a day at a climate observation station, mostly from the US. Since they’re from the US government agency NOAA , all the temperatures are in Fahrenheit. Convert them all to Celsius by first subtracting 32 from them, then multiplying the results by 5 9 . Make sure to ROUND the final result after converting to Celsius to the nearest integer using the np.round function. [13]: max_temperatures = Table . read_table( "temperatures.csv" ) . column( "Daily Max ␣ , → Temperature" ) celsius_max_temperatures = np . round((max_temperatures) -32 ) *5/9 celsius_max_temperatures [13]: array([-3.88888889, 30.55555556, 31.66666667, …, 16.66666667, 22.77777778, 16.11111111]) 4

Question 5. The cell below loads all the lowest temperatures from each day (in Fahrenheit). Compute the size of the daily temperature range for each day. That is, compute the difference between each daily maximum temperature and the corresponding daily minimum temperature. Pay attention to the units, give your answer in Celsius! Make sure NOT to round your answer for this question! [14]: min_temperatures = Table . read_table( "temperatures.csv" ) . column( "Daily Min ␣ , → Temperature" ) celsius_temperature_ranges = ((max_temperatures) -32 ) *5/ , → 9- ((min_temperatures) -32 ) *5/9 celsius_temperature_ranges [14]: array([ 6.66666667, 10. , 12.22222222, …, 17.22222222, 11.66666667, 11.11111111]) 1.4 4. World Population The cell below loads a table of estimates of the world population for different years, starting in 1950. The estimates come from the US Census Bureau website . [15]: world = Table . read_table( "world_population.csv" ) . select( 'Year' , 'Population' ) world . show( 4 ) <IPython.core.display.HTML object> The name population is assigned to an array of population estimates. [16]: population = world . column( 1 ) population [16]: array([2557628654, 2594939877, 2636772306, 2682053389, 2730228104, 2782098943, 2835299673, 2891349717, 2948137248, 3000716593, 3043001508, 3083966929, 3140093217, 3209827882, 3281201306, 3350425793, 3420677923, 3490333715, 3562313822, 3637159050, 3712697742, 3790326948, 3866568653, 3942096442, 4016608813, 4089083233, 4160185010, 4232084578, 4304105753, 4379013942, 4451362735, 4534410125, 4614566561, 4695736743, 4774569391, 4856462699, 4940571232, 5027200492, 5114557167, 5201440110, 5288955934, 5371585922, 5456136278, 5538268316, 5618682132, 5699202985, 5779440593, 5857972543, 5935213248, 6012074922, 6088571383, 6165219247, 6242016348, 6318590956, 6395699509, 6473044732, 6551263534, 6629913759, 6709049780, 6788214394, 6866332358, 6944055583, 7022349283, 7101027895, 7178722893, 7256490011]) In this question, you will apply some built-in Numpy functions to this array. Numpy is a module that is often used in Data Science! 5

and average so that the print statement is correct. [20]: waiting_times = Table . read_table( 'old_faithful.csv' ) . column( 'waiting' ) shortest = np . min(waiting_times) longest = np . max(waiting_times) average = np . max(waiting_times) - np . min(waiting_times) print ( "Old Faithful erupts every" , shortest, "to" , longest, "minutes and ␣ , → every" , average, "minutes on average." ) Old Faithful erupts every 43 to 96 minutes and every 53 minutes on average. Question 2. Assign biggest_decrease to the biggest decrease in waiting time between two consecutive eruptions. For example, the third eruption occurred after 74 minutes and the fourth after 62 minutes, so the decrease in waiting time was 74 - 62 = 12 minutes. Hint 1 : You’ll need an array arithmetic function mentioned in the textbook . You have also seen this function earlier in the homework! Hint 2 : We want to return the absolute value of the biggest decrease. [21]: biggest_decrease = abs (np . max(waiting_times)) biggest_decrease [21]: 96 Question 3. If you expected Old Faithful to erupt every hour, you would expect to wait a total of 60 * k minutes to see k eruptions. Set difference_from_expected to an array with 272 elements, where the element at index i is the absolute difference between the expected and actual total amount of waiting time to see the first i+1 eruptions. Hint : You’ll need to compare a cumulative sum to a range. You’ll go through np.arange more thoroughly in Lab 3, but you can read about it in this textbook section . For example, since the first three waiting times are 79, 54, and 74, the total waiting time for 3 eruptions is 79 + 54 + 74 = 207. The expected waiting time for 3 eruptions is 60 * 3 = 180. Therefore, difference_from_expected.item(2) should be | 207 − 180 | = 27 . [22]: difference_from_expected = make_array(waiting_times) difference_from_expected [22]: array([[79, 54, 74, 62, 85, 55, 88, 85, 51, 85, 54, 84, 78, 47, 83, 52, 62, 84, 52, 79, 51, 47, 78, 69, 74, 83, 55, 76, 78, 79, 73, 77, 66, 80, 74, 52, 48, 80, 59, 90, 80, 58, 84, 58, 73, 83, 64, 53, 82, 59, 75, 90, 54, 80, 54, 83, 71, 64, 77, 81, 59, 84, 48, 82, 60, 92, 78, 78, 65, 73, 82, 56, 79, 71, 62, 76, 60, 78, 76, 83, 75, 82, 70, 65, 73, 88, 76, 80, 48, 86, 60, 90, 50, 78, 63, 72, 84, 75, 51, 82, 62, 88, 49, 83, 81, 47, 84, 52, 86, 81, 75, 59, 89, 79, 59, 81, 50, 85, 59, 87, 53, 69, 77, 56, 88, 81, 45, 82, 55, 90, 45, 83, 56, 89, 46, 82, 51, 86, 53, 79, 81, 60, 82, 77, 7

76, 59, 80, 49, 96, 53, 77, 77, 65, 81, 71, 70, 81, 93, 53, 89, 45, 86, 58, 78, 66, 76, 63, 88, 52, 93, 49, 57, 77, 68, 81, 81, 73, 50, 85, 74, 55, 77, 83, 83, 51, 78, 84, 46, 83, 55, 81, 57, 76, 84, 77, 81, 87, 77, 51, 78, 60, 82, 91, 53, 78, 46, 77, 84, 49, 83, 71, 80, 49, 75, 64, 76, 53, 94, 55, 76, 50, 82, 54, 75, 78, 79, 78, 78, 70, 79, 70, 54, 86, 50, 90, 54, 54, 77, 79, 64, 75, 47, 86, 63, 85, 82, 57, 82, 67, 74, 54, 83, 73, 73, 88, 80, 71, 83, 56, 79, 78, 84, 58, 83, 43, 60, 75, 81, 46, 90, 46, 74]]) Question 4. Let’s imagine your guess for the next wait time was always just the length of the previous waiting time. If you always guessed the previous waiting time, how big would your error in guessing the waiting times be, on average? For example, since the first three waiting times are 79, 54, and 74, the average difference between your guess and the actual time for just the second and third eruption would be | 79 − 54 | + | 54 − 74 | 2 = 22 . 5 . [23]: average_error = ... # average_error 1.6 6. Tables Question 1. Suppose you have 4 apples, 3 oranges, and 3 pineapples. (Perhaps you’re using Python to solve a high school Algebra problem.) Create a table that contains this information. It should have two columns: fruit name and count . Assign the new table to the variable fruits . Note: Use lower-case and singular words for the name of each fruit, like "apple" . [24]: # Our solution uses 1 statement split over 3 lines. fruits = make_array( 'apple' , 'orange' , 'pineapple' ) fruit_count = make_array( 4 , 3 , 3 ) fruits = Table() . with_columns( 'fruits' ,fruits, 'count' ,fruit_count) fruits # fruits [24]: fruits | count apple | 4 orange | 3 pineapple | 3 Question 2. The file inventory.csv contains information about the inventory at a fruit stand. Each row represents the contents of one box of fruit. Load it as a table named inventory using the Table.read_table() function. Table.read_table(...) takes one argument (data file name in string format) and returns a table. [25]: inventory = Table . read_table( 'inventory.csv' ) inventory # inventory 8

Hint: If you’re stuck, think first about how you would compute the total revenue from just the grape sales. [29]: count = sales . column( 2 ) price = sales . column( 3 ) total_revenue = sum (count * price) total_revenue [29]: 106.85 Question 7. Make a new table called remaining_inventory . It should have the same rows and columns as inventory , except that the amount of fruit sold from each box should be subtracted from that box’s count, so that the “count” is the amount of fruit remaining after Saturday. [30]: remaining_inventory = Table() . with_columns( 'box ID' , inventory . column( 'box ␣ , → ID' ), 'fruit name' , inventory . column( 'fruit name' ), 'count' , inventory . , → column( 'count' ) - sales . column( 'count sold' )) remaining_inventory [30]: box ID | fruit name | count 53686 | kiwi | 42 57181 | strawberry | 22 25274 | apple | 20 48800 | orange | 0 26187 | strawberry | 230 57930 | grape | 162 52357 | strawberry | 0 43566 | peach | 23 1.7 7. Unemployment The Federal Reserve Bank of St. Louis publishes data about jobs in the US. Below, we’ve loaded data on unemployment in the United States. There are many ways of defining unemployment, and our dataset includes two notions of the unemployment rate: 1. Among people who are able to work and are looking for a full-time job, the percentage who can’t find a job. This is called the Non-Employment Index, or NEI. 2. Among people who are able to work and are looking for a full-time job, the percentage who can’t find any job or are only working at a part-time job. The latter group is called “Part- Time for Economic Reasons”, so the acronym for this index is NEI-PTER. (Economists are great at marketing.) The source of the data is here . Question 1. The data are in a CSV file called unemployment.csv . Load that file into a table called unemployment . 10

[31]: unemployment = farmers_markets = Table . read_table( 'unemployment.csv' ) unemployment [31]: Date | NEI | NEI-PTER 1994-01-01 | 10.0974 | 11.172 1994-04-01 | 9.6239 | 10.7883 1994-07-01 | 9.3276 | 10.4831 1994-10-01 | 9.1071 | 10.2361 1995-01-01 | 8.9693 | 10.1832 1995-04-01 | 9.0314 | 10.1071 1995-07-01 | 8.9802 | 10.1084 1995-10-01 | 8.9932 | 10.1046 1996-01-01 | 9.0002 | 10.0531 1996-04-01 | 8.9038 | 9.9782 … (80 rows omitted) Question 2. Sort the data in descending order by NEI, naming the sorted table by_nei . Create another table called by_nei_pter that’s sorted in descending order by NEI-PTER instead. [32]: by_nei = unemployment . sort( 'NEI' ,descending = True ) by_nei_pter = unemployment . sort( 'NEI' ,descending = True ) by_nei [32]: Date | NEI | NEI-PTER 2009-10-01 | 10.9698 | 12.8557 2010-01-01 | 10.9054 | 12.7311 2009-07-01 | 10.8089 | 12.7404 2009-04-01 | 10.7082 | 12.5497 2010-04-01 | 10.6597 | 12.5664 2010-10-01 | 10.5856 | 12.4329 2010-07-01 | 10.5521 | 12.3897 2011-01-01 | 10.5024 | 12.3017 2011-07-01 | 10.4856 | 12.2507 2011-04-01 | 10.4409 | 12.247 … (80 rows omitted) Question 3. Use take to make a table containing the data for the 10 quarters when NEI was greatest. Call that table greatest_nei . greatest_nei should be sorted in descending order of NEI . Note that each row of unemployment represents a quarter. [33]: greatest_nei = by_nei . take(np . arange( 10 )) greatest_nei [33]: Date | NEI | NEI-PTER 2009-10-01 | 10.9698 | 12.8557 2010-01-01 | 10.9054 | 12.7311 11

2011-04-01 | 10.4409 | 12.247 | 1.8061 2011-01-01 | 10.5024 | 12.3017 | 1.7993 2011-10-01 | 10.3287 | 12.1214 | 1.7927 … (80 rows omitted) Question 6. Create a line plot of the PTER over time. To do this, create a new table called pter_over_time that adds the year array and the pter array to the unemployment table. Label these columns Year and PTER . Then, generate a line plot using one of the table methods you’ve learned in class. [36]: year = 1994 + np . arange(by_pter . num_rows) /4 pter_over_time = unemployment . with_columns( 'Year' ,year, 'PTER' ,pter) pter_over_time . plot( 'Year' , 'PTER' ) 13

Question 7. Were PTER rates high during the Great Recession (that is to say, were PTER rates particularly high in the years 2008 through 2011)? Assign highPTER to True if you think PTER rates were high in this period, and False if you think they weren’t. [37]: highPTER = True 1.8 8. Birth Rates The following table gives census-based population estimates for each state on both July 1, 2015 and July 1, 2016. The last four columns describe the components of the estimated change in population during this time interval. For all questions below, assume that the word “states” refers to all 52 rows including Puerto Rico & the District of Columbia. The data was taken from here . If you want to read more about the different column descriptions, click here ! The raw data is a bit messy - run the cell below to clean the table and make it easier to work with. [38]: # Don't change this cell; just run it. pop = Table . read_table( 'nst-est2016-alldata.csv' ) . where( 'SUMLEV' , 40 ) . , → select([ 1 , 4 , 12 , 13 , 27 , 34 , 62 , 69 ]) pop = pop . relabeled( 'POPESTIMATE2015' , '2015' ) . relabeled( 'POPESTIMATE2016' , ␣ , → '2016' ) pop = pop . relabeled( 'BIRTHS2016' , 'BIRTHS' ) . relabeled( 'DEATHS2016' , 'DEATHS' ) pop = pop . relabeled( 'NETMIG2016' , 'MIGRATION' ) . relabeled( 'RESIDUAL2016' , ␣ , → 'OTHER' ) pop = pop . with_columns( "REGION" , np . array([ int (region) if region != "X" else 0 ␣ , → for region in pop . column( "REGION" )])) pop . set_format([ 2 , 3 , 4 , 5 , 6 , 7 ], NumberFormatter(decimals =0 )) . show( 5 ) <IPython.core.display.HTML object> Question 1. Assign us_birth_rate to the total US annual birth rate during this time interval. The annual birth rate for a year-long period is the total number of births in that period as a proportion of the population size at the start of the time period. Hint: Which year corresponds to the start of the time period? [39]: us_birth_rate = sum (pop . column( 'BIRTHS' )) / sum (pop . column( '2015' )) us_birth_rate [39]: 0.012358536498646102 Question 2. In the next question, you will be creating a visualization to understand the relationship between birth and death rates. The annual death rate for a year-long period is the total number of deaths in that period as a proportion of the population size at the start of the time period. 14

Boston Table <IPython.core.display.HTML object> Manila Table <IPython.core.display.HTML object> Question 1. Produce histograms of all ride times in Boston using the given bins. [44]: equal_bins = np . arange( 0 , 120 , 5 ) boston . hist( 'ride time' , bins = equal_bins) Question 2. Now, produce histograms of all ride times in Manila using the given bins. [45]: equal_bins = np . arange( 0 , 120 , 5 ) manila . hist( 'ride time' , bins = equal_bins) # Don't delete the following line! plots . ylim( 0 , 0.05 ) [45]: (0.0, 0.05) 16

Question 3. Assign boston_under_10 and manila_under_10 to the percentage of rides that are less than 10 minutes in their respective metropolitan areas. Use the height variables provided below in order to compute the percentages. Your solution should only use height variables, numbers, and mathematical operations. You should not access the tables boston and manila in any way. [46]: boston_under_5_height = 1.2 manila_under_5_height = 0.6 boston_5_to_under_10_height = 3.2 manila_5_to_under_10_height = 1.4 boston_under_10 = ( 1.2+3.2 ) /4.4 manila_under_10 = ( 0.6+1.4 ) /2 boston_under_10,manila_under_10 [46]: (1.0, 1.0) Question 4. Let’s take a closer look at the distribution of ride times in Manila. Assign manila_median_bin to an integer (1, 2, 3, or 4) that corresponds to the bin that contains the median time 1: 0-15 minutes 2: 15-40 minutes 3: 40-60 minutes 4: 60-80 minutes Hint: The median of a sorted list has half of the list elements to its left, and half to its right 17