project1_1

project1 October 17, 2023 [1]: # Initialize Otter import otter grader = otter . Notebook( "project1.ipynb" ) 1 Project 1: World Progress In this project, you’ll explore data from Gapminder.org , a website dedicated to providing a fact- based view of the world and how it has changed. That site includes several data visualizations and presentations, but also publishes the raw data that we will use in this project to recreate and extend some of their most famous visualizations. The Gapminder website collects data from many sources and compiles them into tables that describe many countries around the world. All of the data they aggregate are published in the Systema Globalis . Their goal is “to compile all public statistics; Social, Economic and Environmental; into a comparable total dataset.” All data sets in this project are copied directly from the Systema Globalis without any changes. This project is dedicated to Hans Rosling (1948-2017), who championed the use of data to under- stand and prioritize global development challenges. 1.0.1 Logistics Rules. Don’t share your code with anybody but your partner. You are welcome to discuss questions with other students, but don’t share the answers. The experience of solving the problems in this project will prepare you for exams (and life). If someone asks you for the answer, resist! Instead, you can demonstrate how you would solve a similar problem. Support. You are not alone! Come to offce hours and talk to your classmates. If you want to ask about the details of your solution to a problem, come see me. If you’re ever feeling overwhelmed or don’t know how to make progress, email for help. Tests. The tests that are given are not comprehensive and passing the tests for a question does not mean that you answered the question correctly. Tests usually only check that your table has the correct column labels. However, more tests will be applied to verify the correctness of your submission in order to assign your final score, so be careful and check your work! You might want to create your own checks along the way to see if your answers make sense. Additionally, before you submit, make sure that none of your cells take a very long time to run (several minutes). Free Response Questions: Make sure that you put the answers to the written questions in the indicated cell we provide. Every free response question should include an explanation that 1

adequately answers the question. Tabular Thinking Guide: Feel free to reference Tabular Thinking Guide for extra guidance. Advice. Develop your answers incrementally. To perform a complicated table manipulation, break it up into steps, perform each step on a different line, give a new name to each result, and check that each intermediate result is what you expect. You can add any additional names or functions you want to the provided cells. Make sure that you are using distinct and meaningful variable names throughout the notebook. Along that line, DO NOT reuse the variable names that we use when we grade your answers. For example, in Question 1 of the Global Poverty section we ask you to assign an answer to latest . Do not reassign the variable name latest to anything else in your notebook, otherwise there is the chance that our tests grade against what latest was reassigned to. You never have to use just one line in this project or any others. Use intermediate variables and multiple lines as much as you would like! To get started, load datascience , numpy , plots , and otter . [2]: # Run this cell to set up the notebook, but please don't change it. # These lines import the NumPy and Datascience modules. from datascience import * import numpy as np # These lines do some fancy plotting magic. % matplotlib inline import matplotlib.pyplot as plots plots . style . use( 'fivethirtyeight' ) from ipywidgets import interact, interactive, fixed, interact_manual import ipywidgets as widgets import d8error 1.1 1. Global Population Growth The global population of humans reached 1 billion around 1800, 3 billion around 1960, and 7 billion around 2011. The potential impact of exponential population growth has concerned scientists, economists, and politicians alike. The UN Population Division estimates that the world population will likely continue to grow throughout the 21st century, but at a slower rate, perhaps reaching 11 billion by 2100. However, the UN does not rule out scenarios of more extreme growth. In this part of the project, we will examine some of the factors that influence population growth and how they have been changing over the years and around the world. There are two main sub-parts of this analysis. 2

[5]: grader . check( "q1_1" ) [5]: q1_1 results: All test cases passed! Run the following cell to create a table called b_five that has the population of Bangladesh every five years. At a glance, it appears that the population of Bangladesh has been growing quickly indeed! [6]: b_pop . set_format( 'population_total' , NumberFormatter) fives = np . arange( 1970 , 2021 , 5 ) # 1970, 1975, 1980, ... b_five = b_pop . sort( 'time' ) . where( 'time' , are . contained_in(fives)) b_five . show() <IPython.core.display.HTML object> Question 2. Assign initial to an array that contains the population for every five year interval from 1970 to 2015 (inclusive). Then, assign changed to an array that contains the population for every five year interval from 1975 to 2020 (inclusive). The first array should include both 1970 and 2015, and the second array should include both 1975 and 2020. You should use the b_five table to create both arrays, by first filtering the table to only contain the relevant years. The annual growth rate for a time period is equal to: ⎛ ⎜ ⎜ ⎜ ⎝ ( Population at end of period Population at start of period ) 1 number of years ⎞ ⎟ ⎟ ⎟ ⎠ − 1 We have provided the code below that uses initial and changed in order to add a column to b_five called annual_growth . Don’t worry about the calculation of the growth rates; run the test below to test your solution. If you are interested in how we came up with the formula for growth rates, consult the growth rates section of the textbook. [7]: initial = b_five . where( 'time' , are . between_or_equal_to( 1970 , 2015 )) . ↪ column( 'population_total' ) changed = b_five . where( 'time' , are . between_or_equal_to( 1975 , 2020 )) . ↪ column( 'population_total' ) b_1970_through_2015 = b_five . where( 'time' , are . below_or_equal_to( 2015 )) b_five_growth = b_1970_through_2015 . with_column( 'annual_growth' , (changed / ↪ initial) **0.2-1 ) b_five_growth . set_format( 'annual_growth' , PercentFormatter) [7]: time | population_total | annual_growth 1970 | 64,232,486 | 1.75% 1975 | 70,066,310 | 2.59% 1980 | 79,639,498 | 2.65% 4

1985 | 90,764,180 | 2.60% 1990 | 103,171,957 | 2.22% 1995 | 115,169,933 | 2.08% 2000 | 127,657,862 | 1.72% 2005 | 139,035,505 | 1.20% 2010 | 147,575,433 | 1.15% 2015 | 156,256,287 | 1.06% [8]: grader . check( "q1_2" ) [8]: q1_2 results: All test cases passed! While the population has grown every five years since 1970, the annual growth rate decreased dramatically from 1985 to 2015. Let’s look at some other information in order to develop a possible explanation. Run the next cell to load three additional tables of measurements about countries over time. [9]: life_expectancy = Table . read_table( 'life_expectancy.csv' ) . where( 'time' , are . ↪ below( 2021 )) child_mortality = Table . read_table( 'child_mortality.csv' ) . relabel( 2 , ␣ ↪ 'child_mortality_under_5_per_1000_born' ) . where( 'time' , are . below( 2021 )) fertility = Table . read_table( 'fertility.csv' ) . where( 'time' , are . below( 2021 )) The life_expectancy table contains a statistic that is often used to measure how long people live, called life expectancy at birth . This number, for a country in a given year, does not measure how long babies born in that year are expected to live . Instead, it measures how long someone would live, on average, if the mortality conditions in that year persisted throughout their lifetime. These “mortality conditions” describe what fraction of people at each age survived the year. So, it is a way of measuring the proportion of people that are staying alive, aggregated over different age groups in the population. Run the following cells below to see life_expectancy , child_mortality , and fertility . Refer back to these tables as they will be helpful for answering further questions! [10]: life_expectancy . show( 3 ) <IPython.core.display.HTML object> [11]: child_mortality . show( 3 ) <IPython.core.display.HTML object> [12]: fertility . show( 3 ) <IPython.core.display.HTML object> Question 3. Perhaps population is growing more slowly because people aren’t living as long. Use the life_expectancy table to draw a line graph with the years 1970 and later on the horizontal axis that shows how the life expectancy at birth has changed in Bangladesh. 5

The fertility table contains a statistic that is often used to measure how many babies are being born, the total fertility rate . This number describes the number of children a woman would have in her lifetime , on average, if the current rates of birth by age of the mother persisted throughout her child bearing years, assuming she survived through age 49. Question 5. Complete the function fertility_over_time . It takes the Alpha-3 code of a country as country_code and a start year. It returns a two-column table with labels Year and Children per woman that can be used to generate a line chart of the country’s fertility rate each year, starting at the start year. The plot should include the start year and all later years that appear in the fertility table. Then, determine the Alpha-3 code for Bangladesh. The code for Bangladesh and the year 1970 are used in the call to your fertility_over_time function in order to plot how Bangladesh’s fertility rate has changed since 1970. Note that the function fertility_over_time should not return the plot itself. The expression that draws the line plot is provided for you; please don’t change it. [14]: def fertility_over_time (country_code, start): """Create a two-column table that describes a country's total fertility ␣ ↪ rate each year.""" country_fertility = fertility . where( 'geo' , are . equal_to(country_code)) . ↪ sort( 'time' ) country_fertility_after_start = country_fertility . where( 'time' , are . ↪ above_or_equal_to(start)) cleaned_table = Table() . with_columns( 'Year' , country_fertility_after_start . ↪ column( 1 ), 'Children per woman' , country_fertility_after_start . column( 2 )) return cleaned_table bangladesh_code = 'bgd' fertility_over_time(bangladesh_code, 1970 ) . plot( 0 , 1 ) # You should *not* change ␣ ↪ this line. 7

[15]: grader . check( "q1_5" ) [15]: q1_5 results: All test cases passed! Question 6. Assuming everything else is constant, do the trends in fertility in the graph above help directly explain why the population growth rate decreased from 1980 to 2020 in Bangladesh? Why or why not? This graph shows the declaine in fertility in women as the years go on, this can help explain why Bangladesh’s population started to decrease. It has been observed that lower fertility rates are often associated with lower child mortality rates. The link has been attributed to family planning: if parents can expect that their children will all survive into adulthood, then they will choose to have fewer children. In the reverse direction, having fewer children may allow families to devote more resources to each child, reducing child mortality. We can see if this association is evident in Bangladesh by plotting the relationship between total 8

[17]: grader . check( "q1_7" ) [17]: q1_7 results: All test cases passed! The plot above uses color to encode data about the time column from the table post_1969_fertility_and_child_mortality . The colors, ranging from dark blue to white, rep- resent the passing of time between the 1970s to the 2020s. For example, a point on the scatter plot representing data from the 1970s would appear as dark blue and a point from the 2010s would appear as light blue . Question 8. In one or two sentences, describe the association (if any) that is illustrated by this scatter diagram. Does the diagram show that reduced child mortality causes parents to choose to have fewer children? There seems to be an association with the year and child mortality rate, that as time has gone on since 1970, the amount of child deaths have decreased. The diagram does not show that fewer child deaths necessarily cause women to have fewer children. 10

WOOOHOO Yoshi and friends want to congratulate you on getting this far! To double check your work, the cell below will rerun all of the autograder tests for Section 1. [18]: checkpoint_tests = [ "q1_1" , "q1_2" , "q1_5" , "q1_7" ] for test in checkpoint_tests: display(grader . check(test)) q1_1 results: All test cases passed! q1_2 results: All test cases passed! q1_5 results: All test cases passed! q1_7 results: All test cases passed! 1.2 Submission If your instructor would like you to submit the work in part one as a checkpoint to the project, follow the instructions below. Make sure you have run all cells in your notebook in order before running the cell below, so that all images/graphs appear in the output. The cell below will generate a zip file for you to submit. Please save before exporting! [19]: # Save your notebook first, then run this cell to export your submission. grader . export(pdf = False ) <IPython.core.display.HTML object> 1.2.1 The World The change observed in Bangladesh since 1970 can also be observed in many other developing countries: health services improve, life expectancy increases, and child mortality decreases. At the same time, the fertility rate often plummets, and so the population growth rate decreases despite increasing longevity. Run the cell below to generate two overlaid histograms, one for 1962 and one for 2010, that show the distributions of total fertility rates for these two years among all 201 countries in the fertility table. [20]: Table() . with_columns( '1962' , fertility . where( 'time' , 1962 ) . column( 2 ), '2010' , fertility . where( 'time' , 2010 ) . column( 2 ) ) . hist(bins = np . arange( 0 , 10 , 0.5 ), unit = 'child per woman' ) _ = plots . xlabel( 'Children per woman' ) _ = plots . ylabel( 'Percent per children per woman' ) _ = plots . xticks(np . arange( 10 )) 11

Question 11. Create a function stats_for_year that takes a year and returns a ta- ble of statistics. The table it returns should have four columns: geo , population_total , children_per_woman_total_fertility , and child_mortality_under_5_per_1000_born . Each row should contain one unique Alpha-3 country code and three statistics: population, fertility rate, and child mortality for that year from the population , fertility and child_mortality tables. Only include rows for which all three statistics are available for the country and year. In addition, restrict the result to country codes that appears in big_50 , an array of the 50 most populous countries in 2020. This restriction will speed up computations later in the project. After you write stats_for_year , try calling stats_for_year on any year between 1960 and 2020. Try to understand the output of stats_for_year. Hint : The tests for this question are quite comprehensive, so if you pass the tests, your function is probably correct. However, without calling your function yourself and looking at the output, it 13

will be very diffcult to understand any problems you have, so try your best to write the function correctly and check that it works before you rely on the grader tests to confirm your work. Hint : What do all three tables have in common (pay attention to column names)? [24]: # We first create a population table that only includes the # 50 countries with the largest 2020 populations. We focus on # these 50 countries only so that plotting later will run faster. big_50 = population . where( 'time' , are . equal_to( 2020 )) . sort( "population_total" , ␣ ↪ descending = True ) . take(np . arange( 50 )) . column( 'geo' ) population_of_big_50 = population . where( 'time' , are . above( 1959 )) . where( 'geo' , ␣ ↪ are . contained_in(big_50)) def stats_for_year (year): """Return a table of the stats for each country that year.""" p = population_of_big_50 . where( 'time' , are . equal_to(year)) . drop( 'time' ) f = fertility . where( 'time' , are . equal_to(year)) . drop( 'time' ) c = child_mortality . where( 'time' , are . equal_to(year)) . drop( 'time' ) return p . join( 'geo' , f, 'geo' ) . join( 'geo' , c, 'geo' ) stats_for_year( 1978 ) [24]: geo | population_total | children_per_woman_total_fertility | child_mortality_under_5_per_1000_born afg | 13341199 | 7.45 | 256.75 ago | 7790774 | 7.57 | 243.24 arg | 27061041 | 3.39 | 51.17 bgd | 75450033 | 6.59 | 207.32 bra | 115121158 | 4.26 | 104.82 can | 23901716 | 1.71 | 14.17 chn | 972205441 | 2.74 | 69.93 cod | 24956387 | 6.49 | 217.82 col | 25733669 | 4.18 | 65.65 deu | 78573588 | 1.49 | 17.12 … (40 rows omitted) [25]: grader . check( "q1_11" ) [25]: q1_11 results: All test cases passed! Question 12. Create a table called pop_by_decade with two columns called decade and population , in this order. It has a row for each year that starts a decade, in increasing order starting with 1960 and ending with 2020. For example, 1960 is the start of the 1960’s decade. The population column contains the total population of all countries included in the result of stats_for_year(year) for the first year of the decade. You should see that these countries contain most of the world’s population. Hint: One approach is to define a function pop_for_year that computes this total population, then apply it to the decade column. The stats_for_year function from the previous question 14

[30]: country | name | world_6region afg | Afghanistan | south_asia ago | Angola | sub_saharan_africa alb | Albania | europe_central_asia and | Andorra | europe_central_asia are | United Arab Emirates | middle_east_north_africa arg | Argentina | america arm | Armenia | europe_central_asia atg | Antigua and Barbuda | america aus | Australia | east_asia_pacific aut | Austria | europe_central_asia … (187 rows omitted) Question 13. Create a table called region_counts . It should contain two columns called region and count . The region column should contain regions of the world, and the count column should contain the number of countries in each region that appears in the result of stats_for_year(2020) . For example, one row would have south_asia as its region value and an integer as its count value: the number of large South Asian countries for which we have population, fertility, and child mortality numbers from 2020. Hint : You may have to relabel a column to name it region . [31]: stats_for_2020 = stats_for_year( 2020 ) region_counts = stats_for_2020 . join( 'geo' , countries, 'country' ) . ↪ group( 'world_6region' ) . relabel( 'world_6region' , 'region' ) region_counts [31]: region | count america | 8 east_asia_pacific | 9 europe_central_asia | 10 middle_east_north_africa | 7 south_asia | 5 sub_saharan_africa | 11 [32]: grader . check( "q1_13" ) [32]: q1_13 results: All test cases passed! The following scatter diagram compares total fertility rate and child mortality rate for each country in 1960. The area of each dot represents the population of the country, and the color represents its region of the world. Run the cell. Do you think you can identify any of the dots? [33]: from functools import lru_cache as cache # This cache annotation makes sure that if the same year # is passed as an argument twice, the work of computing # the result is only carried out once. 16

@cache ( None ) def stats_relabeled (year): """Relabeled and cached version of stats_for_year.""" return stats_for_year(year) . relabel( 2 , 'Children per woman' ) . relabel( 3 , ␣ ↪ 'Child deaths per 1000 born' ) def fertility_vs_child_mortality (year): """Draw a color scatter diagram comparing child mortality and fertility.""" with_region = stats_relabeled(year) . join( 'geo' , countries . select( 'country' , ␣ ↪ 'world_6region' ), 'country' ) with_region . scatter( 2 , 3 , sizes =1 , group =4 , s =500 ) plots . xlim( 0 , 10 ) plots . ylim( -50 , 500 ) plots . title(year) plots . show() fertility_vs_child_mortality( 1960 ) Question 14. Assign scatter_statements to an array of the numbers of each statement below that can be inferred from this scatter diagram for 1960. 1. As a whole, the europe_central_asia region had the lowest child mortality rate. 1. The lowest child mortality rate of any country was from an east_asia_pacific country. 1. Most countries had a fertility rate above 5. 1. There was an association between child mortality and fertility. 1. The two largest countries by population also had the two highest child mortality rates. [34]: scatter_statements = make_array( 1 , 3 , 4 ) [35]: grader . check( "q1_14" ) 17

goal : “By 2030, eradicate extreme poverty for all people everywhere.” In this part of the project we will examine aspects of global poverty that might affect whether the goal is achievable. First, load the population and poverty rate by country and year and the country descriptions. While the population table has values for every recent year for many countries, the poverty table only includes certain years for each country in which a measurement of the rate of extreme poverty was available. [39]: population = Table . read_table( 'population.csv' ) countries = Table . read_table( 'countries.csv' ) . where( 'country' , are . ↪ contained_in(population . group( 'geo' ) . column( 'geo' ))) poverty = Table . read_table( 'poverty.csv' ) poverty . show( 25 ) <IPython.core.display.HTML object> Question 1. Assign latest_poverty to a three-column table with one row for each country that appears in the poverty table. The first column should contain the 3-letter code for the country. The second column should contain the most recent year for which an extreme poverty rate is available for the country. The third column should contain the poverty rate in that year. Do not change the last line, so that the labels of your table are set correctly. Hint : think about how group works: it does a sequential search of the table (from top to bottom) and collects values in the array in the order in which they appear, and then applies a function to that array. The first function may be helpful, but you are not required to use it. [40]: def first (values): return values . item( 0 ) latest_poverty = poverty . sort( 'time' , True ) . group( 'geo' , first) latest_poverty = latest_poverty . relabeled( 0 , 'geo' ) . relabeled( 1 , 'time' ) . ↪ relabeled( 2 , 'poverty_percent' ) # You should *not* change this line. latest_poverty [40]: geo | time | poverty_percent ago | 2009 | 43.37 alb | 2012 | 0.46 arg | 2011 | 1.41 arm | 2012 | 1.75 aus | 2003 | 1.36 aut | 2004 | 0.34 aze | 2008 | 0.31 bdi | 2006 | 81.32 bel | 2000 | 0.5 ben | 2012 | 51.61 … (135 rows omitted) [41]: grader . check( "q2_1" ) 19

[41]: q2_1 results: All test cases passed! Question 2. Using both latest_poverty and population , create a four-column table called recent_poverty_total with one row for each country in latest_poverty . The four columns should have the following labels and contents: 1. geo contains the 3-letter country code, 1. poverty_percent contains the most recent poverty percent, 1. population_total contains the population of the country in 2010, 1. poverty_total contains the number of people in poverty rounded to the nearest integer , based on the 2010 population and most recent poverty rate. Hint : You are not required to use poverty_and_pop , and you are always welcome to add any additional names. [42]: poverty_and_pop = latest_poverty . join( 'geo' , population) . where( 'time_2' , are . ↪ equal_to( 2010 )) #poverty_and_pop recent_poverty_total = poverty_and_pop . with_columns( 'poverty_total' , np . ↪ round(poverty_and_pop . column( 'poverty_percent' ) / 100 * poverty_and_pop . column( 'population_total' ))) . ↪ drop( 'time' , 'time_2' ) recent_poverty_total [42]: geo | poverty_percent | population_total | poverty_total ago | 43.37 | 23356247 | 1.01296e+07 alb | 0.46 | 2948029 | 13561 arg | 1.41 | 40895751 | 576630 arm | 1.75 | 2877314 | 50353 aus | 1.36 | 22154687 | 301304 aut | 0.34 | 8409945 | 28594 aze | 0.31 | 9032465 | 28001 bdi | 81.32 | 8675606 | 7.055e+06 bel | 0.5 | 10938735 | 54694 ben | 51.61 | 9199254 | 4.74774e+06 … (135 rows omitted) [43]: grader . check( "q2_2" ) [43]: q2_2 results: All test cases passed! Question 3. Assign the name poverty_percent to the known percentage of the world’s 2010 population that were living in extreme poverty. Assume that the poverty_total numbers in the recent_poverty_total table describe all people in 2010 living in extreme poverty. You should get a number that is above the 2018 global estimate of 9%, since many country-specific poverty rates are older than 2018. Hint : The sum of the population_total column in the recent_poverty_total table is not the world population, because only a subset of the world’s countries are included in the recent_poverty_total table (only some countries have known poverty rates). Use the population table to compute the world’s 2010 total population. 20