hw05b_blank

Homework #5B 3 Properties of Linear Regression Residuals 1. (10 points) In the lecture, we spent a great deal of time talking about Simple Linear Regression (SLR), which you also saw in Data 8. To briefly summarize, the simple linear regression model assumes that given a single observation x , our predicted response for this observation is ˆ y = ✓ 0 + ✓ 1 x . In Lecture 10, we saw that the ✓ 0 = ˆ ✓ 0 and ✓ 1 = ˆ ✓ 1 that minimize the average L 2 loss (or Mean Squared Error - MSE) for the simple linear regression model are: ˆ ✓ 0 = ¯ y - ˆ ✓ 1 ¯ x ˆ ✓ 1 = r σ y σ x Or, rearranging terms, our predictions ˆ y are: ˆ y = ¯ y + r σ y x - ¯ x σ x (a) (3 points) As we saw in the lecture, a residual e i , for data point i 2 { 1 , . . . , n } , is defined to be the di ff erence between a true response y i and predicted response ˆ y i . Specifically, e i = y i - ˆ y i . Note that there are n data points, and each data point is denoted by ( x i , y i ) . Prove, using the equation for ˆ y above, that P n i =1 e i = 0. (b) (2 points) Prove that ¯ y = ¯ ˆ y . You may use your result from part (a). (c) (2 points) Show that (¯ x, ¯ y ) is on the simple linear regression line. e is Yi g t r y EE es si mi no Ein EI g i ng ng ng o g i s g tr EE ai z g s g tag Ii É I get EE o g d a n n s z E E of a g s g d a toss y s y

Homework #5B 6 MSE “Minimizer” 3. (10 points) Recall from calculus that given some function g ( x ) , the x you get from solving dg ( x ) dx = 0 is called a critical point of g – this means it could be a minimizer or a maximizer for g . In this question, we will explore some basic properties and build some intuition on why, for certain loss functions such as squared L 2 loss, the critical point of the empirical risk function (defined as an average loss on the observed data) will always be the minimizer. Given some linear model f ( x ) = ✓ x for some real scalar ✓ , we can write the empirical risk of the model f given the observed data { x i , y i } , for i 2 { 1 , . . . , n } as the average L 2 loss (MSE): 1 n n X i =1 ( y i - ✓ x i ) 2 = n X i =1 1 n ( y i - ✓ x i ) 2 (a) (3 points) Let’s investigate one of the n functions in the summation in the MSE. Define g i ( ✓ ) = 1 n ( y i - ✓ x i ) 2 for i 2 { 1 , . . . , n } . In this case, note that the MSE can be written as P n i =1 g i ( ✓ ) . Recall from calculus that we can use the 2nd derivative of a function to describe its curvature about a certain point (if it is facing concave up, down, or possibly a point of inflection). You can take the following as a fact: A function is convex if and only if the function’s 2nd derivative is non-negative on its domain. Based on this property, verify that g i ( ✓ ) is a convex function . (b) (2 points) Briefly explain intuitively in words why given a convex function g ( ✓ ) , the critical point we get by solving dg ( ✓ ) d ✓ = 0 minimizes g . You can assume that dg ( ✓ ) d ✓ is a function of ✓ (and not a constant). (c) (3 points) Now that we have shown that each term in the summation of the MSE is a convex function, one might wonder if the entire summation is convex, given 9 b s t C y o n t g o s th y o ni gilt 3d n n E Rt i g i is convex For a convex function g 101 its 2nd derivative is non negative so the slope is always increasing

0.0.1 Question 2b Create a plot using any seaborn and/or matplotlib.pyplot functions of your choice to visualize samples , which is the simulated distribution of Pishi votes using a sample of size 50. Include descriptive titles and labels. An example is included below. The total area under the plot must be normalized to 1. Your plot may not match exactly ours due to randomness of the data generating process in np.random.multinomial . Hint : use plt.xlim(left, right) (documentation) to specify the left and right limits of the x-axis. In [22]: sns . set_style( "whitegrid" ) plt . figure(figsize = ( 10 , 6 )) sns . histplot(samples, bins =20 , kde = True , color = 'skyblue' ) plt . title( 'Simulated Distribution of Pishi Votes (Sample Size = 50)' , fontsize =16 ) plt . xlabel( 'Proportion of Votes for Pishi' , fontsize =14 ) plt . ylabel( 'Density' , fontsize =14 ) plt . xlim( 0 , 1 ) plt . show() 1

4