5_Ch11_ISE135

ISE 135 ‐ Fall 2016 1 SIMPLE LINEAR REGRESSION Chapter 11 Dr. Supreeta Amin ISE 135 – Fall 2016 San Jose State University Department of Industrial and Systems Engineering Introduction Is there a relationship between the number of hours and your exam score?

ISE 135 ‐ Fall 2016 2 What is Regression? Regression ¾ Is the attempt to explain the variation in a dependent variable using the variation in independent variables ¾ Is thus an explanation of causation ¾ Is a technique concerned with predicting some variables by knowing others Regression Models ¾ Relationship between one dependent variable and explanatory variable(s) ¾ Use equation to set up relationship Numerical dependent (response) variable 1 or more numerical or categorical independent (explanatory) variables ¾ Used mainly for prediction & estimation

ISE 135 ‐ Fall 2016 4 Assumptions (or the fine print) ¾ Linear regression assumes that… – The relationship between X and Y is linear – Y is distributed normally at each value of X – The variance of Y at every value of X is the same (homogeneity of variances) – The observations are independent Regression ¾ Calculates the “best‐fit” line for a certain set of data ¾ The regression line makes the sum of the squares of the residuals smaller than for any other line ¾ Regression minimizes residuals 80 100 120 140 160 180 200 220 60 70 80 90 100 110 120 Wt (kg)

ISE 135 ‐ Fall 2016 5 Example A family doctor wishes to examine the variables that relationship between a patients’ age and cholesterol. He randomly selects 14 of his female patients and obtains the data presented in the table below. Find the least square regression equation and the coefficient of determination. Age Total Cholesterol Age Total Cholesterol 25 180 42 185 25 195 48 204 28 186 51 221 32 180 51 243 32 210 58 208 32 197 62 228 38 239 65 269 Regression Analysis ANOVA df SS MS F Significance F Regression 1 4840.062 4840.062 13.05434 0.00355946 Residual 12 4449.152 370.7627 Total 13 9289.214 SUMMARY OUTPUT Regression Statistics Multiple R 0.721832 R Square 0.521041 Adjusted R Square 0.481128 Standard Error 19.2552 Observations 14 Coefficients Standard Error t Stat P‐value Lower 95% Upper 95% Intercept 151.4989457 17.08383409 8.867971 1.29E‐06 114.2764688 188.7214226 X Variable 1 1.399006383 0.387206113 3.6130793 0.003559 0.555356737 2.24265603 P‐value < 0.003 Æ model is reasonable

ISE 135 ‐ Fall 2016 7 Linear Equations Linear Regression Model ¾ Each pair of observation satisfies the relationship ࢟ ෝ ൌ ࢼ ૙ ෢ + ࢼ ૚ ෢ ࢞ ࢏ ൅ ࢋ ࢏ Dependent (Response) Variable Independent (Explanatory) Variable Population Slope Population Y‐Intercept Random Error

ISE 135 ‐ Fall 2016 8 Estimates of Slope and Intercept ¾ Intercept ߚ ଴ ෢ ߚ ଴ ෢ ൌ ݕ ത െ ߚ ଵ ෢ ݔ̅ ¾ Slope ߚ ଵ ෢ ߚ ଵ ෢ ൌ ∑ ݕ ௜ ݔ ௜ െ ∑ ݕ ௜ ∗ ௡ ௜ୀଵ ∑ ݔ ௜ ௡ ௜ୀଵ ݊ ௡ ௜ୀଵ ∑ ݔ ௜ ଶ െ ∑ ݔ ௜ ௡ ௜ୀଵ ଶ ݊ ௡ ௜ୀଵ ൌ ܵ ௫௬ ܵ ௫௫ ݕ ത ൌ ଵ ௡ ∑ ݕ ௜ ௡ ௜ୀଵ and ݔ̅ ൌ ଵ ௡ ∑ ݔ ௜ ௡ ௜ୀଵ Example Given: ∑ ݔ ௜ ൌ 589 ∑ ݕ ௜ ൌ 2945 ݔ̅ ൌ 42.07 ; ݕ ത ൌ 210.35 ∑ ݔ ௜ ଶ ൌ 27253 ∑ ݕ ௜ ଶ ൌ 628791 ∑ ݔ ௜ ݕ ௜ ൌ 127360 n = 14 Find the equation of the line Age (x) Total Cholesterol (y) Age (x) Total Cholesterol (y) 25 180 42 185 25 195 48 204 28 186 51 221 32 180 51 243 32 210 58 208 32 197 62 228 38 239 65 269 ߚ ଵ ෢ ൌ ∑ ݕ ௜ ݔ ௜ െ ∑ ݕ ௜ ∗ ௡ ௜ୀଵ ∑ ݔ ௜ ௡ ௜ୀଵ ݊ ௡ ௜ୀଵ ∑ ݔ ௜ ଶ െ ∑ ݔ ௜ ௡ ௜ୀଵ ଶ ݊ ௡ ௜ୀଵ ߚ ଴ ෢ ൌ ݕ ത െ ߚ ଵ ෢ ݔ̅

ISE 135 ‐ Fall 2016 10 Simple Linear Regression The output of a regression is a function that predicts the dependent variable based upon values of the independent variables Simple Linear Regression The function will make a prediction for each observed data point The observation is denoted by y and the prediction is denoted by ݕ ො

ISE 135 ‐ Fall 2016 11 Simple Linear Regression For each observation, the variation can be described as ݕ ൌ ݕ ො ൅ ߝ Actual = Explained + Error Observation (measured Prediction Sum of Square Error (SSE) ¾ A least squares regression selects the line with the lowest total sum of squared prediction errors ¾ This value is called the Sum of Squares of Error, or SSE

ISE 135 ‐ Fall 2016 13 Example Given: ݕ ത ൌ 210.35 n = 14 Age (x) Total Cholesterol (y) Age (x) Total Cholesterol (y) 25 180 42 185 25 195 48 204 28 186 51 221 32 180 51 243 32 210 58 208 32 197 62 228 38 239 65 269 Regression SUMMARY OUTPUT Regression Statistics Multiple R 0.721832 R Square 0.521041 Adjusted R Square 0.481128 Standard Error 19.2552 Observations 14 Coefficients Standard Error t Stat P‐value Lower 95% Upper 95% Intercept 151.4989457 17.08383409 8.867971 1.29E‐06 114.2764688 188.7214226 X Variable 1 1.399006383 0.387206113 3.6130793 0.003559 0.555356737 2.24265603 df SS MS F Significance F Regression 1 4840.062441 4840.0624 13.05434 0.00355946 Residual 12 4449.151844 370.76265 Total 13 9289.214286

ISE 135 ‐ Fall 2016 14 Standard Error of Regression ¾ A measure of its variability – It can be used in a similar manner to standard deviation, allowing for prediction intervals ¾ y ± 2 standard errors will provide approximately 95% accuracy, – And 3 standard errors will provide a 99% confidence interval ¾ Standard Error is calculated by taking the square root of the average prediction error ܵݐܽ݊݀ܽݎ݀ ܧݎݎ݋ݎ ൌ ܵܵܧ ݊ െ ݇ ¾ Where n is the number of observations in the sample and k is the total number of variables in the model Standard Error ¾ Shows the variability of data around regression line ¾ Also called – Root Mean Square Error (RMSE) – Residuals Standard Error – Denoted by S in Minitab ࢙ ࢋ ൌ ∑ ࢟ ࢏ െ ࢟ ࢏ ෝ ૛ ࢔ െ ૛ ൌ ࡿࡿࡱ ࢔ െ ૛ ൌ ∑ ࡾࢋ࢙࢏ࢊ࢛ࢇ࢒࢙ ૛ ࢔ െ ૛

ISE 135 ‐ Fall 2016 16 Slope and Intercept Properties Slope Properties Intercept Properties 1 1 ) ˆ ( E E E V ( ˆ E 1 ) V 2 S xx se ( ˆ E 1 ) V 2 S xx » » ¼ º « « ¬ ª ± V E E E xx S x n V E 2 2 0 0 0 1 ) ˆ ( and ) ˆ ( se ( ˆ E 0 ) V 2 1 n ± x 2 S xx ª ¬ « º ¼ » Regression SUMMARY OUTPUT Regression Statistics Multiple R 0.721832 R Square 0.521041 Adjusted R Square 0.481128 Standard Error 19.2552 Observations 14 Coefficients Standard Error t Stat P‐value Lower 95% Upper 95% Intercept 151.4989457 17.08383409 8.867971 1.29E‐06 114.2764688 188.7214226 X Variable 1 1.399006383 0.387206113 3.6130793 0.003559 0.555356737 2.24265603 df SS MS F Significance F Regression 1 4840.062441 4840.0624 13.05434 0.00355946 Residual 12 4449.151844 370.76265 Total 13 9289.214286 ࡱ ࢼ ෡ ૙ ൌ ࢼ ૙ ൌ ૚૞૚. ૝ૡૢ ࢙ࢋ ࢼ ෡ ૙ ൌ ૚ૠ. ૙ૡ૜

ISE 135 ‐ Fall 2016 17 Regression SUMMARY OUTPUT Regression Statistics Multiple R 0.721832 R Square 0.521041 Adjusted R Square 0.481128 Standard Error 19.2552 Observations 14 Coefficients Standard Error t Stat P‐value Lower 95% Upper 95% Intercept 151.4989457 17.08383409 8.867971 1.29E‐06 114.2764688 188.7214226 X Variable 1 1.399006383 0.387206113 3.6130793 0.003559 0.555356737 2.24265603 df SS MS F Significance F Regression 1 4840.062441 4840.0624 13.05434 0.00355946 Residual 12 4449.151844 370.76265 Total 13 9289.214286 ࡱ ࢼ ෡ ૚ ൌ ࢼ ૚ ൌ ૚. ૜ૢૢ ࢙ࢋ ࢼ ෡ ૚ ൌ ૙. ૜ૡૠ Inference on Slope and Intercept ¾ Test the claim that a linear relationship exists between the explanatory and the response variables ¾ Do the sample provide sufficient evidence to support the claim that a linear relationship exists between the two variables? ¾ If there is no linear relation between the response and explanatory variables – Slope of the true regression line will be zero – Slope of zero Æ x does not change our “guess” as to the value of y

ISE 135 ‐ Fall 2016 19 Regression SUMMARY OUTPUT Regression Statistics Multiple R 0.721832 R Square 0.521041 Adjusted R Square 0.481128 Standard Error 19.2552 Observations 14 Coefficients Standard Error t Stat P‐value Lower 95% Upper 95% Intercept 151.4989457 17.08383409 8.867971 1.29E‐06 114.2764688 188.7214226 X Variable 1 1.399006383 0.387206113 3.6130793 0.003559 0.555356737 2.24265603 df SS MS F Significance F Regression 1 4840.062441 4840.0624 13.05434 0.00355946 Residual 12 4449.151844 370.76265 Total 13 9289.214286 T‐stat = 1.399/0.3872 = 3.613 Confidence Interval about Slope ¾ Confidence interval = Point Estimate ± Margin of Error ¾ (1‐ α) * 100% Confidence Interval for the slope of the true regression line, β 1 , is given by ߚ ଵ ෢ േ ݐ ഀ మ ⁄ ,௡ିଶ ∗ ௦ ೐ ௌ ೣೣ ¾ (1‐ α) * 100% Confidence Interval for the intercept of the true regression line, β 0 , is given by ߚ ଴ ෢ േ ݐ ഀ మ ⁄ ,௡ିଶ ∗ ݏ ௘ ଵ ௡ ൅ ௫̅ మ ௌ ೣೣ

ISE 135 ‐ Fall 2016 20 Regression SUMMARY OUTPUT Regression Statistics Multiple R 0.721832 R Square 0.521041 Adjusted R Square 0.481128 Standard Error 19.2552 Observations 14 Coefficients Standard Error t Stat P‐value Lower 95% Upper 95% Intercept 151.4989457 17.08383409 8.867971 1.29E‐06 114.2764688 188.7214226 X Variable 1 1.399006383 0.387206113 3.6130793 0.003559 0.555356737 2.24265603 df SS MS F Significance F Regression 1 4840.062441 4840.0624 13.05434 0.00355946 Residual 12 4449.151844 370.76265 Total 13 9289.214286 UCL (intercept) = 151.49 + 17.08*2.179 = 188.72 LCL (intercept) = 151.49 ‐ 17.08*2.179 = 114.27 Correlation Coefficient(R) and Coefficient of Determination (R 2 ) ¾ R : Correlation Coefficient – Magnitude of the relationship between the dependent variable and the best linear combination of the predictor variables ¾ R 2 : Coefficient of Determination – The proportion of variation in Y accounted for by the set of independent variables (X’s)