Chapter 13.8, Problem 49ES

A common problem in experimental work is to obtain a mathematical relationship $y=f\left(x\right)$ between two variables x and y by "fitting" a curve to points in the plane that correspond to experimentally determined values of x and y, say $\left(x_1,y_1\right),\left(x_2,y_2\right),\mathrm{...},\left(x_n,y_n\right)$ The curve $y=f\left(x\right)$ is called a mathematical model of the data. The general form of the function f is commonly determined by some underlying physical principle, but sometimes it is just determined by the pattern of the data. We are concerned with fitting a straight line $y=mx+b$ to data. Usually, the data will not lie on a line (possibly due to experimental error or variations in experimental conditions), so the problem is to find a line that fits the data "best" according to some criterion. One criterion for selecting the line of best fit is to choose m and b to minimize the function $g\left(m,b\right)={\displaystyle \underset{i=1}{\overset{n}{\sum }}{\left(m{x}_i+b-y_i\right)}^2}$ This is called the method of least squares, and the resulting line is called the regression line or the least squares line of best fit. Geometrically, $\left|m{x}_i+b-y_i\right|$ is the vertical distance between the data point $\left(x_i,y_i\right)$ and the line $y=mx+b.$

These vertical distances arc called the residuals of the data points, so the effect of minimizing $g\left(m,b\right)$ is to minimize the sum of the squares of the residuals. In these exercises, we will derive a formula for the regression line.

The purpose of this exercise is to find the values of m and b that produce the regression line.

(a) To minimize $g\left(m,b\right),$ we start by finding values of m and b such that $\partial g/\partial m=0\text{and}\partial g/\partial b=0.$ Show that these equations are satisfied if m and b satisfy the conditions

$\begin{array}{l}\left({\displaystyle \underset{i=1}{\overset{n}{\sum }}{x}_i^2}\right)m+\left({\displaystyle \underset{i=1}{\overset{n}{\sum }}xi}\right)b={\displaystyle \underset{i=1}{\overset{n}{\sum }}{x}_i{y}_i}\\ \left({\displaystyle \underset{i=1}{\overset{n}{\sum }}xi}\right)m+nb={\displaystyle \underset{i=1}{\overset{n}{\sum }}{y}_i}\end{array}$

(b) Let $\bar{x}=\left(x_1+x_2+\cdot \cdot \cdot +x_n\right)/n$ denote the arithmetic average of $x_1,x_2,\mathrm{...},x_n.$ Use the fact that ${\displaystyle \underset{i=1}{\overset{n}{\sum }}{\left(x_i-\bar{x}\right)}^2\ge 0}$ to show that $n\left({\displaystyle \underset{i=1}{\overset{n}{\sum }}{x}_i^2}\right)-{\left({\displaystyle \underset{i=1}{\overset{n}{\sum }}{x}_i}\right)}^2\ge 0$ with equality if and only if all the $x_i$ ’s are the same.

(c) Assuming that not all the $x_i$ ’s are the same, prove that the equations in part (a) have the unique solution

$\begin{array}{l}m=\frac{n{\displaystyle \underset{i=1}{\overset{n}{\sum }}{x}_i{y}_i}-{\displaystyle \underset{i=1}{\overset{n}{\sum }}{x}_i}{\displaystyle \underset{i=1}{\overset{n}{\sum }}{y}_i}}{n{\displaystyle \underset{i=1}{\overset{n}{\sum }}{x}_i^2}-{\left({\displaystyle \underset{i=1}{\overset{n}{\sum }}{x}_i}\right)}^2}\\ b=\frac{1}{n}\left({\displaystyle \underset{i=1}{\overset{n}{\sum }}{y}_i}-m{\displaystyle \underset{i=1}{\overset{n}{\sum }}{x}_i}\right)\end{array}$

[Note: We have shown that g has a critical point at these values of m and b. In the next exercise we will show that g has an absolute minimum at this critical point. Accepting this to be so, we have shown that the line $y=mx+b$ is the regression line for these values of m and b.]