Chapter 12, Problem 4CR

(a) Explain the logic of the ordinary least squares (OLS) method. (b) How are the least squares formulas for the slope and intercept derived? (c) What sums are needed to calculate the least squares estimates?

To determine

Explain the logic of the ordinary least squares method.

Explanation of Solution

Ordinary least squares method:

For getting the best fit of regression, ordinary least square method can be used. The slope and the intercept are estimated in the way that the residual sum of squares will be minimized.

If only the sum of residuals has used for minimizing the error, there is possibility that the positive and the negative error will be dismissed. Therefor it is logical to use the sum of squares for minimizing the residual.

To determine

Show how the least squares formula for the slope and the intercept was derived.

Explanation of Solution

Calculation:

Let Y is the response variable and X is the predictor variable. The linear regression line is $y=b_0+b_{1}x+e$.

The error sum of squares is,

${\displaystyle \sum _i{e}_i{}^2}={{\displaystyle \sum _i\left(y_i-b_0-b_1{x}_i\right)}}^2$

For minimizing the sum of squares, the expression should be derivative by $b_1$ and $b_0$ and substituting the values ${\widehat{b}}_1\text{and }{\widehat{b}}_0$.

${\frac{\partial }{\partial b_1}{\displaystyle \sum _{i=1}^n{\left(y_i-b_0-b_1{x}_i\right)}^2}|}_{{\widehat{b}}_0}{}_{{\widehat{b}}_1}=-2{\displaystyle \sum _{i=1}^n\left(y_i-{\widehat{b}}_0-{\widehat{b}}_1{x}_i\right)}{x}_i$ (1)

${\frac{\partial }{\partial b_0}{\displaystyle \sum _{i=1}^n{\left(y_i-b_0-b_1{x}_i\right)}^2}|}_{{\widehat{b}}_0}{}_{{\widehat{b}}_1}=-2{\displaystyle \sum _{i=1}^n\left(y_i-{\widehat{b}}_0-{\widehat{b}}_1{x}_i\right)}$ (2)

Now, equating the derivatives with 0,

$\begin{array}{c}-2{\displaystyle \sum _{i=1}^n\left(y_i-{\widehat{b}}_0-{\widehat{b}}_1{x}_i\right)}=0\\ {\displaystyle \sum _{i=1}^n{y}_i-{\widehat{b}}_0{\displaystyle \sum _{i=1}^{n}1}-{\widehat{b}}_1{\displaystyle \sum _{i=1}^n{x}_i}}=0\\ n\overline{y}-n{\widehat{b}}_0=n{\widehat{b}}_1\overline{x}\\ {\widehat{b}}_0=\overline{y}-{\widehat{b}}_1\overline{x}\end{array}$

$\begin{array}{c}-2{\displaystyle \sum _{i=1}^n\left(y_i-{\widehat{b}}_0-{\widehat{b}}_1{x}_i\right)}{x}_i=0\\ {\displaystyle \sum _{i=1}^n{y}_i{x}_i-{\widehat{b}}_0{\displaystyle \sum _{i=1}^n{x}_i}-{\widehat{b}}_1{\displaystyle \sum _{i=1}^n{x}_i{}^2}}=0\\ {\widehat{b}}_1{\displaystyle \sum _{i=1}^n{x}_i{}^2}={\displaystyle \sum _{i=1}^n{x}_i{y}_i}+{\widehat{b}}_0{\displaystyle \sum _{i=1}^n{x}_i}\\ {\widehat{b}}_1{\displaystyle \sum _{i=1}^n{x}_i{}^2}={\displaystyle \sum _{i=1}^n{x}_i{y}_i}+\left(\overline{y}-{\widehat{b}}_1\overline{x}\right){\displaystyle \sum _{i=1}^n{x}_i}\\ {\widehat{b}}_1{\displaystyle \sum _{i=1}^n{x}_i{}^2}={\displaystyle \sum _{i=1}^n{x}_i{y}_i}+\overline{y}{\displaystyle \sum _{i=1}^n{x}_i}-{\widehat{b}}_1\overline{x}{\displaystyle \sum _{i=1}^n{x}_i}\\ {\widehat{b}}_1\left({\displaystyle \sum _{i=1}^n{x}_i{}^2}+n{\overline{x}}^2\right)={\displaystyle \sum _{i=1}^n{x}_i{y}_i}+n\overline{x}\overline{y}\\ {\widehat{b}}_1=\frac{{\displaystyle \sum _{i=1}^n{x}_i{y}_i}+n\overline{x}\overline{y}}{\left({\displaystyle \sum _{i=1}^n{x}_i{}^2}+n{\overline{x}}^2\right)}\end{array}$

Hence, the least square estimator of $b_1$ and $b_0$ are ${\widehat{b}}_1=\frac{{\displaystyle \sum _{i=1}^n{x}_i{y}_i}+n\overline{x}\overline{y}}{\left({\displaystyle \sum _{i=1}^n{x}_i{}^2}+n{\overline{x}}^2\right)}$ and ${\widehat{b}}_0=\overline{y}-{\widehat{b}}_1\overline{x}$ respectively.

To determine

Explain the sums which should be calculated for finding the least squares estimators.

Explanation of Solution

For finding the least square estimates, ${\widehat{b}}_1=\frac{{\displaystyle \sum _{i=1}^n{x}_i{y}_i}+n\overline{x}\overline{y}}{\left({\displaystyle \sum _{i=1}^n{x}_i{}^2}+n{\overline{x}}^2\right)}$ and ${\widehat{b}}_0=\overline{y}-{\widehat{b}}_1\overline{x}$, sum of the squared values of X and the sum of the product of X and Y values should be calculated.