Simple Linear Regression

187. Simple Linear Regression

𝑦 = 𝛽_{0} + 𝛽_{1} 𝑥_{1} + 𝜀

$𝑦$ : dependent variable
$𝑥$ : independent variables
$𝛽_{0}$ : intercept (value of $𝑦$ when $𝑥 = 0$ )
$𝛽_{1}$ : slope (change in $𝑦$ for a one-unit change in $𝑥$ )
$𝜀$ : error term (difference between the actual data points and the predicted values)

Example

Data

$𝑥$ (Hours Studied)	$𝑦$ (Test Score)
1	2
2	3
3	5
4	4
5	6

Step 1: Calculate Means

\bar{𝑥} = 3

\bar{𝑦} = 4

Step 2: Calculating Slope $𝛽_{1}$

𝛽_{1} = \frac{\sum_{𝑖 = 1}^{𝑛} (𝑥_{𝑖} - \bar{𝑥}) (𝑦_{𝑖} - \bar{𝑦})}{\sum_{𝑖 = 1}^{𝑛} {(𝑥_{𝑖} - \bar{𝑥})}^{2}}

The numerator

\sum_{𝑖 = 1}^{𝑛} (𝑥_{𝑖} - \bar{𝑥}) (𝑦_{𝑖} - \bar{𝑦}) = 9

The denominator

\sum_{𝑖 = 1}^{𝑛} {(𝑥_{𝑖} - \bar{𝑥})}^{2} = 10

The slope $𝛽_{1}$ is

𝛽_{1} = \frac{9}{10} = 0.9

Step 3: Calculate Intercept $𝛽_{0}$

𝛽_{1} = \bar{𝑦} - 𝛽_{1} \bar{𝑥} = 1.3

Step 4: Calculate p-Value

Calculate Standard Error of the Slope ( $𝑆 𝐸_{𝛽_{1}}$ )

𝑆 𝐸_{𝛽_{1}} = \sqrt{\frac{\sum_{𝑛 = 1}^{𝑛} {(𝑦_{𝑖} - {\hat{𝑦}}_{𝑖})}^{2}}{(𝑛 - 2) \cdot \sum_{𝑖 = 1}^{𝑛} {(𝑥_{𝑖} - \bar{𝑥})}^{2}}}

Calculate the Residual Sum of Squares (RSS)

𝑅 𝑆 𝑆 = \sum_{𝑖 = 1}^{𝑛} {(𝑦_{𝑖} - {\hat{𝑦}}_{𝑖})}^{2}

Calculate the t-statistic for the Slope

𝑡 = \frac{𝛽_{1}}{𝑆 𝐸_{𝛽_{1}}}

Determine Degrees of Freedom

df = 𝑛 - 2

Look up the p-value corresponding to $𝑡$ with 3 degrees of freedom

Step 5: Calculate $𝑅^{2}$ ( $𝑅_{adj}^{2}$ )

𝑅^{2} = \frac{{SS}_{reg}}{{SS}_{total}}

${SS}_{reg}$ (Regression Sum of Squares): sum of the squared differences between the predicted $\hat{𝑦}$ values and the mean of the observed $𝑦$ values.

{SS}_{reg} = \sum_{𝑖 = 1}^{𝑛} {({\hat{𝑦}}_{𝑖} - \bar{𝑦})}^{2}

${SS}_{total}$ (Total Sum of Squares): sum of the squared differences between the observed $𝑦$ values and the mean of the observed $𝑦$ values.

{SS}_{total} = \sum_{𝑖 = 1}^{𝑛} {(𝑦_{𝑖} - \bar{𝑦})}^{2}

Adjusting for the number of independent variables

𝑅_{adj}^{2} = 1 - (\frac{(1 - 𝑅^{2}) (𝑛 - 1)}{𝑛 - 𝑘 - 1})

$𝑛$ : number of observations
$𝑘$ : number of independent variables