01-01-linear-regression.Rmd

# (PART) Introductory Univariate Statistics {-}

# Simple Linear Regression {#simple_linear_regression}

The first analysis that we will go over is the simple linear regression. Typically, this is one of the last analyses that is taught in introductory statistics in psychology; however, we will go over the simple linear regression first because its steps are essentially identical to the GLM (i.e., OLS regression) and provides a foundation for the GLM framework.

The simple linear regression is typically taught to be used in understanding the relationship between two continuous variables, especially if one can predict the other.

However, the simple linear regression is a specific case of regression [^2] where variables can actually be either continuous or categorical, the number of predictor variables can be more than one, and causation is not a requirement. Thus, regression is applicable to most data types.

[^2]: Specifically, the simple linear regression is a specific case of ordinary least squares (OLS) regression. OLS is one of the several ways that we can calculate the error term.

The simple linear regression is written as:

$$Y = \beta_0 + \beta_1X + \varepsilon$$

where $Y$ is the dependent (or response) variable, $\beta_0$ is the intercept, $\beta_1$ is the slope, $X$ is the independent (or predictor) variable, and $\varepsilon$ is the error.

We will use the terms dependent variable (DV) and response variable interchangeably throughout the book. We will also use the terms independent variable (IV) or predictor variable interchangeably.

$\beta$ represents the true population value, while $b$ represents the unstandardized estimate of the true population value that we will obtain from our analysis. For example, $\beta_1$ represents the true population slope, while $b_1$ represents the unstandardized estimate of the true population slope (or sample).

Thus, a predicted model (as opposed to the above true hypothesized model) would be written as:

$$\hat{Y} = b_0 + b_1X$$

where $\hat{Y}$ represents the predicted dependent (or response) variable.

Both equations should look familiar to the slope-intercept form learned in algebra:

$$y = mx + b$$

where $b$ is the intercept, and $m$ is the slope.

The intercept and slope have the **same** meaning as they did in algebra. The intercept represents the value of $Y$ when $X$ is 0, and the slope represents the change in $Y$ ($\Delta Y$) over the change in $X$ ($\Delta X$). In other words, the intercept represents the value of the DV when the IV is 0, and the slope represents the change in the DV over the change in the IV.

Note that the $b$ in the slope-intercept form is not the same and should not be confused with the $b$ in the simple linear regression. The $b$ in the simple linear regression is the slope, while the $b$ in the slope-intercept form is the intercept. Unfortunately, using the same symbol for different terms and using different symbols for the same term is prevalent in statistics.

For a practical example of the simple linear regression, let's say that we were interested in examining if there is a relationship between the amount of years since a professor has earned their Ph.D. and their salary.

For this example, we will use the [`datasetSalaries`][Salaries Dataset] dataset.

## Null and research hypotheses
$$Model: Salary = \beta_0 + \beta_1 YearsSincePhD + \varepsilon$$
$$H_0: \beta_1 = 0$$
$$H_1: \beta_1 \ne 0$$

where $\beta_0$ is the intercept, $\beta_1$ is the slope, $H_0$ is the null hypothesis and $H_1$ is the research or alternative hypothesis. Typically, the model is written as the hypothesized true model and thus the use of $\beta$ rather than $b$.

In this example, the null hypothesis states that the slope is equal to 0, or that there is no relationship between $X$ (years since a professor has earned their Ph.D.) and $Y$ (their 9-month academic salary). (Note: The null hypothesis is sometimes referred to as the nil hypothesis in psychology since we are typically comparing our estimate against 0, or nil).

The research hypothesis states that the slope is not equal to 0, or that there is a relationship between $X$ (years since a professor has earned their Ph.D.) and $Y$ (their 9-month academic salary). 

The model, null hypothesis, and research hypothesis are identical for both traditional and GLM approaches.

The intercept can also be tested and its null and research hypotheses are:
$$H_0: \beta_0 = 0$$
$$H_1: \beta_0 \ne 0$$

The null hypothesis states that the intercept is equal to 0. In other words, the value of $Y$ (9-month academic salary) when $X$ (years since a professor has earned their Ph.D.) is 0 is not different than 0 (\$0). (Note: Zero represents professors who have just earned their Ph.D. up to those who have earned their Ph.D. for less than a year because `yrs.since.phd` was collected as integers and anything less than a year would be coded as 0.)

The research hypothesis states that the intercept is not equal to 0. In other words, the value of $Y$ (9-month academic salary) when $X$ (years since a professor has earned their Ph.D.) is 0 is different than 0 (\$0)

Again, the null and research hypotheses are identical for both traditional and GLM approaches.

## Statistical analyses
To perform the simple linear regression in R, we can use the `lm()` function.
```{r}
model <- lm(formula = salary ~ 1 + yrs.since.phd, data = datasetSalaries)
```

The first input is the formula of the model, which is the DV, followed by a tilde sign (`~`), followed by the IV. (Note: 1 is a placeholder for the intercept. The same analysis with identical results can be obtained without the 1. However, we place 1 there to remind us that 1 represents the intercept to gain intuition of the formula).

In our case, the DV of `salary` is be the first variable, followed by a `~` and the IV of `yrs.since.phd`. Notice the similarity between the model and formula. The second input is the dataset (i.e., `datasetSalaries`).

We will save the analysis into an object named `model`. `model` can be any name and was chosen for simplicity.

<br>

We can obtain the ANOVA source table by using the `Anova` function within the `cars` package.

```{r}
Anova(model, type = 3)
```

Note: We prefer the `Anova` function with a capitol **A** over the `anova` function with a lowercase **a** because we can specify the type of sum of squares. Breifly, The sum of squares in the `anova` function is type I sums of squares and cannot be changed while we can specify type III sums of squares, which is the default among most statistical softwares, using the `Anova` function. To learn more about different types of sum of squares, check out Mangiafico's <a href="https://rcompanion.org/rcompanion/d_04.html" target="_blank">explanation</a>.

### How do we read the ANOVA source table?
We can look at the variable of interest in the first column and read across that particular variable's row for its statistical information.

For example, let's first examine the statistics associated with the variable `(Intercept)`. The sums of squares (`Sum Sq`, which is also abbreviated as $SS$) associated with the `(Intercept)` is `8.37e+11`, the degrees of freedom (`Df`, specifically, degrees of freedom reduced) is `1`, the *F*-statistic (`F value`) is 1099.71, and the p-value (`Pr>F`) is `< 2.2e-16`.

The `F value` or *F*-statistic conceptually provides a ratio of explained to unexplained variability. Explained variability is how much the DV can be explained by IVs while unexplained variability is how much variability is not explained by the IV.  

The mathematical formula of the *F*-statistic can conceptually be thought of as:

$$F = \frac{explained\ variability}{unexplained\ variability}$$

Thus, if the IV can't explain any of the variability within the DV, then the *F*-statistic is less than or equal to 1.

However, if the IV(s) can explain the DV, the *F*-statistic will be larger than 1. In other words, the higher the *F*-statistic, the more the IV(s) explain(s) the variability in the DV compared to its inability to explain the DV.

Thus, the interpretation of the *F*-statistic of the `(Intercept)` is that the `(Intercept)` can explain the variability in `salary` `1099.71` times more than the `(Intercept)`'s inability to explain the variability in salary. [^3]

[^3]: `(Intercept)`: $$F = \frac{MSR}{MSE} = \frac{\frac{SSR}{df_R}}{\frac{SSE}{df_E}} = \frac{\frac{8.37e11}{1}}{\frac{2.99e11}{395}} = \frac{8.34e11}{7.58e8} = 1099.71$$

The `Pr(>F)` or *p*-value associated with the `Intercept` is `< 2.2e-16`. The *p*-value is the probability of obtaining that particular *F*-statistic or more extreme assuming that the null hypothesis is true. In other words, the probability of obtaining this *F*-statistic by chance is extremely small.

Since `yrs.since.phd` was our main predictor of interest (and for extra practice), let's also look at the statistics associated with `yrs.since.phd`.

The *F*-statistic associated with `yrs.since.phd` is `84.23`, and can be interpreted as `yrs.since.phd` explains the variability in `salary` `84.23` times more than `years.since.phd`'s inability to expain the variability in salary. [^4].

[^4]: `yrs.since.phd`: $$F = \frac{MSR}{MSE} = \frac{\frac{SSR}{df_R}}{\frac{SSE}{df_E}} = \frac{\frac{6.39e10}{1}}{\frac{2.99e11}{395}} = \frac{6.39e10}{7.58e8} = 84.23$$

The *p*-value associated with `yrs.since.phd` is also `< 2.2e-16` with identical interpretations for the *p*-value of the `(Intercept)`.

<br>

We can also obtain the coefficients  table by using the `summary()` function.
```{r}
summary(model)
```

### How do we read the coefficients table?

Similarly, we can look at the variable of interest in the first column and read across that particular variable's row for its statistical information.

Again, let's first take a look at the statistics associated with the `(Intercept)`. The `Estimate` of the `(Intercept)` is `91718.7`, the standard error (`Std. Error`) is `2765.8`, the *t*-statistic (`t value`) is `33.162`, and the *p*-value (`(Pr>|t|)`) is `< 2e-16`.

The `Estimate` of the `(Intercept)` is the predicted intercept value, $b_0 = 91718.7$. Thus, the predicted `salary` when $yrs.since.phd = 0$ is $91,718.70. In other words, when a professor has had their Ph.D. for less than a year, their salary is predicted to be \$91,718.70.

The `t value` or *t*-statistic is similar to the *F*-statistic, and can also be conceptually thought of as the ratio of explained variability to unexplained variability. The *t*-statistic is equal to the square root of the *F*-statistic when the degrees of freedom is 1 [^5]  [^6]. Thus, the *t*-statistic interpretation is that the `(Intercept)` explains the variability in `salary` `33.162` times more than its inability to explain `salary`.

[^5]: `(Intercept)`: $$t = \frac{Estimate}{Std.\ Error} = \frac{91718.7}{2765.8} = 33.16$$
[^6]: `(Intercept)`: $$t = \sqrt{F} = \sqrt{1099.71} = 33.16$$

The *p*-value or `Pr(>|t|)` of `<2e-16` is the probability of obtaining that particular *t*-statistic or a value more extreme than the *t*-statistic assuming that the null hypothesis is true, which is the same value from the ANOVA source table.

Let's also examine statistics for `yrs.since.phd`. The `Estimate` of `yrs.since.phd` is `985.3`, the `Std. Error` is `107.4`, the `t value` is `9.18`, and the `Pr(>|t|)` is `<2e-16`.

The `Estimate` of the `yrs.since.phd` is the estimated slope, $b_1 = 985.3$. Thus, `salary` is predicted to change by `$985.30` for every 1 unit increase in `yrs.since.phd` [^7]. In other words, salary is predicted to increase by $985.30 for every additional year a professor has earned their Ph.D.

[^7]: $$b_1 = \frac{\Delta Y}{\Delta X} = \frac{\Delta DV}{\Delta IV} = \frac{\Delta salary}{\Delta yrs.since.phd} = \frac{985.3}{1} = 985.3$$

The *t*-statistic can be interpreted as `yrs.since.phd` explains the variability in `salary` 9.18 times more than `yrs.since.phd`'s inability to the explain the variability in `salary`.

The *p*-value of `yrs.since.phd` is identical to the *p*-value of the `(Intercept)` and has identical interpretations.

Now, if wanted, we can fill in our predicted model, $\hat{Y} = b_0 + b_1X$, with the estimates from the coefficients table to predict (or estimate) other values of salary given a certain number of years since a professor has earned their Ph.D.

$$\hat{salary} = b_0 + b_1*yrs.since.phd = 91718.7+985.3*yrs.since.phd$$

For example, let's say that we were interested in estimating the salary of professor's who have worked for 2 years.

$$\hat{salary} = 91718.7+985.3*yrs.since.phd = 91718.7+985.3*2 = 93689.3$$

## Statistical decision
To determine if `yrs.since.phd` is statistically significant in predicting `salary`, we will compare its *p*-value with the alpha level ($\alpha$) of `0.05`. $\alpha$ is the Type I error rate, or the probability that we are incorrectly rejecting the null hypothesis when it is actually true.

Given that the *p*-value of `<2.2e-16` for the `yrs.since.phd` is less than the alpha level ($\alpha$) of `0.05`, we will `reject` the null hypothesis.

We can also determine if the intercept is statistically significant. Given, that the `(Intercept)` had an identical *p*-value, we will also `reject` the null hypothesis.

## APA statement
A regression was performed to test if the amount of years since obtaining a Ph.D. was related to their salary. The amount of years since obtaining a Ph.D. was significantly related to their salary, *b* = 985.3, *t*(395) = 9.18, *p* < .001. For every passing year that a professor has earned their Ph.D., their salary is estimated to increase by $985.30.

## Visualization
```{r, fig.cap="A scatterplot of years since Ph.D. and salary along with the line of best fit. The gray band represents the 95% confidence interval (CI)."}
ggplot(datasetSalaries, aes(yrs.since.phd, salary)) +
  geom_point(alpha = 0.25) +
  geom_smooth(method = "lm", color = "#3182bd") +
  labs(
    x = "Years since Ph.D.",
    y = "9-Month Salary (USD)"
  ) +
  scale_y_continuous(labels = dollar) +
  theme_classic()
```