Correlation and Regression Essentials

Correlation measures the linear relationship between two variables, while regression analysis uses this relationship to predict one variable from another. Understanding these concepts is key to interpreting data and making informed predictions.

Core Principles

Correlation quantifies the strength and direction of a linear association between two variables.
Regression analysis models the relationship between variables to predict outcomes.
Regression to the mean occurs with imperfect correlations, where predicted values are less extreme than the observed values.
The regression line minimizes the sum of squared errors (residuals).
The correlation coefficient (r) ranges from -1 to +1.
The coefficient of determination (r²) represents the proportion of variance in the dependent variable explained by the independent variable.

Action Steps

Calculate the covariance between variables X and Y.
Calculate the standard deviations for X and Y.
Compute the correlation coefficient (r) by dividing covariance by the product of standard deviations.
Determine the slope (b) of the regression line using r, Sy, and Sx.
Calculate the y-intercept (a) using the means of X and Y, and the slope.
Formulate the regression equation: Ŷ = bX + a.
Use the regression equation to predict Y for a given value of X.
Interpret the residuals to assess the accuracy of predictions.

Formulas

$S_{XY} = \frac{\sum (X - M_X)(Y - M_Y)}{n-1}$
$r_{XY} = \frac{S_{XY}}{S_X S_Y}$
$ \hat{Y} = bX + a $
$ b = r \frac{S_Y}{S_X} = \frac{S_{XY}}{S_X^2} $
$ a = M_Y - bM_X $
$ t = \frac{r - \rho}{\sqrt{\frac{1-r^2}{n-2}}} $
$ S_{Y-\hat{Y}} = S_Y \sqrt{(1-r^2)\left(\frac{n-1}{n-2}\right)} $

Key Terms

Covariance: The product of deviation scores from two variables, indicating how they vary together.
Correlation Coefficient (r): A measure of the linear association between two variables, ranging from -1 to +1.
Regression Line: The line that best fits the data points on a scatter plot, used for prediction.
Slope (b): The rate of change of the dependent variable (Y) for a one-unit change in the independent variable (X).
Y-intercept (a): The value of Y when X is zero.
Residual: The difference between an observed value and the value predicted by the regression line.
Regression to the Mean: The tendency for extreme scores to be followed by scores closer to the mean.

Real World Examples

Predicting a student's exam score based on hours studied.: The regression equation can estimate a score given study time, and the correlation indicates the strength of this relationship.
Estimating a person's weight based on their height.: A regression equation derived from height-weight data can predict weight for a given height, considering the correlation between the two.
Pearson Father-Son Height Study.: Demonstrates regression to the mean, where sons' heights tend to be less extreme than their fathers' heights, especially with moderate correlation (r=0.50).

Timeline

Late 19th Century: Sir Francis Galton coins the term 'regression' while studying the inheritance of stature.
Early 20th Century: Karl Pearson develops the Pearson correlation coefficient (r) and formalizes regression analysis.
Mid-20th Century: Development of statistical software and computational methods makes regression analysis more accessible.
Late 20th Century - Present: Expansion of regression techniques to include multiple regression, logistic regression, and other advanced models.

People

Sir Francis Galton: Coined the term 'regression' and observed the phenomenon of 'regression to the mean'.
Karl Pearson: Developed the Pearson correlation coefficient (r) and foundational regression formulas.