Correlation and Regression Analysis

Assoc. Prof. Boryana Bogdanova, PhD

Let’s consider the following data set
Money Supply
Inflation Rate
Growth Rate
Yi
Xi
Australia 0.1166 0.0676
Canada 0.0915 0.0519
New Zealand 0.1060 0.0815
Switzerland 0.0575 0.0339
UK 0.1258 0.0758
USA 0.0634 0.0509

Annual average MS grwoth from 1970-2001

Annual average inflation rate from 1970-2001
Scatterplot
Correlation Analysis
• Calculating and interpreting the correlation
coefficient:
• Sample covariance:
𝑛
𝑖=1 𝑋𝑖 − 𝑋 𝑌𝑖 − 𝑌
𝑐𝑜𝑣 𝑋, 𝑌 =
(𝑛 − 1)

• Sample variance and sample SD:

𝑛 2
𝑖=1𝑋𝑖 − 𝑋
𝑠𝑋2 = ; 𝑠𝑋 = 𝑠𝑋2
(𝑛 − 1)
Sample covariance and sample SD
Correlation Analysis
• Calculating and interpreting the correlation
coefficient:
• Sample correlation coefficient:

𝑐𝑜𝑣(𝑋, 𝑌)
1 : 𝑟= =
𝑠𝑋 𝑠𝑌
𝑛
𝑖=1 𝑋𝑖 − 𝑋 𝑌𝑖 − 𝑌
=
𝑛 2 𝑛 2
𝑖=1 𝑋𝑖 − 𝑋 𝑖=1 𝑌𝑖 − 𝑌
Sample correlation coefficient
Correlation Analysis
• Calculating and interpreting the correlation
coefficient:
• A measure of linear association
• No unit of measurement attached
Testing the significance of the correlation
coefficient
• Significance tests allow us to assess whether
apparent relationships between random
variables are the result of chance.
• Is the estimate of 0.8704 significantly different
from 0?
Testing the significance of the correlation
coefficient
• Assumptions: Both of the variables are normally
distributed
• Null hypothesis: the correlation in the population
is 0.
• Alternative hypothesis: the correlation in the
population is different from 0.
• The following test statistics is used:
𝒓 𝒏−𝟐
𝒕= ~𝒕(𝒏 − 𝟐)
𝟏 − 𝒓𝟐
Perform significance test

𝒓 𝒏−𝟐
𝒕= = 𝟑. 𝟓𝟑𝟔
𝟏 − 𝒓𝟐
Visualize the t-test
Perform significance test in R
Linear regression with one independent
variable:
𝒀𝒊 = 𝜷𝟏 + 𝜷𝟐 𝑿𝒊 + 𝒖𝒊

𝒀𝒊 = 𝜷𝟏 + 𝜷𝟐 𝑿𝒊 + 𝒆𝒊
Ordinary Least Squares (OLS)
• The values of 𝛽1 and 𝛽2 minimize the following
function: 𝑛
2
𝑌𝑖 − 𝛽1 − 𝛽2 𝑋2
𝑖=1
Ordinary Least Squares (OLS)

𝒏
𝒊=𝟏 𝑿𝒊 − 𝑿 𝒀𝒊 − 𝒀
𝜷𝟐 = 𝟐
𝒏
𝒊=𝟏 𝑿𝒊 − 𝑿

𝜷𝟏 = 𝒀 − 𝜷𝟐 𝑿
OLS: An illustrative example
OLS: Fitted values and Residuals
OLS Assumptions

• Zero mean value of 𝑢𝑖

• No autocorrelation between 𝑢’s
• Homoscedasticity or equal variance of 𝑢𝑖
• Zero covariance between 𝑢𝑖 and 𝑋𝑖
• The regression model is correctly specified
Precision or standard errors of OLS
estimates
𝝈𝟐
𝒗𝒂𝒓 𝜷𝟐 = 𝟐
𝒏
𝒊=𝟏 𝑿𝒊 − 𝑿

𝝈
𝑺𝑬 𝜷𝟐 =
𝒏 𝟐
𝒊=𝟏 𝑿𝒊 − 𝑿

𝒏 𝟐
𝒆
𝒊=𝟏 𝒊
𝝈𝟐 =
𝒏−𝟐
Precision or standard errors of OLS
estimates

𝝈
𝑺𝑬 𝜷𝟐 = =
𝒏 𝟐
𝒊=𝟏 𝑿𝒊 − 𝑿

𝑛 2
𝑒
𝑖=1 𝑖
= 2
𝑛
𝑛−2 𝑖=1 𝑋𝑖 −𝑋
Precision or standard errors of OLS
estimates

𝒏 𝟐
𝑿
𝒊=𝟏 𝒊
𝑺𝑬 𝜷𝟏 = 𝟐
𝝈=
𝒏
𝒏 𝒊=𝟏 𝑿𝒊 − 𝑿

𝑛 2 𝑛 2
𝑋
𝑖=1 𝑖 𝑒
𝑖=1 𝑖
= 2
𝑛
𝑛 𝑛 − 2 𝑖=1 𝑋𝑖 − 𝑋
Precision or standard errors of OLS
estimates: An illustrative example
Gauss-Markov Theorem

Given the assumptions of the classical linear

regression model, the OLS estimators, in the
class of unbiased linear estimators, have
minimum variance, that is, they are BLUE.
Reading

• Ch. 3 of the textbook.