Department of Management
Science and Technology
Seminar: Researchers training
Basics on Regression Analysis
Irini Voudouris
Before starting.
Statistics is a science assisting you to make
decisions under uncertainties
Deficiencies of nonstatistician instructors teaching
statistics lead students to develop phobias for the
sweet science of statistics
Since everybody in the world thinks he can teach
statistics even though he does not know any, I shall
put myself in the position of teaching biology even
though I do not know any
Prof. Herman Chernoff
Since I do not think I can teach statistics, I will
limit myself to describing my experiences as a
statistics user
Irini Voudouris
2
Overview
Part 1: Basics of Econometrics
Part 2: Basics of Regression Analysis
Econometrics is
Methodology of Econometrics
Elements of Econometrics
Regression analysis: some basic ideas, terminology
10 assumptions underlying the linear regression model
Goodness of fit
Regression modeling selection process
What if
Part 3: Regression Analysis in praxis
Planning, development and maintenance of the model
An example: Determinants of atypical work
3
Econometrics is
an amalgam of economic theory, mathematical
economics, economic statistics, and
mathematical statistics
Gujarati
the quantitative analysis of actual economic
phenomena based on the concurrent
development of theory and observation, related
by appropriate methods of inference
Samuelson, Koopmans & Stone
using sample data on observable variables to
learn about the functional relationships among
economic variables
Abbott
5
Econometrics consists
mainly of:
estimating relationships from sample data
testing hypotheses about how variables are
related
the existence of relationships between variables
the direction of the relationships between the
dependent variable and its hypothesized observable
determinants
the magnitude of the relationships between a
dependent variable and the independent variables thought
to determine it
Methodology of
Econometrics
Statement of theory or hypothesis
Specification of the mathematical &
econometric model of the theory
Obtaining the data
Estimation of the parameters of the
econometric model
Hypothesis testing
Forecasting or prediction
Using the model for control or policy
purposes
Elements of Econometrics
Specification of the econometric model that
we think (hope) generated the sample data. It
consists of:
An economic model: specifies the dependent
variable to be explained and the independent
variables thought to be related to the dependent
Suggested or derived from theory
Sometimes obtained from informal intuition/ observation
A statistical model: specifies the statistical elements
of the relationship under investigation, in particular the
statistical properties of the random variables in the
relationship
8
Elements of Econometrics
Collecting and coding the sample data
Most economic data is observational, or nonexperimental
Sample data consist of observations on randomly selected
members of populations (individual persons, households or
families, firms, industries, provinces or states, countries)
Estimation consists of using the sample data on the
observable variables to compute estimates of the
numerical values of all the unknown parameters in the
model
Inference consists of using the parameter estimates
computed from sample data to test hypotheses about the
numerical values of the unknown population
parameters that describe the behavior of the population
from which the sample was selected
Suggestions for further
reading
Damodar N. Gujarati,
Basic Econometrics, 3d eds, Mc Graw Hill,
1995
Adrian C. Darnell & J. Lynne Evans,
The limits of Econometrics, Edward Elgar
Publishing Ltd., Hants, England, 1990
10
11
Regression analysis
Regression models:
one variable called the dependent variable is
expressed as a linear function of one or more
other variables, called the explanatory
variables
it is assumed implicitly that causal
relationships, if any, between dependent and
explanatory variables flow in one direction
only, namely, from the explanatory variables to
the dependent variables
12
Regression Analysis: Some
basic ideas
Key idea: the statistical dependence of one variable on one or more other
variables
Objective: to estimate and/or predict the mean or average value of the
dependent variable on the basis of the known or fixed values of the explanatory
variables
Yi = f(X2i ,, Xki) + ui = 1 + 2X2i + +kXki + ui
Population Regression Function (PRF)
The PRF is an idealized concept
Generally one uses the stochastic sample regression (SRF) function to estimate
the PRF
Ordinary least squares (OLS) is the most used method of constructing the SFR
Sample Data: A random sample of N members of the population for which
the observed values of Y and Xkis are measured
Sample size is critical because it influences the confidence level of conclusions
The larger the sample, the higher the associated confidence
BUT larger samples require more effort & time
13
Regression Analysis:
Terminology
Where
Yi , Xi , ui variables of the regression model
Yi dependent or explained variable or regressand
Xi independent or explanatory variable or
regressor
Ui random error term
Yi, Xi are observable variables
it is unobservable variable
It is called residual for sample observation
1 and 2 parameters of the regression model
1 intercept coefficient,
2 slope coefficient of X
they are called regression coefficients
the true population values of 1 and 2 are unknown
They are called estimators of the regression coefficients for
14
the sample observation
10 assumptions underlying the
linear regression model
Linear regression model
X values are fixed in repeated sampling
More technically X is assumed to be nonstochastic
Zero mean value of random error term ui
Homoscedasticity or equal variance of ui
It is always linear in the coefficients being estimated, not
necessarily linear in the variables
A scatterplot is essential to examining the relationship
between the two variables
It means that Y populations corresponding to various X values
have the same variance
No autocorrelation between the random error terms
In other words ui & uj are uncorrelated
15
10 assumptions underlying the
linear regression model
Zero covariance between ui and Xi
The number of observations n must be greater than the
number of parameters to be estimated
The X values in a given sample must not be all the same
The regression model is correctly specified
Alternatively the number of observations must be greater than
the number of explanatory variables
Variability in X values
The error term and explanatory variable are uncorrelated
There is no bias or error in the model used in empirical
analysis
There is no perfect multicollinearity
There are no perfect linear relationships among the
explanatory variables, the variables should be independent
16
Goodness of fit
The coefficient of determination r2 (2 variable
case) or R2 (multiple variable case) is a
summary measure that tells how well the
regression line fits the data
It measure the proportion or percentage of the total
variation in Y explained by the regression model
It is nonnegative quantity (0 R2 1)
The coefficient of correlation r is a measure of
the degree of association between two
variables
Quantity closely related but conceptually very much
different from r2
17
Regression vs Correlation
Correlation analysis
Primary objective: to measure the degree of linear association
between two variables
Symmetry in the way variables are treated
Both variables are assumed to be random
It does not imply any cause and effect relationship
The correlation between two random variables is often due only to
the fact that both variables are correlated with the same third
variable
Regression analysis
Primary objective: to estimate or predict the average value of one
variable on the basis of fixed values of other variables
Asymmetry in the way dependent & explanatory variables are
treated
Dependent variable is random (normal probability distribution)
Independent variables are assumed to have fixed values
It implies causality
18
Regression modeling selection
process
When you have more than one regression
equation based on data, to select the "best
model", you should compare:
R2 or adjusted R2, i.e., the percentage of
variance in Y accounted for variance in X
captured by the model
Standard deviation of error terms, i.e., observed
yvalue/ predicted yvalue for each X
The Tstatistic of individual parameters
The values of the parameters and its content to
content underpinnings
19
20
What ifwe have qualitative
explanatory variables?
Introduce dummy variables Di (taking values: 1,0)
dummies can be used in regression models just as easily as
quantitative variables
If there is only dummy explanatory variables use ANOVA model
(Analysis of Variance)
Possibly consider Interactions effects
There may be interaction between two dummies so that their effect on
mean Y is not simply additive but also multiplicative
The dummy variable technique must be handled carefully
The number of dummy variables must be less than the number of
classifications of each qualitative variable
The coefficient attached to the dummy variable must be always
interpreted in relation to the base, that is the group that gets the
value of zero
Weight the number of dummies introduced against the number of
observations
21
What if we have dummy
dependent variable?
Use a dummy dependent variable
regression model
Logistic regression model
Probit
Logit
Tobit
22
What if
In regression analysis involving time series
the regression includes not only current
but also lagged values of the explanatory
variables (Xs)?
Distributed lag model
the model includes one or more lagged
values of the dependent variable (Yt1)
among its explanatory variables?
Autoregressive (dynamic) model
23
What if
a regression model has been estimated
using the available data sample and an
additional data sample become available?
Use the analysis of covariance (ANCOVA) to
test if previous model is still valid or the two
separate models are equivalent or not
24
25
Planning the model
Define the problem; select response; suggest variables.
Collect data
Check the quality of data
plot; try models
Find the basic statistics, correlation matrix and first regression runs
Establish goal: The final equation should have
Are the proposed variables fundamental to the problem, are they variables?
Are they measurable/countable?
Can one get a complete set of observations at the same time?
Is the problem potentially solvable?
adjusted R2 = 0.8
coefficient of variation of less than 0.10
appropriate number of predictors
estimated coefficients must be significant at m = 0.05
no pattern in the residuals
Are goals and budget acceptable?
26
Development of the model
Check the regression conditions
Remove outliers they may have major impact
Examine all the points in the scatter diagram is Y a linear
function of X? Consider transformation
The distribution of the residual must be normal
The residuals should have mean equal to zero, and
constant standard deviation
The residuals constitute a set of random variables
Check for residuals autocorrelation
Consult experts for criticism
Plot new variable and examine same fitted model
DurbinWatson (DW) (values [0,4])
No correlation ~2
Also transformed predictor variable may be used
Are goals met?
Have you found "the best" model?
27
Validation & maintenance of
the model
Are parameters stable over the sample
space?
Is there a lack of fit?
Are the coefficients reasonable?
Are any obvious variables missing?
Is the equation usable for control or for
prediction?
Maintenance of the Model
Need to have control chart to check the model
periodically by statistical techniques
28
An example: Determinants of
atypical work
Hypotheses drawn from three
perspectives:
the integration perspective
the work environment perspective
the capacity planning perspective
Identification of three groups of variables
29
Hypotheses
Factor
category
Integration
costs
Explanatory Factors
Environment
Firm
Atypical
workers
Training costs
Fringe benefits
Technological complexity
Interpersonal complexity
Specific knowhow
+

Employment variability
Demands predictability
Unionization
Difficulty of finding
workers on the external
labor market
Industry capital intensive
Size
Number of nonfrequent
tasks
Difficulty of monitoring
the employees
+
+

30
Research Methodology
An empirical evaluation of the factors
affecting:
firms decision to use temporaries, independent
contractors & subcontractors
Logistic Regression
the proportion of the firms labor force
belonging to temporaries, independent
contractors & subcontractors
Regression
31
Data selection
Data from 75 companies belonging to 4
industrial sectors:
construction (29%)
metallurgy (30%)
oil/ chemical (25%)
electrical/ electronical (16%)
Data selection based on:
semidirected personal interviews
questionnaires
complementary data through internal documents
32
Data selection
Examination of 3 types of atypical workers:
temporaries
independent contractors
subcontractors
for 6 task categories:
engineers
managers, financiers, counsels
technologists
administrative personnel
workers
subsidiary personnel
450 observations
33
Measures
Independent variables
Measures
Benefit costs
Two measures based on a proportional
scale
1 indicator based on an additive 5
points scale of 3 items
4 measures based on a proportional
scale
1 indicator based on an additive 5
point scale of 3 items
2 measures based on a proportional
scale
1 indicator based on a 5 points additive
scale of 4 items
1 indicator based on a 5 points additive
scale of 5 items
1 indicator based on a 5 points additive
scale of 3 items
1 indicator based on a 5 points additive
scale of 2 items
1 measure based on a proportional
scale
1 indicator based on a 5 points additive
scale of 3 items
1 item
Training
Employment
variability
Demands
predictability
Unionization
Monitoring problems
Interpersonal
complexity
Technological
complexity
Firm specific
how
Size
Low frequency
Difficult to find
know
Cronbachs
Alpha
= 0,8175
= 0,844
= 0,94
= 0,9364
= 0,9048
= 0,8736
= 0,9048
34
Results: Determinants of
Temporaries use
Benefit costs
Training
Employment variability
Demands predictability
Difficult to find
Unionization
Monitoring problems
Size
Low frequency
Interpersonal complexity
Technological complexity
Firm specific knowhow
Sample, N=
2
R2
Logistic
regression
Regression
0,052
2,260***
0,760*
0,985*
1,394**
0,037***
3,450***
0,001
0,090
5,007***
0,142
1,196**
450
453,032**
*

0,078
3,145**
1,477*
0,048
0,021
0,060**
16,163***
0,072
2,141**
0,047
0,014
0,052
140
0,723
*** significant at 0,001, ** significant at 0,01, * significant at 0,05
35
Inference: Temporaries
Mostly lowskilled workers performing well
defined elementary tasks
Training costs & monitoring problems are
the most influential factors
Companies tend to resort to temporaries
when their integration is easy & cheap
This policy can increase the volume
flexibility
36
Results: Determinants of
Independent Contractors use
Benefit costs
Training
Employment variability
Demands predictability
Difficult to find
Unionization
Monitoring problems
Size
Low frequency
Interpersonal complexity
Technological complexity
Firm specific knowhow
Sample, N=
2
R2
Logistic
regression
Regression
0,0508
0,513*
1,118***
0,128
0,814*
0,009
0,830**
0,006*
1,126***
0,064
0,408
0,196
450
398,407***
0,915**
0,013
0,046
5,897***
6,076***
0,083**
5,010***
0,064**
7,326***
0,062
0,014
0,059
160


0,635
*** significant at 0,001, ** significant at 0,01, * significant at 0,05
37
Inference: Independent
Contractors
Mostly high skilled workers often used for
the realization of fixed term projects
Low frequency, difficulty to find
& monitoring problems are the most
influential factors
This policy can increase firms capacity to
adapt to unanticipated conditions implying
new set of competencies
38
Results: Determinants of
Subcontractors use
Benefit costs
Training
Employment variability
Demands predictability
Difficult to find
Unionization
Monitoring problems
Size
Low frequency
Interpersonal complexity
Technological complexity
Firm specific knowhow
Sample, N=
2
R2
Logistic
regression
Regression
0,0302
0,1004
1,080***
0,252
0,008
0,034***
0,470
0,0004
0,520**
0,344
0,206
0,473*
450
248,149**
*
0,065
0,039
0,018
0,124
5,062*
0,611
0,018
0,109
5,205*
0,057
0,003
0,050
0,392
*** significant at 0,001, ** significant at 0,01, * significant at 0,05
39
Inference: Subcontractors
Both low & mediumskilled workers for
the realization of peripheral as well as of
specialized tasks
Low frequency is the most influential
factor
This policy can increase both the volume
flexibility and the creation of new set of
competencies
40
Conclusion
Multiple types of atypical workers may be found
in the same firm, for the accomplishment of
different tasks and different types of atypical
work appear to be influenced by different factors
temporaries are used mainly as a source of
quantitative flexibility
independent contractors are sources of qualitative
flexibility
subcontractors, though not very developed is however
present and used as a source of both types of
flexibility
41