Economics 20 - Prof,Anderson 1
Multiple Regression Analysis
y = b0 + b1x1 + b2x2 +,,, bkxk + u
4,Further Issues
Economics 20 - Prof,Anderson 2
Redefining Variables
Changing the scale of the y variable will
lead to a corresponding change in the scale
of the coefficients and standard errors,so no
change in the significance or interpretation
Changing the scale of one x variable will
lead to a change in the scale of that
coefficient and standard error,so no change
in the significance or interpretation
Economics 20 - Prof,Anderson 3
Beta Coefficients
Occasional you’ll see reference to a
“standardized coefficient” or,beta
coefficient” which has a specific meaning
Idea is to replace y and each x variable with
a standardized version – i.e,subtract mean
and divide by standard deviation
Coefficient reflects standard deviation of y
for a one standard deviation change in x
Economics 20 - Prof,Anderson 4
Functional Form
OLS can be used for relationships that are
not strictly linear in x and y by using
nonlinear functions of x and y – will still be
linear in the parameters
Can take the natural log of x,y or both
Can use quadratic forms of x
Can use interactions of x variables
Economics 20 - Prof,Anderson 5
Interpretation of Log Models
If the model is ln(y) = b0 + b1ln(x) + u
b1 is the elasticity of y with respect to x
If the model is ln(y) = b0 + b1x + u
b1 is approximately the percentage change
in y given a 1 unit change in x
If the model is y = b0 + b1ln(x) + u
b1 is approximately the change in y for a
100 percent change in x
Economics 20 - Prof,Anderson 6
Why use log models?
Log models are invariant to the scale of the
variables since measuring percent changes
They give a direct estimate of elasticity
For models with y > 0,the conditional
distribution is often heteroskedastic or
skewed,while ln(y) is much less so
The distribution of ln(y) is more narrow,
limiting the effect of outliers
Economics 20 - Prof,Anderson 7
Some Rules of Thumb
What types of variables are often used in
log form?
Dollar amounts that must be positive
Very large variables,such as population
What types of variables are often used in
level form?
Variables measured in years
Variables that are a proportion or percent
Economics 20 - Prof,Anderson 8
Quadratic Models
For a model of the form y = b0 + b1x + b2x2 + u
we can’t interpret b1 alone as measuring the
change in y with respect to x,we need to take into
account b2 as well,since
? ?
x
x
y
xxy
21
21
?2??
so,?2??
bb
bb
??
?
?
????
Economics 20 - Prof,Anderson 9
More on Quadratic Models
Suppose that the coefficient on x is positive and
the coefficient on x2 is negative
Then y is increasing in x at first,but will
eventually turn around and be decreasing in x
? ?
21
*
21
?2?a t be w i l l
p o i n t t u r n i n g t h e0? a n d 0?F o r
bb
bb
?
??
x
Economics 20 - Prof,Anderson 10
More on Quadratic Models
Suppose that the coefficient on x is negative and
the coefficient on x2 is positive
Then y is decreasing in x at first,but will
eventually turn around and be increasing in x
? ?
0? a n d 0? w h e n as s a m e t h e
is w h i c h,?2?a t be w i ll
p o i n t t u r n i n g t h e0? a n d 0?F o r
21
21
*
21
??
?
??
bb
bb
bb
x
Economics 20 - Prof,Anderson 11
Interaction Terms
For a model of the form y = b0 + b1x1 + b2x2 +
b3x1x2 + u we can’t interpret b1 alone as measuring
the change in y with respect to x1,we need to take
into account b3 as well,since
2
1
231
1
a t a b o v e t h ee v a lu a t e
t y p ic a lly w eon ofe f f e c t t h e
s u m m a r iz e t oso,
x
yx
x
x
y
bb ??
?
?
Economics 20 - Prof,Anderson 12
Adjusted R-Squared
Recall that the R2 will always increase as more
variables are added to the model
The adjusted R2 takes into account the number of
variables in a model,and may decrease
? ?? ?
? ?? ?
? ?? ?1
?
1
1
1
1
2
2
?
??
?
??
??
nSST
nSST
knSSR
R
?
Economics 20 - Prof,Anderson 13
Adjusted R-Squared (cont)
It’s easy to see that the adjusted R2 is just
(1 – R2)(n – 1) / (n – k – 1),but most
packages will give you both R2 and adj-R2
You can compare the fit of 2 models (with
the same y) by comparing the adj-R2
You cannot use the adj-R2 to compare
models with different y’s (e.g,y vs,ln(y))
Economics 20 - Prof,Anderson 14
Goodness of Fit
Important not to fixate too much on adj-R2
and lose sight of theory and common sense
If economic theory clearly predicts a
variable belongs,generally leave it in
Don’t want to include a variable that
prohibits a sensible interpretation of the
variable of interest – remember ceteris
paribus interpretation of multiple regression
Economics 20 - Prof,Anderson 15
Standard Errors for Predictions
Suppose we want to use our estimates to
obtain a specific prediction?
First,suppose that we want an estimate of
E(y|x1=c1,…xk=ck) = q0 = b0+b1c1+ …+
bkck
This is easy to obtain by substituting the x’s
in our estimated model with c’s,but what
about a standard error?
Really just a test of a linear combination
Economics 20 - Prof,Anderson 16
Predictions (cont)
Can rewrite as b0 = q0 – b1c1 – … – bkck
Substitute in to obtain y = q0 + b1 (x1 - c1)
+ … + bk (xk - ck) + u
So,if you regress yi on (xij - cij) the
intercept will give the predicted value and
its standard error
Note that the standard error will be smallest
when the c’s equal the means of the x’s
Economics 20 - Prof,Anderson 17
Predictions (cont)
This standard error for the expected value is not
the same as a standard error for an outcome on y
We need to also take into account the variance in
the unobserved error,Let the prediction error be
? ?
? ? ? ? ? ? ? ?
? ? ? ? ? ?? ?? ? 2
1
220020
0000
0
000
110
000
??? so,?
?? a n d 0?
???
??
bbb
???
????
????????
yseeseyV a r
uV a ryV a reV a reE
yuxxyye
kk
?
Economics 20 - Prof,Anderson 18
Prediction interval
? ?
? ?0
0 2 5.
0
0
000
1
00
??
f o r i n t e r v a l p r e d i c t i o n 9 5 % a h a v e we
?? g i v e n t h a t so,~??
esety
y
yyetesee
kn
??
??
??
Usually the estimate of s2 is much larger than the
variance of the prediction,thus
This prediction interval will be a lot wider than the
simple confidence interval for the prediction
Economics 20 - Prof,Anderson 19
Residual Analysis
Information can be obtained from looking
at the residuals (i.e,predicted vs,observed)
Example,Regress price of cars on
characteristics – big negative residuals
indicate a good deal
Example,Regress average earnings for
students from a school on student
characteristics – big positive residuals
indicate greatest value-added
Economics 20 - Prof,Anderson 20
Predicting y in a log model
Simple exponentiation of the predicted ln(y) will
underestimate the expected value of y
Instead need to scale this up by an estimate of the
expected value of exp(u)
? ? ? ?
? ? ? ?yy
NuuE
n?le x p2?e x p?
f o l l o w s asy p r e d i c t c a n c a s e I n t h i s
,0~ if )2e x p ()e x p (
2
2
?
??
?
?
Economics 20 - Prof,Anderson 21
Predicting y in a log model
If u is not normal,E(exp(u)) must be
estimated using an auxiliary regression
Create the exponentiation of the predicted
ln(y),and regress y on it with no intercept
The coefficient on this variable is the
estimate of E(exp(u)) that can be used to
scale up the exponentiation of the predicted
ln(y) to obtain the predicted y
Economics 20 - Prof,Anderson 22
Comparing log and level models
A by-product of the previous procedure is a
method to compare a model in logs with
one in levels,
Take the fitted values from the auxiliary
regression,and find the sample correlation
between this and y
Compare the R2 from the levels regression
with this correlation squared
Multiple Regression Analysis
y = b0 + b1x1 + b2x2 +,,, bkxk + u
4,Further Issues
Economics 20 - Prof,Anderson 2
Redefining Variables
Changing the scale of the y variable will
lead to a corresponding change in the scale
of the coefficients and standard errors,so no
change in the significance or interpretation
Changing the scale of one x variable will
lead to a change in the scale of that
coefficient and standard error,so no change
in the significance or interpretation
Economics 20 - Prof,Anderson 3
Beta Coefficients
Occasional you’ll see reference to a
“standardized coefficient” or,beta
coefficient” which has a specific meaning
Idea is to replace y and each x variable with
a standardized version – i.e,subtract mean
and divide by standard deviation
Coefficient reflects standard deviation of y
for a one standard deviation change in x
Economics 20 - Prof,Anderson 4
Functional Form
OLS can be used for relationships that are
not strictly linear in x and y by using
nonlinear functions of x and y – will still be
linear in the parameters
Can take the natural log of x,y or both
Can use quadratic forms of x
Can use interactions of x variables
Economics 20 - Prof,Anderson 5
Interpretation of Log Models
If the model is ln(y) = b0 + b1ln(x) + u
b1 is the elasticity of y with respect to x
If the model is ln(y) = b0 + b1x + u
b1 is approximately the percentage change
in y given a 1 unit change in x
If the model is y = b0 + b1ln(x) + u
b1 is approximately the change in y for a
100 percent change in x
Economics 20 - Prof,Anderson 6
Why use log models?
Log models are invariant to the scale of the
variables since measuring percent changes
They give a direct estimate of elasticity
For models with y > 0,the conditional
distribution is often heteroskedastic or
skewed,while ln(y) is much less so
The distribution of ln(y) is more narrow,
limiting the effect of outliers
Economics 20 - Prof,Anderson 7
Some Rules of Thumb
What types of variables are often used in
log form?
Dollar amounts that must be positive
Very large variables,such as population
What types of variables are often used in
level form?
Variables measured in years
Variables that are a proportion or percent
Economics 20 - Prof,Anderson 8
Quadratic Models
For a model of the form y = b0 + b1x + b2x2 + u
we can’t interpret b1 alone as measuring the
change in y with respect to x,we need to take into
account b2 as well,since
? ?
x
x
y
xxy
21
21
?2??
so,?2??
bb
bb
??
?
?
????
Economics 20 - Prof,Anderson 9
More on Quadratic Models
Suppose that the coefficient on x is positive and
the coefficient on x2 is negative
Then y is increasing in x at first,but will
eventually turn around and be decreasing in x
? ?
21
*
21
?2?a t be w i l l
p o i n t t u r n i n g t h e0? a n d 0?F o r
bb
bb
?
??
x
Economics 20 - Prof,Anderson 10
More on Quadratic Models
Suppose that the coefficient on x is negative and
the coefficient on x2 is positive
Then y is decreasing in x at first,but will
eventually turn around and be increasing in x
? ?
0? a n d 0? w h e n as s a m e t h e
is w h i c h,?2?a t be w i ll
p o i n t t u r n i n g t h e0? a n d 0?F o r
21
21
*
21
??
?
??
bb
bb
bb
x
Economics 20 - Prof,Anderson 11
Interaction Terms
For a model of the form y = b0 + b1x1 + b2x2 +
b3x1x2 + u we can’t interpret b1 alone as measuring
the change in y with respect to x1,we need to take
into account b3 as well,since
2
1
231
1
a t a b o v e t h ee v a lu a t e
t y p ic a lly w eon ofe f f e c t t h e
s u m m a r iz e t oso,
x
yx
x
x
y
bb ??
?
?
Economics 20 - Prof,Anderson 12
Adjusted R-Squared
Recall that the R2 will always increase as more
variables are added to the model
The adjusted R2 takes into account the number of
variables in a model,and may decrease
? ?? ?
? ?? ?
? ?? ?1
?
1
1
1
1
2
2
?
??
?
??
??
nSST
nSST
knSSR
R
?
Economics 20 - Prof,Anderson 13
Adjusted R-Squared (cont)
It’s easy to see that the adjusted R2 is just
(1 – R2)(n – 1) / (n – k – 1),but most
packages will give you both R2 and adj-R2
You can compare the fit of 2 models (with
the same y) by comparing the adj-R2
You cannot use the adj-R2 to compare
models with different y’s (e.g,y vs,ln(y))
Economics 20 - Prof,Anderson 14
Goodness of Fit
Important not to fixate too much on adj-R2
and lose sight of theory and common sense
If economic theory clearly predicts a
variable belongs,generally leave it in
Don’t want to include a variable that
prohibits a sensible interpretation of the
variable of interest – remember ceteris
paribus interpretation of multiple regression
Economics 20 - Prof,Anderson 15
Standard Errors for Predictions
Suppose we want to use our estimates to
obtain a specific prediction?
First,suppose that we want an estimate of
E(y|x1=c1,…xk=ck) = q0 = b0+b1c1+ …+
bkck
This is easy to obtain by substituting the x’s
in our estimated model with c’s,but what
about a standard error?
Really just a test of a linear combination
Economics 20 - Prof,Anderson 16
Predictions (cont)
Can rewrite as b0 = q0 – b1c1 – … – bkck
Substitute in to obtain y = q0 + b1 (x1 - c1)
+ … + bk (xk - ck) + u
So,if you regress yi on (xij - cij) the
intercept will give the predicted value and
its standard error
Note that the standard error will be smallest
when the c’s equal the means of the x’s
Economics 20 - Prof,Anderson 17
Predictions (cont)
This standard error for the expected value is not
the same as a standard error for an outcome on y
We need to also take into account the variance in
the unobserved error,Let the prediction error be
? ?
? ? ? ? ? ? ? ?
? ? ? ? ? ?? ?? ? 2
1
220020
0000
0
000
110
000
??? so,?
?? a n d 0?
???
??
bbb
???
????
????????
yseeseyV a r
uV a ryV a reV a reE
yuxxyye
kk
?
Economics 20 - Prof,Anderson 18
Prediction interval
? ?
? ?0
0 2 5.
0
0
000
1
00
??
f o r i n t e r v a l p r e d i c t i o n 9 5 % a h a v e we
?? g i v e n t h a t so,~??
esety
y
yyetesee
kn
??
??
??
Usually the estimate of s2 is much larger than the
variance of the prediction,thus
This prediction interval will be a lot wider than the
simple confidence interval for the prediction
Economics 20 - Prof,Anderson 19
Residual Analysis
Information can be obtained from looking
at the residuals (i.e,predicted vs,observed)
Example,Regress price of cars on
characteristics – big negative residuals
indicate a good deal
Example,Regress average earnings for
students from a school on student
characteristics – big positive residuals
indicate greatest value-added
Economics 20 - Prof,Anderson 20
Predicting y in a log model
Simple exponentiation of the predicted ln(y) will
underestimate the expected value of y
Instead need to scale this up by an estimate of the
expected value of exp(u)
? ? ? ?
? ? ? ?yy
NuuE
n?le x p2?e x p?
f o l l o w s asy p r e d i c t c a n c a s e I n t h i s
,0~ if )2e x p ()e x p (
2
2
?
??
?
?
Economics 20 - Prof,Anderson 21
Predicting y in a log model
If u is not normal,E(exp(u)) must be
estimated using an auxiliary regression
Create the exponentiation of the predicted
ln(y),and regress y on it with no intercept
The coefficient on this variable is the
estimate of E(exp(u)) that can be used to
scale up the exponentiation of the predicted
ln(y) to obtain the predicted y
Economics 20 - Prof,Anderson 22
Comparing log and level models
A by-product of the previous procedure is a
method to compare a model in logs with
one in levels,
Take the fitted values from the auxiliary
regression,and find the sample correlation
between this and y
Compare the R2 from the levels regression
with this correlation squared
