Economics 20 - Prof,Anderson 1
Multiple Regression Analysis
y = b0 + b1x1 + b2x2 +,,, bkxk + u
6,Heteroskedasticity
Economics 20 - Prof,Anderson 2
What is Heteroskedasticity
Recall the assumption of homoskedasticity
implied that conditional on the explanatory
variables,the variance of the unobserved error,u,
was constant
If this is not true,that is if the variance of u is
different for different values of the x’s,then the
errors are heteroskedastic
Example,estimating returns to education and
ability is unobservable,and think the variance in
ability differs by educational attainment
Economics 20 - Prof,Anderson 3
,
x x1 x2
f(y|x)
Example of Heteroskedasticity
x3
., E(y|x) = b0 + b1x
Economics 20 - Prof,Anderson 4
Why Worry About
Heteroskedasticity?
OLS is still unbiased and consistent,even if
we do not assume homoskedasticity
The standard errors of the estimates are
biased if we have heteroskedasticity
If the standard errors are biased,we can not
use the usual t statistics or F statistics or LM
statistics for drawing inferences
Economics 20 - Prof,Anderson 5
Variance with Heteroskedasticity
? ?
? ?
? ?
? ?
? ?
? ?
r e s i d u a l s O L S t h ea r e a r e ? w h e r e,
?
is w h e n f o r t h i se s t i m a t o r A v a l i d
w h e r e,
?
so,
?
c a s e,s i m p l e F o r t h e
2
22
22
i
2
2
2
2
1
211
i
x
ii
ix
x
ii
i
ii
u
SST
uxx
xxSST
SST
xx
V a r
xx
uxx
?
?
?
?
?
?
?
??
?
?
?
?
??
??
?
b
bb
Economics 20 - Prof,Anderson 6
Variance with Heteroskedasticity
? ?
? ?
r e g r e s s i o n t h i sf r o m r e s i d u a l s s q u a r e d of s u m t h eis
a n d s,t v a r i a b l ei n d e p e n d e no t h e r a l lo n r e g r e s s i n g
f r o m r e s i d u a l t h eis ? w h e r e,
??
??
isa s t i c i t y h e t e r o s k e d w i t h
?
ofe s t i m a t o r
v a l i da m o d e l,r e g r e s s i o n m u l t i p l e g e n e r a l F o r t h e
th
2
2
j
j
ij
j
iij
j
j
S S T
x
ir
S S T
ur
rVa
V a r
?
?b
b
Economics 20 - Prof,Anderson 7
Robust Standard Errors
Now that we have a consistent estimate of
the variance,the square root can be used as
a standard error for inference
Typically call these robust standard errors
Sometimes the estimated variance is
corrected for degrees of freedom by
multiplying by n/(n – k – 1)
As n → ∞ it’s all the same,though
Economics 20 - Prof,Anderson 8
Robust Standard Errors (cont)
Important to remember that these robust
standard errors only have asymptotic
justification – with small sample sizes t
statistics formed with robust standard errors
will not have a distribution close to the t,
and inferences will not be correct
In Stata,robust standard errors are easily
obtained using the robust option of reg
Economics 20 - Prof,Anderson 9
A Robust LM Statistic
Run OLS on the restricted model and save the
residuals ?
Regress each of the excluded variables on all of
the included variables (q different regressions) and
save each set of residuals ?1,?2,…,?q
Regress a variable defined to be = 1 on ?1 ?,?2
?,…,?q ?,with no intercept
The LM statistic is n – SSR1,where SSR1 is the
sum of squared residuals from this final regression
Economics 20 - Prof,Anderson 10
Testing for Heteroskedasticity
Essentially want to test H0,Var(u|x1,x2,…,
xk) = ?2,which is equivalent to H0,E(u2|x1,
x2,…,xk) = E(u2) = ?2
If assume the relationship between u2 and xj
will be linear,can test as a linear restriction
So,for u2 = d0 + d1x1 +…+ dk xk + v) this
means testing H0,d1 = d2 = … = dk = 0
Economics 20 - Prof,Anderson 11
The Breusch-Pagan Test
Don’t observe the error,but can estimate it with
the residuals from the OLS regression
After regressing the residuals squared on all of
the x’s,can use the R2 to form an F or LM test
The F statistic is just the reported F statistic for
overall significance of the regression,F =
[R2/k]/[(1 – R2)/(n – k – 1)],which is distributed Fk,
n – k - 1
The LM statistic is LM = nR2,which is distributed
c2k
Economics 20 - Prof,Anderson 12
The White Test
The Breusch-Pagan test will detect any
linear forms of heteroskedasticity
The White test allows for nonlinearities by
using squares and crossproducts of all the x’s
Still just using an F or LM to test whether
all the xj,xj2,and xjxh are jointly significant
This can get to be unwieldy pretty quickly
Economics 20 - Prof,Anderson 13
Alternate form of the White test
Consider that the fitted values from OLS,?,
are a function of all the x’s
Thus,?2 will be a function of the squares
and crossproducts and ? and ?2 can proxy
for all of the xj,xj2,and xjxh,so
Regress the residuals squared on ? and ?2
and use the R2 to form an F or LM statistic
Note only testing for 2 restrictions now
Economics 20 - Prof,Anderson 14
Weighted Least Squares
While it’s always possible to estimate
robust standard errors for OLS estimates,if
we know something about the specific form
of the heteroskedasticity,we can obtain
more efficient estimates than OLS
The basic idea is going to be to transform
the model into one that has homoskedastic
errors – called weighted least squares
Economics 20 - Prof,Anderson 15
Case of form being known up to
a multiplicative constant
Suppose the heteroskedasticity can be
modeled as Var(u|x) = ?2h(x),where the
trick is to figure out what h(x) ≡ hi looks like
E(ui/√hi|x) = 0,because hi is only a function
of x,and Var(ui/√hi|x) = ?2,because we
know Var(u|x) = ?2hi
So,if we divided our whole equation by √hi
we would have a model where the error is
homoskedastic
Economics 20 - Prof,Anderson 16
Generalized Least Squares
Estimating the transformed equation by
OLS is an example of generalized least
squares (GLS)
GLS will be BLUE in this case
GLS is a weighted least squares (WLS)
procedure where each squared residual is
weighted by the inverse of Var(ui|xi)
Economics 20 - Prof,Anderson 17
Weighted Least Squares
While it is intuitive to see why performing
OLS on a transformed equation is
appropriate,it can be tedious to do the
transformation
Weighted least squares is a way of getting
the same thing,without the transformation
Idea is to minimize the weighted sum of
squares (weighted by 1/hi)
Economics 20 - Prof,Anderson 18
More on WLS
WLS is great if we know what Var(ui|xi)
looks like
In most cases,won’t know form of
heteroskedasticity
Example where do is if data is aggregated,
but model is individual level
Want to weight each aggregate observation
by the inverse of the number of individuals
Economics 20 - Prof,Anderson 19
Feasible GLS
More typical is the case where you don’t
know the form of the heteroskedasticity
In this case,you need to estimate h(xi)
Typically,we start with the assumption of a
fairly flexible model,such as
Var(u|x) = ?2exp(d0 + d1x1 + …+ dkxk)
Since we don’t know the d,must estimate
Economics 20 - Prof,Anderson 20
Feasible GLS (continued)
Our assumption implies that u2 = ?2exp(d0
+ d1x1 + …+ dkxk)v
Where E(v|x) = 1,then if E(v) = 1
ln(u2) = a0 + d1x1 + …+ dkxk + e
Where E(e) = 1 and e is independent of x
Now,we know that ? is an estimate of u,so
we can estimate this by OLS
Economics 20 - Prof,Anderson 21
Feasible GLS (continued)
Now,an estimate of h is obtained as ? =
exp(?),and the inverse of this is our weight
So,what did we do?
Run the original OLS model,save the
residuals,?,square them and take the log
Regress ln(?2) on all of the independent
variables and get the fitted values,?
Do WLS using 1/exp(?) as the weight
Economics 20 - Prof,Anderson 22
WLS Wrapup
When doing F tests with WLS,form the weights
from the unrestricted model and use those weights to
do WLS on the restricted model as well as the
unrestricted model
Remember we are using WLS just for efficiency –
OLS is still unbiased & consistent
Estimates will still be different due to sampling error,
but if they are very different then it’s likely that some
other Gauss-Markov assumption is false
Multiple Regression Analysis
y = b0 + b1x1 + b2x2 +,,, bkxk + u
6,Heteroskedasticity
Economics 20 - Prof,Anderson 2
What is Heteroskedasticity
Recall the assumption of homoskedasticity
implied that conditional on the explanatory
variables,the variance of the unobserved error,u,
was constant
If this is not true,that is if the variance of u is
different for different values of the x’s,then the
errors are heteroskedastic
Example,estimating returns to education and
ability is unobservable,and think the variance in
ability differs by educational attainment
Economics 20 - Prof,Anderson 3
,
x x1 x2
f(y|x)
Example of Heteroskedasticity
x3
., E(y|x) = b0 + b1x
Economics 20 - Prof,Anderson 4
Why Worry About
Heteroskedasticity?
OLS is still unbiased and consistent,even if
we do not assume homoskedasticity
The standard errors of the estimates are
biased if we have heteroskedasticity
If the standard errors are biased,we can not
use the usual t statistics or F statistics or LM
statistics for drawing inferences
Economics 20 - Prof,Anderson 5
Variance with Heteroskedasticity
? ?
? ?
? ?
? ?
? ?
? ?
r e s i d u a l s O L S t h ea r e a r e ? w h e r e,
?
is w h e n f o r t h i se s t i m a t o r A v a l i d
w h e r e,
?
so,
?
c a s e,s i m p l e F o r t h e
2
22
22
i
2
2
2
2
1
211
i
x
ii
ix
x
ii
i
ii
u
SST
uxx
xxSST
SST
xx
V a r
xx
uxx
?
?
?
?
?
?
?
??
?
?
?
?
??
??
?
b
bb
Economics 20 - Prof,Anderson 6
Variance with Heteroskedasticity
? ?
? ?
r e g r e s s i o n t h i sf r o m r e s i d u a l s s q u a r e d of s u m t h eis
a n d s,t v a r i a b l ei n d e p e n d e no t h e r a l lo n r e g r e s s i n g
f r o m r e s i d u a l t h eis ? w h e r e,
??
??
isa s t i c i t y h e t e r o s k e d w i t h
?
ofe s t i m a t o r
v a l i da m o d e l,r e g r e s s i o n m u l t i p l e g e n e r a l F o r t h e
th
2
2
j
j
ij
j
iij
j
j
S S T
x
ir
S S T
ur
rVa
V a r
?
?b
b
Economics 20 - Prof,Anderson 7
Robust Standard Errors
Now that we have a consistent estimate of
the variance,the square root can be used as
a standard error for inference
Typically call these robust standard errors
Sometimes the estimated variance is
corrected for degrees of freedom by
multiplying by n/(n – k – 1)
As n → ∞ it’s all the same,though
Economics 20 - Prof,Anderson 8
Robust Standard Errors (cont)
Important to remember that these robust
standard errors only have asymptotic
justification – with small sample sizes t
statistics formed with robust standard errors
will not have a distribution close to the t,
and inferences will not be correct
In Stata,robust standard errors are easily
obtained using the robust option of reg
Economics 20 - Prof,Anderson 9
A Robust LM Statistic
Run OLS on the restricted model and save the
residuals ?
Regress each of the excluded variables on all of
the included variables (q different regressions) and
save each set of residuals ?1,?2,…,?q
Regress a variable defined to be = 1 on ?1 ?,?2
?,…,?q ?,with no intercept
The LM statistic is n – SSR1,where SSR1 is the
sum of squared residuals from this final regression
Economics 20 - Prof,Anderson 10
Testing for Heteroskedasticity
Essentially want to test H0,Var(u|x1,x2,…,
xk) = ?2,which is equivalent to H0,E(u2|x1,
x2,…,xk) = E(u2) = ?2
If assume the relationship between u2 and xj
will be linear,can test as a linear restriction
So,for u2 = d0 + d1x1 +…+ dk xk + v) this
means testing H0,d1 = d2 = … = dk = 0
Economics 20 - Prof,Anderson 11
The Breusch-Pagan Test
Don’t observe the error,but can estimate it with
the residuals from the OLS regression
After regressing the residuals squared on all of
the x’s,can use the R2 to form an F or LM test
The F statistic is just the reported F statistic for
overall significance of the regression,F =
[R2/k]/[(1 – R2)/(n – k – 1)],which is distributed Fk,
n – k - 1
The LM statistic is LM = nR2,which is distributed
c2k
Economics 20 - Prof,Anderson 12
The White Test
The Breusch-Pagan test will detect any
linear forms of heteroskedasticity
The White test allows for nonlinearities by
using squares and crossproducts of all the x’s
Still just using an F or LM to test whether
all the xj,xj2,and xjxh are jointly significant
This can get to be unwieldy pretty quickly
Economics 20 - Prof,Anderson 13
Alternate form of the White test
Consider that the fitted values from OLS,?,
are a function of all the x’s
Thus,?2 will be a function of the squares
and crossproducts and ? and ?2 can proxy
for all of the xj,xj2,and xjxh,so
Regress the residuals squared on ? and ?2
and use the R2 to form an F or LM statistic
Note only testing for 2 restrictions now
Economics 20 - Prof,Anderson 14
Weighted Least Squares
While it’s always possible to estimate
robust standard errors for OLS estimates,if
we know something about the specific form
of the heteroskedasticity,we can obtain
more efficient estimates than OLS
The basic idea is going to be to transform
the model into one that has homoskedastic
errors – called weighted least squares
Economics 20 - Prof,Anderson 15
Case of form being known up to
a multiplicative constant
Suppose the heteroskedasticity can be
modeled as Var(u|x) = ?2h(x),where the
trick is to figure out what h(x) ≡ hi looks like
E(ui/√hi|x) = 0,because hi is only a function
of x,and Var(ui/√hi|x) = ?2,because we
know Var(u|x) = ?2hi
So,if we divided our whole equation by √hi
we would have a model where the error is
homoskedastic
Economics 20 - Prof,Anderson 16
Generalized Least Squares
Estimating the transformed equation by
OLS is an example of generalized least
squares (GLS)
GLS will be BLUE in this case
GLS is a weighted least squares (WLS)
procedure where each squared residual is
weighted by the inverse of Var(ui|xi)
Economics 20 - Prof,Anderson 17
Weighted Least Squares
While it is intuitive to see why performing
OLS on a transformed equation is
appropriate,it can be tedious to do the
transformation
Weighted least squares is a way of getting
the same thing,without the transformation
Idea is to minimize the weighted sum of
squares (weighted by 1/hi)
Economics 20 - Prof,Anderson 18
More on WLS
WLS is great if we know what Var(ui|xi)
looks like
In most cases,won’t know form of
heteroskedasticity
Example where do is if data is aggregated,
but model is individual level
Want to weight each aggregate observation
by the inverse of the number of individuals
Economics 20 - Prof,Anderson 19
Feasible GLS
More typical is the case where you don’t
know the form of the heteroskedasticity
In this case,you need to estimate h(xi)
Typically,we start with the assumption of a
fairly flexible model,such as
Var(u|x) = ?2exp(d0 + d1x1 + …+ dkxk)
Since we don’t know the d,must estimate
Economics 20 - Prof,Anderson 20
Feasible GLS (continued)
Our assumption implies that u2 = ?2exp(d0
+ d1x1 + …+ dkxk)v
Where E(v|x) = 1,then if E(v) = 1
ln(u2) = a0 + d1x1 + …+ dkxk + e
Where E(e) = 1 and e is independent of x
Now,we know that ? is an estimate of u,so
we can estimate this by OLS
Economics 20 - Prof,Anderson 21
Feasible GLS (continued)
Now,an estimate of h is obtained as ? =
exp(?),and the inverse of this is our weight
So,what did we do?
Run the original OLS model,save the
residuals,?,square them and take the log
Regress ln(?2) on all of the independent
variables and get the fitted values,?
Do WLS using 1/exp(?) as the weight
Economics 20 - Prof,Anderson 22
WLS Wrapup
When doing F tests with WLS,form the weights
from the unrestricted model and use those weights to
do WLS on the restricted model as well as the
unrestricted model
Remember we are using WLS just for efficiency –
OLS is still unbiased & consistent
Estimates will still be different due to sampling error,
but if they are very different then it’s likely that some
other Gauss-Markov assumption is false
