Economics 20 - Prof,Anderson 1
Multiple Regression Analysis
y = b0 + b1x1 + b2x2 +,,, bkxk + u
2,Inference
Economics 20 - Prof,Anderson 2
Assumptions of the Classical
Linear Model (CLM)
So far,we know that given the Gauss-
Markov assumptions,OLS is BLUE,
In order to do classical hypothesis testing,
we need to add another assumption (beyond
the Gauss-Markov assumptions)
Assume that u is independent of x1,x2,…,xk
and u is normally distributed with zero
mean and variance s2,u ~ Normal(0,s2)
Economics 20 - Prof,Anderson 3
CLM Assumptions (cont)
Under CLM,OLS is not only BLUE,but is
the minimum variance unbiased estimator
We can summarize the population
assumptions of CLM as follows
y|x ~ Normal(b0 + b1x1 +…+ bkxk,s2)
While for now we just assume normality,
clear that sometimes not the case
Large samples will let us drop normality
Economics 20 - Prof,Anderson 4
,
,
x1 x2
The homoskedastic normal distribution with
a single explanatory variable
E(y|x) = b0 + b1x
y
f(y|x)
Normal
distributions
Economics 20 - Prof,Anderson 5
Normal Sampling Distributions ? ?? ?
? ?
? ?
? ?
e r r o r s t h eofn c o m b i n a t i ol i n e a r a is
it b e c a u s en o r m a l l y dd i s t r i b u t e is
?
0,1N o r m a l ~
?
?
t h a tso,
?
,N o r m a l ~
?
st v a r i a b l ei n d e p e n d e n t h eof v a l u e ss a m p l e t h e
on lc o n d i t i o n a s,a s s u m p t i o n C L M U n d e r t h e
j
b
b
bb
bbb
j
jj
jjj
sd
V a r
?
Economics 20 - Prof,Anderson 6
The t Test ? ?
? ?
1:f r e e d o m of d e g r e e s t h eN o t e
?b y e s t i m a t e t oh a v e w eb e c a u s e
n o r m a l ) ( v so n d i s t r i b u t i a is t h i sN o t e
~
?
?
sa s s u m p t i o n C L M U n d e r t h e
22
1
j
??
?
??
kn
t
t
se
kn
j
j
ss
b
bb
Economics 20 - Prof,Anderson 7
The t Test (cont)
Knowing the sampling distribution for the
standardized estimator allows us to carry
out hypothesis tests
Start with a null hypothesis
For example,H0,bj=0
If accept null,then accept that xj has no
effect on y,controlling for other x’s
Economics 20 - Prof,Anderson 8
The t Test (cont)
? ?
0
?j
H,h y p o t h e s is n u ll a c c e p t t h e
o w h e t h e r td e t e r m in e t or u ler e je c t i o n a
w it ha lo n g s t a t is t ic o u r u s e t h e n w il lWe
?
?
,
?
f o r s t a t is t ic t h e ""
f o r m t on e e df i r s t eo u r t e s t w p e r f o r m To
t
se
tt
j
j
j
b
b
b
b
?
Economics 20 - Prof,Anderson 9
t Test,One-Sided Alternatives
Besides our null,H0,we need an alternative
hypothesis,H1,and a significance level
H1 may be one-sided,or two-sided
H1,bj > 0 and H1,bj < 0 are one-sided
H1,bj ? 0 is a two-sided alternative
If we want to have only a 5% probability of
rejecting H0 if it is really true,then we say
our significance level is 5%
Economics 20 - Prof,Anderson 10
One-Sided Alternatives (cont)
Having picked a significance level,a,we
look up the (1 – a)th percentile in a t
distribution with n – k – 1 df and call this c,
the critical value
We can reject the null hypothesis if the t
statistic is greater than the critical value
If the t statistic is less than the critical value
then we fail to reject the null
Economics 20 - Prof,Anderson 11
yi = b0 + b1xi1 + … + bkxik + ui
H0,bj = 0 H1,bj > 0
c 0
a ?1 ? a?
One-Sided Alternatives (cont)
Fail to reject
reject
Economics 20 - Prof,Anderson 12
One-sided vs Two-sided
Because the t distribution is symmetric,
testing H1,bj < 0 is straightforward,The
critical value is just the negative of before
We can reject the null if the t statistic < –c,
and if the t statistic > than –c then we fail to
reject the null
For a two-sided test,we set the critical
value based on a/2 and reject H1,bj ? 0 if
the absolute value of the t statistic > c
Economics 20 - Prof,Anderson 13
yi = b0 + b1Xi1 + … + bkXik + ui
H0,bj = 0 H1,bj > 0
c 0
a/2 ?1 ? a?
-c
a/2
Two-Sided Alternatives
reject reject
fail to reject
Economics 20 - Prof,Anderson 14
Summary for H0,bj = 0
Unless otherwise stated,the alternative is
assumed to be two-sided
If we reject the null,we typically say,xj is
statistically significant at the a % level”
If we fail to reject the null,we typically say
“xj is statistically insignificant at the a %
level”
Economics 20 - Prof,Anderson 15
Testing other hypotheses
A more general form of the t statistic
recognizes that we may want to test
something like H0,bj = aj
In this case,the appropriate t statistic is
? ?
? ?
t e s ts t a n d a r d f o r t h e 0
w h e r e,
?
?
?
?
?
j
j
jj
a
se
a
t
b
b
Economics 20 - Prof,Anderson 16
Confidence Intervals
Another way to use classical statistical testing is
to construct a confidence interval using the same
critical value as was used for a two-sided test
A (1 - a) % confidence interval is defined as
? ?
ond is t r ib u t i ain
p e r c e n t il e
2
-1 t h eis c w h e r e,??
1??
?
?
?
?
?
?
??
kn
jj
t
sec
a
bb
Economics 20 - Prof,Anderson 17
Computing p-values for t tests
An alternative to the classical approach is
to ask,“what is the smallest significance
level at which the null would be rejected?”
So,compute the t statistic,and then look up
what percentile it is in the appropriate t
distribution – this is the p-value
p-value is the probability we would observe
the t statistic we did,if the null were true
Economics 20 - Prof,Anderson 18
Stata and p-values,t tests,etc,
Most computer packages will compute the
p-value for you,assuming a two-sided test
If you really want a one-sided alternative,
just divide the two-sided p-value by 2
Stata provides the t statistic,p-value,and
95% confidence interval for H0,bj = 0 for
you,in columns labeled,t”,“P > |t|” and
“[95% Conf,Interval]”,respectively
Economics 20 - Prof,Anderson 19
Testing a Linear Combination
Suppose instead of testing whether b1 is equal to a
constant,you want to test if it is equal to another
parameter,that is H0, b1 = b2
Use same basic procedure for forming a t statistic
? ?
21
21
??
??
bb
bb
?
?
?
se
t
Economics 20 - Prof,Anderson 20
Testing Linear Combo (cont)
? ? ? ?
? ? ? ? ? ? ? ?
? ? ? ?? ? ? ?? ?? ?
? ?
2112
2
1
12
2
2
2
121
212121
2121
?
,
?
of e s t im a t ea n is w h e r e
2
????
?
,
?
2
????
t h e n,
????
S in c e
bb
bbbb
bbbbbb
bbbb
C o vs
ssesese
C o vV a rV a rV a r
V a rse
????
????
???
Economics 20 - Prof,Anderson 21
Testing a Linear Combo (cont)
So,to use formula,need s12,which
standard output does not have
Many packages will have an option to get it,
or will just perform the test for you
In Stata,after reg y x1 x2 … xk you would
type test x1 = x2 to get a p-value for the test
More generally,you can always restate the
problem to get the test you want
Economics 20 - Prof,Anderson 22
Example,
Suppose you are interested in the effect of
campaign expenditures on outcomes
Model is voteA = b0 + b1log(expendA) +
b2log(expendB) + b3prtystrA + u
H0,b1 = - b2,or H0,q1 = b1 + b2 = 0
b1 = q1 – b2,so substitute in and rearrange
? voteA = b0 + q1log(expendA) +
b2log(expendB - expendA) + b3prtystrA + u
Economics 20 - Prof,Anderson 23
Example (cont),
This is the same model as originally,but
now you get a standard error for b1 – b2 = q1
directly from the basic regression
Any linear combination of parameters
could be tested in a similar manner
Other examples of hypotheses about a
single linear combination of parameters,
? b1 = 1 + b2 ; b1 = 5b2 ; b1 = -1/2b2 ; etc
Economics 20 - Prof,Anderson 24
Multiple Linear Restrictions
Everything we’ve done so far has involved
testing a single linear restriction,(e.g,b1 = 0
or b1 = b2 )
However,we may want to jointly test
multiple hypotheses about our parameters
A typical example is testing,exclusion
restrictions” – we want to know if a group
of parameters are all equal to zero
Economics 20 - Prof,Anderson 25
Testing Exclusion Restrictions
Now the null hypothesis might be
something like H0,bk-q+1 = 0,...,bk = 0
The alternative is just H1,H0 is not true
Can’t just check each t statistic separately,
because we want to know if the q
parameters are jointly significant at a given
level – it is possible for none to be
individually significant at that level
Economics 20 - Prof,Anderson 26
Exclusion Restrictions (cont)
To do the test we need to estimate the,restricted
model” without xk-q+1,,…,xk included,as well as
the,unrestricted model” with all x’s included
Intuitively,we want to know if the change in SSR
is big enough to warrant inclusion of xk-q+1,,…,xk
? ?
? ?
edu n r e s t r ic t isu r a n d r e s t r ic t e d isr
w h e r e,
1??
?
?
knSSR
qSSRSSR
F
ur
urr
Economics 20 - Prof,Anderson 27
The F statistic
The F statistic is always positive,since the
SSR from the restricted model can’t be less
than the SSR from the unrestricted
Essentially the F statistic is measuring the
relative increase in SSR when moving from
the unrestricted to restricted model
q = number of restrictions,or dfr – dfur
n – k – 1 = dfur
Economics 20 - Prof,Anderson 28
The F statistic (cont)
To decide if the increase in SSR when we
move to a restricted model is,big enough”
to reject the exclusions,we need to know
about the sampling distribution of our F stat
Not surprisingly,F ~ Fq,n-k-1,where q is
referred to as the numerator degrees of
freedom and n – k – 1 as the denominator
degrees of freedom
Economics 20 - Prof,Anderson 29
0 c
a ?1 ? a?
f(F)
F
The F statistic (cont)
reject
fail to reject
Reject H0 at a
significance level
if F > c
Economics 20 - Prof,Anderson 30
The R2 form of the F statistic
Because the SSR’s may be large and unwieldy,an
alternative form of the formula is useful
We use the fact that SSR = SST(1 – R2) for any
regression,so can substitute in for SSRu and SSRur
? ?
? ? ? ?
edu n r e s t r ic t isu r a n d r e s t r ic t e d isr
a g a in w h e r e,
11
2
22
???
?
?
knR
qRR
F
ur
rur
Economics 20 - Prof,Anderson 31
Overall Significance
A special case of exclusion restrictions is to test
H0,b1 = b2 =…= bk = 0
Since the R2 from a model with only an intercept
will be zero,the F statistic is simply
? ? ? ?11 2
2
???
?
knR
kR
F
Economics 20 - Prof,Anderson 32
General Linear Restrictions
The basic form of the F statistic will work
for any set of linear restrictions
First estimate the unrestricted model and
then estimate the restricted model
In each case,make note of the SSR
Imposing the restrictions can be tricky –
will likely have to redefine variables again
Economics 20 - Prof,Anderson 33
Example,
Use same voting model as before
Model is voteA = b0 + b1log(expendA) +
b2log(expendB) + b3prtystrA + u
now null is H0,b1 = 1,b3 = 0
Substituting in the restrictions,voteA = b0
+ log(expendA) + b2log(expendB) + u,so
Use voteA - log(expendA) = b0 +
b2log(expendB) + u as restricted model
Economics 20 - Prof,Anderson 34
F Statistic Summary
Just as with t statistics,p-values can be
calculated by looking up the percentile in
the appropriate F distribution
Stata will do this by entering,display
fprob(q,n – k – 1,F),where the appropriate
values of F,q,and n – k – 1 are used
If only one exclusion is being tested,then F
= t2,and the p-values will be the same
Multiple Regression Analysis
y = b0 + b1x1 + b2x2 +,,, bkxk + u
2,Inference
Economics 20 - Prof,Anderson 2
Assumptions of the Classical
Linear Model (CLM)
So far,we know that given the Gauss-
Markov assumptions,OLS is BLUE,
In order to do classical hypothesis testing,
we need to add another assumption (beyond
the Gauss-Markov assumptions)
Assume that u is independent of x1,x2,…,xk
and u is normally distributed with zero
mean and variance s2,u ~ Normal(0,s2)
Economics 20 - Prof,Anderson 3
CLM Assumptions (cont)
Under CLM,OLS is not only BLUE,but is
the minimum variance unbiased estimator
We can summarize the population
assumptions of CLM as follows
y|x ~ Normal(b0 + b1x1 +…+ bkxk,s2)
While for now we just assume normality,
clear that sometimes not the case
Large samples will let us drop normality
Economics 20 - Prof,Anderson 4
,
,
x1 x2
The homoskedastic normal distribution with
a single explanatory variable
E(y|x) = b0 + b1x
y
f(y|x)
Normal
distributions
Economics 20 - Prof,Anderson 5
Normal Sampling Distributions ? ?? ?
? ?
? ?
? ?
e r r o r s t h eofn c o m b i n a t i ol i n e a r a is
it b e c a u s en o r m a l l y dd i s t r i b u t e is
?
0,1N o r m a l ~
?
?
t h a tso,
?
,N o r m a l ~
?
st v a r i a b l ei n d e p e n d e n t h eof v a l u e ss a m p l e t h e
on lc o n d i t i o n a s,a s s u m p t i o n C L M U n d e r t h e
j
b
b
bb
bbb
j
jj
jjj
sd
V a r
?
Economics 20 - Prof,Anderson 6
The t Test ? ?
? ?
1:f r e e d o m of d e g r e e s t h eN o t e
?b y e s t i m a t e t oh a v e w eb e c a u s e
n o r m a l ) ( v so n d i s t r i b u t i a is t h i sN o t e
~
?
?
sa s s u m p t i o n C L M U n d e r t h e
22
1
j
??
?
??
kn
t
t
se
kn
j
j
ss
b
bb
Economics 20 - Prof,Anderson 7
The t Test (cont)
Knowing the sampling distribution for the
standardized estimator allows us to carry
out hypothesis tests
Start with a null hypothesis
For example,H0,bj=0
If accept null,then accept that xj has no
effect on y,controlling for other x’s
Economics 20 - Prof,Anderson 8
The t Test (cont)
? ?
0
?j
H,h y p o t h e s is n u ll a c c e p t t h e
o w h e t h e r td e t e r m in e t or u ler e je c t i o n a
w it ha lo n g s t a t is t ic o u r u s e t h e n w il lWe
?
?
,
?
f o r s t a t is t ic t h e ""
f o r m t on e e df i r s t eo u r t e s t w p e r f o r m To
t
se
tt
j
j
j
b
b
b
b
?
Economics 20 - Prof,Anderson 9
t Test,One-Sided Alternatives
Besides our null,H0,we need an alternative
hypothesis,H1,and a significance level
H1 may be one-sided,or two-sided
H1,bj > 0 and H1,bj < 0 are one-sided
H1,bj ? 0 is a two-sided alternative
If we want to have only a 5% probability of
rejecting H0 if it is really true,then we say
our significance level is 5%
Economics 20 - Prof,Anderson 10
One-Sided Alternatives (cont)
Having picked a significance level,a,we
look up the (1 – a)th percentile in a t
distribution with n – k – 1 df and call this c,
the critical value
We can reject the null hypothesis if the t
statistic is greater than the critical value
If the t statistic is less than the critical value
then we fail to reject the null
Economics 20 - Prof,Anderson 11
yi = b0 + b1xi1 + … + bkxik + ui
H0,bj = 0 H1,bj > 0
c 0
a ?1 ? a?
One-Sided Alternatives (cont)
Fail to reject
reject
Economics 20 - Prof,Anderson 12
One-sided vs Two-sided
Because the t distribution is symmetric,
testing H1,bj < 0 is straightforward,The
critical value is just the negative of before
We can reject the null if the t statistic < –c,
and if the t statistic > than –c then we fail to
reject the null
For a two-sided test,we set the critical
value based on a/2 and reject H1,bj ? 0 if
the absolute value of the t statistic > c
Economics 20 - Prof,Anderson 13
yi = b0 + b1Xi1 + … + bkXik + ui
H0,bj = 0 H1,bj > 0
c 0
a/2 ?1 ? a?
-c
a/2
Two-Sided Alternatives
reject reject
fail to reject
Economics 20 - Prof,Anderson 14
Summary for H0,bj = 0
Unless otherwise stated,the alternative is
assumed to be two-sided
If we reject the null,we typically say,xj is
statistically significant at the a % level”
If we fail to reject the null,we typically say
“xj is statistically insignificant at the a %
level”
Economics 20 - Prof,Anderson 15
Testing other hypotheses
A more general form of the t statistic
recognizes that we may want to test
something like H0,bj = aj
In this case,the appropriate t statistic is
? ?
? ?
t e s ts t a n d a r d f o r t h e 0
w h e r e,
?
?
?
?
?
j
j
jj
a
se
a
t
b
b
Economics 20 - Prof,Anderson 16
Confidence Intervals
Another way to use classical statistical testing is
to construct a confidence interval using the same
critical value as was used for a two-sided test
A (1 - a) % confidence interval is defined as
? ?
ond is t r ib u t i ain
p e r c e n t il e
2
-1 t h eis c w h e r e,??
1??
?
?
?
?
?
?
??
kn
jj
t
sec
a
bb
Economics 20 - Prof,Anderson 17
Computing p-values for t tests
An alternative to the classical approach is
to ask,“what is the smallest significance
level at which the null would be rejected?”
So,compute the t statistic,and then look up
what percentile it is in the appropriate t
distribution – this is the p-value
p-value is the probability we would observe
the t statistic we did,if the null were true
Economics 20 - Prof,Anderson 18
Stata and p-values,t tests,etc,
Most computer packages will compute the
p-value for you,assuming a two-sided test
If you really want a one-sided alternative,
just divide the two-sided p-value by 2
Stata provides the t statistic,p-value,and
95% confidence interval for H0,bj = 0 for
you,in columns labeled,t”,“P > |t|” and
“[95% Conf,Interval]”,respectively
Economics 20 - Prof,Anderson 19
Testing a Linear Combination
Suppose instead of testing whether b1 is equal to a
constant,you want to test if it is equal to another
parameter,that is H0, b1 = b2
Use same basic procedure for forming a t statistic
? ?
21
21
??
??
bb
bb
?
?
?
se
t
Economics 20 - Prof,Anderson 20
Testing Linear Combo (cont)
? ? ? ?
? ? ? ? ? ? ? ?
? ? ? ?? ? ? ?? ?? ?
? ?
2112
2
1
12
2
2
2
121
212121
2121
?
,
?
of e s t im a t ea n is w h e r e
2
????
?
,
?
2
????
t h e n,
????
S in c e
bb
bbbb
bbbbbb
bbbb
C o vs
ssesese
C o vV a rV a rV a r
V a rse
????
????
???
Economics 20 - Prof,Anderson 21
Testing a Linear Combo (cont)
So,to use formula,need s12,which
standard output does not have
Many packages will have an option to get it,
or will just perform the test for you
In Stata,after reg y x1 x2 … xk you would
type test x1 = x2 to get a p-value for the test
More generally,you can always restate the
problem to get the test you want
Economics 20 - Prof,Anderson 22
Example,
Suppose you are interested in the effect of
campaign expenditures on outcomes
Model is voteA = b0 + b1log(expendA) +
b2log(expendB) + b3prtystrA + u
H0,b1 = - b2,or H0,q1 = b1 + b2 = 0
b1 = q1 – b2,so substitute in and rearrange
? voteA = b0 + q1log(expendA) +
b2log(expendB - expendA) + b3prtystrA + u
Economics 20 - Prof,Anderson 23
Example (cont),
This is the same model as originally,but
now you get a standard error for b1 – b2 = q1
directly from the basic regression
Any linear combination of parameters
could be tested in a similar manner
Other examples of hypotheses about a
single linear combination of parameters,
? b1 = 1 + b2 ; b1 = 5b2 ; b1 = -1/2b2 ; etc
Economics 20 - Prof,Anderson 24
Multiple Linear Restrictions
Everything we’ve done so far has involved
testing a single linear restriction,(e.g,b1 = 0
or b1 = b2 )
However,we may want to jointly test
multiple hypotheses about our parameters
A typical example is testing,exclusion
restrictions” – we want to know if a group
of parameters are all equal to zero
Economics 20 - Prof,Anderson 25
Testing Exclusion Restrictions
Now the null hypothesis might be
something like H0,bk-q+1 = 0,...,bk = 0
The alternative is just H1,H0 is not true
Can’t just check each t statistic separately,
because we want to know if the q
parameters are jointly significant at a given
level – it is possible for none to be
individually significant at that level
Economics 20 - Prof,Anderson 26
Exclusion Restrictions (cont)
To do the test we need to estimate the,restricted
model” without xk-q+1,,…,xk included,as well as
the,unrestricted model” with all x’s included
Intuitively,we want to know if the change in SSR
is big enough to warrant inclusion of xk-q+1,,…,xk
? ?
? ?
edu n r e s t r ic t isu r a n d r e s t r ic t e d isr
w h e r e,
1??
?
?
knSSR
qSSRSSR
F
ur
urr
Economics 20 - Prof,Anderson 27
The F statistic
The F statistic is always positive,since the
SSR from the restricted model can’t be less
than the SSR from the unrestricted
Essentially the F statistic is measuring the
relative increase in SSR when moving from
the unrestricted to restricted model
q = number of restrictions,or dfr – dfur
n – k – 1 = dfur
Economics 20 - Prof,Anderson 28
The F statistic (cont)
To decide if the increase in SSR when we
move to a restricted model is,big enough”
to reject the exclusions,we need to know
about the sampling distribution of our F stat
Not surprisingly,F ~ Fq,n-k-1,where q is
referred to as the numerator degrees of
freedom and n – k – 1 as the denominator
degrees of freedom
Economics 20 - Prof,Anderson 29
0 c
a ?1 ? a?
f(F)
F
The F statistic (cont)
reject
fail to reject
Reject H0 at a
significance level
if F > c
Economics 20 - Prof,Anderson 30
The R2 form of the F statistic
Because the SSR’s may be large and unwieldy,an
alternative form of the formula is useful
We use the fact that SSR = SST(1 – R2) for any
regression,so can substitute in for SSRu and SSRur
? ?
? ? ? ?
edu n r e s t r ic t isu r a n d r e s t r ic t e d isr
a g a in w h e r e,
11
2
22
???
?
?
knR
qRR
F
ur
rur
Economics 20 - Prof,Anderson 31
Overall Significance
A special case of exclusion restrictions is to test
H0,b1 = b2 =…= bk = 0
Since the R2 from a model with only an intercept
will be zero,the F statistic is simply
? ? ? ?11 2
2
???
?
knR
kR
F
Economics 20 - Prof,Anderson 32
General Linear Restrictions
The basic form of the F statistic will work
for any set of linear restrictions
First estimate the unrestricted model and
then estimate the restricted model
In each case,make note of the SSR
Imposing the restrictions can be tricky –
will likely have to redefine variables again
Economics 20 - Prof,Anderson 33
Example,
Use same voting model as before
Model is voteA = b0 + b1log(expendA) +
b2log(expendB) + b3prtystrA + u
now null is H0,b1 = 1,b3 = 0
Substituting in the restrictions,voteA = b0
+ log(expendA) + b2log(expendB) + u,so
Use voteA - log(expendA) = b0 +
b2log(expendB) + u as restricted model
Economics 20 - Prof,Anderson 34
F Statistic Summary
Just as with t statistics,p-values can be
calculated by looking up the percentile in
the appropriate F distribution
Stata will do this by entering,display
fprob(q,n – k – 1,F),where the appropriate
values of F,q,and n – k – 1 are used
If only one exclusion is being tested,then F
= t2,and the p-values will be the same
