LAW OF ITERATED EXPECTATIONS 1
Law of Iterated Expectations
Theorem 1 Law of Iterated Expectations
E [y] = Ex [E [y|x]]
The notation Ex [·] indicates the expectation over the value of x.
Example 1
y = bx + ε
ε ～ N (0,1)
x ～ U [0,1]
ε and x are independent.
E [y|x] = E [bx + ε|x]
= E [bx|x] + E [ε|x]
Since ε and x are independent, E [ε|x] = E [ε] = 0,
E [y|x] = bx
Now, using the law of iterated expectations and
E [x] = 12
E [y] = Ex [E [y|x]]
= Ex [bx]
= bEx [x]
= 12b
Matrix Algebra (Continue)
Definition 1 Idempotent Matrix
An idempotent matrix is one that is equal to its square, that is, M2= MM = M.
Example 2 The identity matrix I
I2 = I·I = I
Definition 2 Linear Dependence
A set of vectors is linearly dependent if any one of the vectors in the set can be written as a linear
combination of the others.
MATRIX ALGEBRA (CONTINUE) 2
Example 3
a =
bracketleftbigg 1
2
bracketrightbigg
, b =
bracketleftbigg 3
3
bracketrightbigg
, c =
bracketleftbigg 10
14
bracketrightbigg
2a+b?12c = 0
Example 4 Geometric Interpretation
fill45
fill54
y
xfill0
fill0
fill0
fill0
fill0
fill0
fill0
fill0fill0fill18
v1
fill1
fill1
fill1
fill1
fill1
fill1
fill1
fill1
fill1
fill1
fill1fill1fill21
v2
v1 and v2 are independent
fill45
fill54
y
xfill0
fill0
fill0
fill0
fill0
fill0
fill0
fill0fill0fill18
v1
fill0
fill0
fill0
fill0
fill0
fill0
fill0
fill0fill0fill9
v2
v1 and v2 are not independent
Definition 3 Column Space
The column space of a matrix is the vector space that is spanned by its column vectors.
Example 5
A =
bracketleftbigg 1 3
2 4
bracketrightbigg
spanned the R2 space.
Definition 4 Column Rank
The column rank of a matrix is the dimension of the vector space that is spanned by its columns.
Example 6
B =
?
??
?
1 2 3 2
5 1 5 7
6 4 5 7
3 1 4 1
?
??
?
MATRIX ALGEBRA (CONTINUE) 3
spanned the R4 space
C =
?
?
1 5 6 3
2 1 4 1
3 5 5 4
?
?
spanned the R3 space
Theorem 2 Equality of Row and Column Rank
The column rank and row rank of a matrix are equal. By the definition of row rank and its counterpart
for column rank, the row space and column space of a matrix have the same dimension.
Theorem 3
rank(AB) ≤ min(rank(A),rank (B))
Theorem 4 For any matrix A and nonsingular matrices B and C, the rank of BAC is equal to the
rank of A.
(The meaning of nonsingular matrices will be introduced later).
Theorem 5
rank (A) = rank(AprimeA)
Definition 5 Determinant of a matrix
For a n×n matrix (square matrix), the area of the matrix is the determinant.
Example 7
A =
bracketleftbigg 4 2
1 3
bracketrightbigg
detA =|A| = 4×3?2×1
= 10
Proposition 1 The determinant of a matrix is nonzero if and only if it has full rank.
rank (A) = dim(A)
Definition 6 Inverse of a matrix
Suppose that we could find a square matrix B such that BA = I, B is the inverse of A, denoted
B = A?1
Example 8
A =
bracketleftbigg 4 2
1 3
bracketrightbigg
B = A?1 =
bracketleftbigg 3
10 ?
1
5? 1
10
2
5
bracketrightbigg
Definition 7 Nonsingular Matrix
A matrix whose inverse exists is nonsingular.
MATRIX ALGEBRA (CONTINUE) 4
Partitioned Matrices
A =
?
?
1 4 5
2 9 3
8 9 6
?
?
=
bracketleftbigg A
11 A12
A21 A22
bracketrightbigg
Block diagonal matrix
A =
bracketleftbigg A
11 0
0 A22
bracketrightbigg
Addition and Multiplication Matrices
A =
bracketleftbigg A
11 A12
A21 A22
bracketrightbigg
, B =
bracketleftbigg B
11 B12
B21 B22
bracketrightbigg
A+B =
bracketleftbigg A
11 +B11 A12 +B12
A21 +B21 A22 +B22
bracketrightbigg
AB =
bracketleftbigg A
11B11 +A12B21 A11B12 +A12B22
A21B11 +A22B21 A21B12 +A22B22
bracketrightbigg
Determinants of Partitioned Matrices
In general for a 2 × 2 partitioned matrix
vextendsinglevextendsingle
vextendsinglevextendsingle A11 A12A
21 A22
vextendsinglevextendsingle
vextendsinglevextendsingle = |A11| · vextendsinglevextendsingleA22 ?A21A?111 A12vextendsinglevextendsingle
= |A22| · vextendsinglevextendsingleA11 ?A12A?122 A21vextendsinglevextendsingle
For a block diagonal matrix vextendsinglevextendsingle
vextendsinglevextendsingle A 00 B
vextendsinglevextendsingle
vextendsinglevextendsingle = |A| ·|B|
Inverses of Partitioned Matrices
In general for a 2 × 2 partitioned matrix
bracketleftbigg A
11 A12
A21 A22
bracketrightbigg?1
=
bracketleftbigg A?1
11
parenleftbigI+A
12F2A21A?111
parenrightbig ?A?1
11 A12F2
?F2A21A?111 F2
bracketrightbigg
where
F2 = parenleftbigA22 ?A21A?111 A12parenrightbig?1
For a block diagnoal matrix bracketleftbigg
A11 0
0 A22
bracketrightbigg?1
=
bracketleftbigg A?1
11 0
0 A?122
bracketrightbigg
MATRIX ALGEBRA (CONTINUE) 5
Kronecker Products
Definition 8 For two matrices A and B, their Kronecker product
A?B =
?
??
??
a11B a12B ··· a1KB
a21B a22B ··· a2KB
... ... ...
an1B an2B ··· anKB
?
??
??
Note: For any matrix AK×L, Bm×n, their Kronecter product
A?B
has dimension (Km)×(Ln). No conformability is required.
Example 9
A =
bracketleftbigg 1 2
3 4
bracketrightbigg
, B =
bracketleftbigg 5 7
6 8
bracketrightbigg
A?B =
?
??
?
1
bracketleftbigg 5 7
6 8
bracketrightbigg
2
bracketleftbigg 5 7
6 8
bracketrightbigg
3
bracketleftbigg 5 7
6 8
bracketrightbigg
4
bracketleftbigg 5 7
6 8
bracketrightbigg
?
??
?
Trace of a Matrix
Definition 9 The trace of a square K ×K matrix is the sum of its diagonal elements:
tr(A) =
Ksummationdisplay
i=1
aii.
Example 10
A =
?
?
1 0 0
0 1 0
0 0 1
?
?, tr(A) = 3
B =
?
?
1 2 3
2 1 2
3 2 1
?
?, tr(B) = 3
Some identities
tr(cA) = c·tr(A)
tr(Aprime) = tr(A)
tr(A+B) = tr(A) + tr(B)
tr(Ik) = K
tr(AB) = tr(BA)
tr(ABCD) = tr(BCDA) = tr(CDAB) = tr(DABC)
MATRIX ALGEBRA (CONTINUE) 6
The Generalized Inverse of a Matrix
Definition 10 A generalized inverse of a matrix A is another matrix A+ that satisfies
1. AA+A = A
2. A+AA+ = A+
3. A+A is symmetric
4. AA+ is symmetric
Quadratic Forms and Definite Matrices
Fortheoptimizationproblem
q =
nsummationdisplay
i=1
nsummationdisplay
j=1
xixjaij
Thequadraticformcanbewrittenas
q = xprimeAx
ForagivenmatrixA,
1. IfxprimeAx > (<)0 forallnonzerox, thenA ispositive(negative)definite.
2. IfxprimeAx ≥ (≤)0 forallnonzero x, thenA isnonnegativedefiniteorpositivesemi—definite.
Someusefulresults:
? IfA isnonnegativedefinite,then|A| ≥ 0.
? IfA ispositivedefinite,sois A?1.
? TheidentitymatrixI ispositivedefinite.
? If A is n × K withfullrankand n > K, then AprimeA ispositivedefiniteand AAprime isnonnegative
definite.
? IfA ispositivedefiniteandB isanonsingularmatrix,thenBprimeAB ispositivedefinite.
? If A is symmetric and idempotent, n × n with rank J, then every duadratic form in A can be
writtenxprimeAx =summationtextJi=1y2i .
