16.322 Stochastic Estimation and Control Professor Vander Velde Lecture 2 Lecture 2 Last time: Given a set of events with are mutually exclusive and equally likely, () () nE PE N = . Example: card games Number of different 5-card poker hands 52 2,598,960 5 ?? == ?? ?? Number of different 13-card bridge hands 52 635,013,559,600 13 ?? == ?? ?? Poker probabilities: 1. 4444 13 12 11 10 2111 (one pair) 0.423 52 3! 5 P ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? == ?? ?? ?? 2. 444 13 12 11 221 (two pair) 0.0476 52 2! 5 P ?? ?? ?? ?? ?? ?? ?? ?? ?? == ?? ?? ?? Independence () ()()PAB PAPB= when A,B independent () ()(|)PAB PAPBA= when A,B dependent Definition of independence of events: 123 1 2 ( ... ) ( ) ( )... ( ) mm PAAA A PAPA PA= for all m (pairwise, threewise, etc.) Definition of conditional probability: () (|) () PAB PBA PA = (|)PBArestricts our attention to the situation where A has already occurred. If A,B are independent ()() (|) () () PAPB PBA PB PA == Two useful results: 16.322 Stochastic Estimation and Control Professor Vander Velde Lecture 2 1. ( ...) ( ) ( | ) ( | ) ( | )...P ABCD P A P B A P C AB P D ABC= Derive this by letting A=CD. Then ()()(|)()(|)(|)PBCD PCDPBCD PCPDCPDCD== 2. If 1 A , 2 A ,… is a set of mutually exclusive and collectively exhaustive events, then 12 1122 ( ) ( ) ( ) ... ( ) ( ) ( | ) ( ) ( | ) ... ( ) ( | ) nnn PE PEA PEA PEA PAPE A PA PE A PA PE A= + ++ = + ++ ()()()()PA B PA PB PAB+= + ? must subtract off P(AB) because it is counted twice by the first two terms of the RHS. ()()()()()()()()PA B C PA PB PC PAB PAC PBC PABC++ = + + ? ? ? + (for three events). If four events, ()PA B C D+++ would be the sum of probabilities of the single events, minus the probability of the joint events taken two at a time, plus the probability of the joint events taken three at a time, minus the probability of the joint event taken four at a time. Conclusion: If events are mutually exclusive, express result in terms of the sum of events; if independent, in terms of joint events. Note that this reduces to just the sum of the probabilities of the events if the events are mutually exclusive. Example: Simplex system Probability of component failure () i PF= 12 (system failure) (any component fails) ( ... ) n PP PFF==+ Assume component failures are independent. Take advantage of this by working with the complementary event. 16.322 Stochastic Estimation and Control Professor Vander Velde Lecture 2 12 (system failure) 1 (system works) 1 ( ... ) 1 ( ) 1 (1 ( )) nicii PPPWPWPF=? =? =?Π =?Π ? where ()1 () ii PW PF=? . Example: n-bit message e p - bit error probability A system may be able to tolerate up to k independent errors and still decode the message. Compute the probability of k errors. ee ... ( errors) (1 ) knk ee n Pk p p k ? ?? =? ?? ?? Conditional Independence Sometimes events depend on more than one random quantity, some of which may be common to two or more events, the others different and independent. In such cases the concept of conditional independence is useful. Example: Two measurements of the same quantity, x, each corrupted by additive independent noise. 11 22 11 2 2 (),() mxn mxn Em Em =+ =+ Since m 1 and m 2 both depend on the same quantity, x, any events which depend on m 1 and m 2 , E 1 (m 1 ) and E 2 (m 2 ), are clearly not independent. 12 1 2 ()()()PEE PE PE≠ in general i n are independent (|) (|)(|) ( | ) ( ) if A,B are independent ()()(|)(|)(|) (| ) (|) () ()(|) (|) PABE PAEPBE PBA PB PABE PEPABE PAEPBE PBEA PBE PEA PEPAE PAE = ? == = = 16.322 Stochastic Estimation and Control Professor Vander Velde Lecture 2 But if the value of x is given, the statistical properties of 1 m and 2 m depend only on 1 n and 2 n , which are independent. Thus the events 1 E and 2 E , conditioned on a given value of x , would be independent. Intuitively we would then say 12 1 2 (|)(|)(|)PEE x PE xPE x= In general, the statement (|) (|)(|)PABE PAEPBE= is the definition of the conditional independence of the events A and B , conditioned on the event E . If A and B are conditionally independent, given E , and B is conditioned on E , further conditioning on A should not change the probability of the occurrence of B since if E is given, A and B are conditionally independent. Bayes’ Theorem An interesting application of conditional probability. Let i A be mutually exclusive and collectively exhaustive. () () ()(|) (|) () ( ) ( )( | ) kk kk k iii ii PAE PAE PA PEA PA E PE PAE PAPE A == = ∑∑ Bayes’ rule This relation results directly from the definition of conditional probability and cannot be questioned. But over the years, the application of this relation to statistical inference has been subjected to a great deal of mathematical and philosophical criticism. Of late the notion has been more generally accepted, and serves as the basis for a well-defined theory of estimation and decision-making. The point of the controversy is the use of a probability function to express one’s state of imperfect knowledge about something which is itself not probabilistic at all. Example: Acceptance testing Components are manufactured by a process which is uncertain – some come out good and some bad. A test is devised such that a good component has 16.322 Stochastic Estimation and Control Professor Vander Velde Lecture 2 80% chance of passing and a bad component has 30% chance of passing. On the average, we think that the process produces 60% good components, 40% bad ones. If a component passes the test, what is the probability that it is a good one? Note that whether a certain component is good or bad is really not probabilistic – it is either good or bad. But we don’t know what it is. Our knowledge based on all prior experience, intuition, or whatever is expressed in the prior probabilities. 1 2 ("good") ( ) 0.6 ("bad") ( ) 0.4 (|)0.8 (|)0.3 ( ) ( | ) 0.6(0.8) 0.48 (|) 0.8 ( ) ( | ) ( ) ( | ) 0.6(0.8) 0.4(0.3) 0.60 PPGA B PPG PPB PGPPG PG P PGPPG PBPPB === === = = === ++ Suppose we have 1n? observations 11 ( | ... ) kn PA E E ? Take one more observation, n E 1 1 1 1 1 11 1 111 1 1 11 11 ( ... ) ( | ... ) ( ... ) ( ... ) ( ... ) ( ... ) ( | ... ) ( | ... ) ( ... ) ( | ... ) ( | ... ) ( | ... ) ( | ) ( | ... ) ( | ) kn kn n kn in i nk nn n ni n ni n i knnk inni i PAE E PA E E PE E PAE E PAE E PE E PA E E PE AE E PE E PA E E PE AE E PA E E PE A PA E E PE A ?? ?? ? ? = = = = ∑ ∑ ∑ This is of the same form as the relation first written down, with () 11 | ... kn PA E E ? in place of () k PA . This says that Bayes’ rule can be applied repetitively to account for any number of observations if the observations are conditionally independent – conditioned on the alternatives.