1 ERROR ANALYSIS (UNCERTAINTY ANALYSIS) 16.621 Experimental Projects Lab I 2 TOPICS TO BE COVERED ? Why do error analysis? ? If we don’t ever know the true value, how do we estimate the error in the true value? ? Error propagation in the measurement chain – How do errors combine? (How do they behave in general?) – How do we do an end-to-end uncertainty analysis? – What are ways to mitigate errors? ? A hypothetical dilemma (probably nothing to do with anyone in the class) – When should I throw out some data that I don’t like? – Answer: NEVER, but there are reasons to throw out data ? Backup slides: an example of an immense amount of money and effort directed at error analysis and mitigation - jet engine testing 3 ERROR AND UNCERTAINTY ? In engineering the word “error”, when used to describe an aspect of measurement does not necessarily carry the connotation of mistake or blunder (although it can!) ? Error in a measurement means the inevitable uncertainty that attends all measurements ? We cannot avoid errors in this sense ? We can ensure that they are as small as reasonably possible and that we have a reliable estimate of how small they are [Adapted from Taylor, J. R, An Introduction to Error Analysis; The Study of Uncertainties in Physical Measurements] 4 USES OF UNCERTAINTY ANALYSIS (I) ? Assess experimental procedure including identification of potential difficulties – Definition of necessary steps – Gaps ? Advise what procedures need to be put in place for measurement ? Identify instruments and procedures that control accuracy and precision – Usually one, or at most a small number, out of the large set of possibilities ? Inform us when experiment cannot meet desired accuracy 5 USES OF UNCERTAINTY ANALYSIS (II) ? Provide the only known basis for deciding whether: – Data agrees with theory – Tests from different facilities (jet engine performance) agree – Hypothesis has been appropriately assessed (resolved) – Phenomena measured are real ? Provide basis for defining whether a closure check has been achieved – Is continuity satisfied (does the same amount of mass go in as goes out?) – Is energy conserved? ? Provide an integrated grasp of how to conduct the experiment [Adapted from Kline, S. J., 1985, “The Purposes of Uncertainty Analysis”, ASME J. Fluids Engineering, pp. 153-160] 6 UNCERTAINTY ESTIMATES AND HYPOTHESIS ASSESSMENT 0 100 200 300 400 500 600 0 20 40 60 80 100 120 Ma ss [g] Di st an ce [ c m ] 0 100 200 300 400 500 600 0 20 40 60 80 100 120 Mass [g] Di s t anc e [ c m ] 0 100 200 300 400 500 600 0 20 40 60 80 100 120 Mass [g] Di s t anc e [ c m ] 7 HOW DO WE DEAL WITH NOT KNOWING THE TRUE VALUE? ? In “all” real situations we don’t know the true value we are looking for ? We need to decide how to determine the best representation of this from our measurements ? We need to decide what the uncertainty is in our best representation 8 AN IMPLICATION OF NOT KNOWING THE TRUE VALUE ? We easily divided errors into precision (bias) errors and random errors when we knew what the value was ? The target practice picture in the next slide is an example ? How about if we don’t know the true value? Can we, by looking at the data in the slide after this, say that there are bias errors? ? How do we know if bias errors exist or not? 11 A TEAM EXERCISE ? List the variables you need to determine in order to carry out your hypothesis assessment ? What uncertainties do you foresee? (Qualitative description) ? Are you more concerned about bias errors or random errors? ? What level of uncertainty in the final result do you need to assess your hypothesis in a rigorous manner? ? Can you make an estimate of the level of the uncertainty in the final result? – If so, what is it? – If not, what additional information do you need to do this? 12 HOW DO WE COMBINE ERRORS? ? Suppose we measure quantity X with an error of dx and quantity Y with an error of dy ? What is the error in quantity Z if: ? Z = AX where A is a numerical constant such as π? ? Z = X + Y? ? Z = X - Y? ? Z = XY? ? Z = X/Y? ? Z is a general function of many quantities? 13 ERRORS IN THE FINAL QUANTITY ? Z = X + Y ? Linear combination – – Error in Z is BUT this is worst case ? For random errors we could have – or – These errors are much smaller ? In general if different errors are not correlated, are independent, the way to combine them is ? This is true for random and bias errors Z+ dz = X + dx +Y + dy dz = dx + dy dz = dx ? dy dy ? dx dz = dx 2 +dy 2 14 THE CASE OF Z = X - Y ? Suppose Z = X - Y is a number much smaller than X or Y ? Say (say 2%) ? may be much larger than ? MESSAGE ==> Avoid taking the difference of two numbers of comparable size dx X = dy Y =ε dz = dx 2 +dy 2 dz Z = 2 dx X ?Y dx X 15 ESTIMATES FOR THE TRUE VALUE AND THE ERROR ? Is there a “best” estimate of the true value of a quantity? ? How do I find it? ? How do I estimate the random error? ? How do I estimate the bias error? 16 SOME “RULES” FOR ESTIMATING RANDOM ERRORS AND TRUE VALUE ? An internal estimate can be given by repeat measurements ? Random error is generally of same size as standard deviation (root mean square deviation) of measurements ? Mean of repeat measurements is best estimate of true value ? Standard deviation of the mean (random error) is smaller than standard deviation of a single measurement by ? To increase precision by 10, you need 100 measurements 1 Number of measurements 17 GENERAL RULE FOR COMBINATION OF ERRORS ? If Z = F (X 1 , X 2 , X 3 , X 4 ) is quantity we want ? The error in Z, dz, is given by our rule from before ? So, if the error F due to X 1 can be estimated as and so on ? ? The important consequence of this is that generally one or few of these factors is the main player and others can be ignored dF 1 = ?F ?X 1 dx 1 Influence coeff. Error in X 1 dz = ?F ?X 1 ? ? ? ? ? ? 2 dx 1 2 + ?F ?X 2 ? ? ? ? ? ? 2 dx 2 2 +null ?F ?X n ? ? ? ? ? ? 2 dx n 2 18 DISTRIBUTION OF RANDOM ERRORS ? A measurement subject to many small random errors will be distributed “normally” ? Normal distribution is a Gaussian ? If x is a given measurement and X is the true value ? σ is the standard deviation Gaussian or normal distribution= 1 σ 2π e ? x?X 2 ( ) 2σ 2 20 A REVELATION ? The universal gas constant is accepted R = 8.31451 ±0.00007 J/mol K ? This is not a true value but can be “accepted” as one 21 ONE ADDITIONAL ASPECT OF COMBINING ERRORS ? We have identified two different types of errors, bias (systematic) and random – Random errors can be assessed by repetition of measurements – Bias errors cannot; these need to be estimated using external information (mfrs. specs., your knowledge) ? How should the two types of errors be combined? – One practice is to treat each separately using our rule, and then report the two separately at the end – One other practice is to combine them as “errors” ? Either seems acceptable, as long as you show that you are going to deal (have dealt) with both 22 REPORTING OF MEASUREMENTS ? Experimental uncertainties should almost always be rounded to one significant figure ? The last significant figure in any stated answer should usually be of the same order of magnitude (in the same decimal position) as the uncertainty [from Taylor, J., An Introduction to Error Analysis] 23 COMMENTS ON REJECTION OF DATA ? Should you reject (delete) data? ? Sometimes on measurement appears to disagree greatly with all others. How do we decide: – Is this significant? – Is this a mistake? ? One criteria (Chauvenaut’s criteria) is as follows – Suppose that errors are normally distributed – If measurement is more than M standard deviations (say 3), probability is < 0.003 that measurement should occur – Is this improbable enough to throw out measurement? ? The decision of “ridiculous improbability” [Taylor, 1997] is up to the investigator, but it allows the reader to understand the basis for the decision – If beyond this range, delete the data 26 A CAVEAT ON REJECTION OF DATA ? If more than one measurement is different, it may be that something is really happening that has not been envisioned, e.g., discovery of radon ? You may not be controlling all the variables that you need to ? Bottom line: Rigorous uncertainty analysis can give rationale to decide what data to pay attention to 27 SUMMARY ? Both the number and the fidelity of the number are important in a measurement ? We considered two types of uncertainties, bias (or systematic errors) and random errors ? Uncertainty analysis addresses fidelity and is used in different phases of an experiment, from initial planning to final reporting – Attention is needed to ensure uncertainties do not invalidate your efforts ? In propagating uncorrelated errors from individual measurement to final result, use the square root of the sums of the squares of the errors – There are generally only a few main contributors (sometimes one) to the overall uncertainty which need to be addressed ? Uncertainty analysis is a critical part of “real world” engineering projects 28 SOME REFERENCES I HAVE FOUND USEFUL ? Baird, D. C., 1962, Experimentation: An Introduction to Measurement Theory and Experiment, Prentice-Hall, Englewood Cliffs, NJ ? Bevington, P. R, and Robinson, D. K., 1992, Data Reduction and Error Analysis for the Physical Sciences, McGraw-Hill, New York, NY ? Lyons, L., 1991, A Practical Guide to Data Analysis for Physical Science Students, Cambridge University Press, Cambridge, UK ? Rabinowicz, E, 1970, An Introduction to Experimentation, Addison- Wesley, Reading, MA ? Taylor, J. R., 1997, An Introduction to Error Analysis, University Science Books, Sauselito, CA 29 BACKUP EXAMPLE: MEASUREMENT OF JET ENGINE PEFORMANCE ? We want to measure Thrust, Airflow, and Thrust Specific Fuel Consumption (TSFC) – Engine program can be $1B or more, take three years or more – Engine companies give guarantees in terms of fuel burn – Engine thrust needs to be correct or aircraft can’t take off in the required length – Airflow fundamental in diagnosing engine performance – These are basic and essential measures ? How do we measure thrust? ? How do we measure airflow? ? How do we measure fuel flow? 32 THRUST STANDS ? In practice, thrust is measured with load cells ? The engines, however, are often part of a complex test facility and are connected to upstream ducting ? There are thus certain systematic errors which need to be accounted for ? The level of uncertainty in the answer is desired to be less than one per cent ? There are a lot of corrections to be made to the raw data (measured load) to give the thrust 35 TEST STAND-TO-TEST STAND DIFFERENCES ? Want to have a consistent view of engine performance no matter who quotes the numbers ? This means that different test stands must be compared to see the differences ? Again, this is a major exercise involving the running of a jet engine in different locations under specified conditions ? The next slide shows the level of differences in the measurements