1 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Multidisciplinary System Design Optimization (MSDO) Design Space Exploration Lecture 5 18 February 2004 Karen Willcox 2 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Today’s Topics Design of Experiments Overview Full Factorial Design Parameter Study One at a Time Latin Hypercubes Orthogonal Arrays Effects DoE Paper Airplane Experiment 3 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Design of Experiments A collection of statistical techniques providing a systematic way to sample the design space Useful when tackling a new problem for which you know very little about the design space. Study the effects of multiple input variables on one or more output parameters Often used before setting up a formal optimization problem – Identify key drivers among potential design variables – Identify appropriate design variable ranges – Identify achievable objective function values Often, DOE is used in the context of robust design. Today we will just talk about it for design space exploration. 4 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Design of Experiments Design variables = factors Values of design variables = levels Noise factors = variables over which we have no control e.g. manufacturing variation in blade thickness Control factors = variables we can control e.g. nominal blade thickness Outputs = observations (= objective functions) Factors + Levels “Experiment” Observation (Often an analysis code) 5 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Matrix Experiments Each row of the matrix corresponds to one experiment. Each column of the matrix corresponds to one factor. Each experiment corresponds to a different combination of factor levels and provides one observation. Expt No. Factor A Factor B Observation 1A1B1 K 1 2A1B2 K 2 3A2B1 K 3 4A2B2 K 4 Here, we have two factors, each of which can take two levels. 6 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Full-Factorial Experiment Specify levels for each factor Evaluate outputs at every combination of values – complete but expensive!n factors l levels l n observations Factor Expt No. AB 1 A1 B1 2 A1 B2 3 A1 B3 4 A2 B1 5 A2 B2 6 A2 B3 7 A3 B1 8 A3 B2 9 A3 B3 2 factors, 3 levels each: l n = 3 2 = 9 expts 4 factors, 3 levels each: l n = 3 4 = 81 expts 7 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Fractional Factorial Experiments Due to the combinatorial explosion, we cannot usually perform a full factorial experiment So instead we consider just some of the possible combinations Questions: – How many experiments do I need? – Which combination of levels should I choose? Need to balance experimental cost with design space coverage 8 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Fractional Factorial Design Initially, it may be useful to look at a large number of factors superficially rather than a small number of factors in detail: 1112 2212 12 , , , nnn fll fll fll # # 11121314 22122324 33132334 ,,,, ,,,,, ,,,,, fllll fllll fllll ! ! ! vs. many levels many factors 9 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics DoE Techniques Overview TECHNIQUE COMMENT EXPENSE (l=# levels, n=# factors) Full factorial design Evaluates all possible designs. l n -grows exponentially with number of factors Orthogonal arrays Don’t always seem to work - interactions? Moderate – depends on which array One at a time Order of factors? 1+n(l-1) - cheap Latin hypercubes Not reproducible, poor coverage if divisions are large. l - cheap Parameter study Captures no interactions. 1+n(l-1) - cheap 10 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Parameter Study Specify levels for each factor Change one factor at a time, all others at base level Consider each factor at every level Factor Expt No. ABCD C1 D1 D1 D1 D1 D1 D1 D1 D2 D3 C1 C1 C1 C1 C2 C3 C1 C1 4 A1 B2 5 A1 B3 6 A1 B1 7 A1 B1 8 A1 B1 1 A1 B1 2 A2 B1 3 A3 B1 9 A1 B1 n factors 1+n(l-1) evaluations l levels 4 factors, 3 levels each: 1+n(l-1) = 1+4(3-1) = 9 expts Baseline : A1, B1, C1, D1 11 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Parameter Study Select the best result for each factor Factor Expt No. AB C D Observation D1 K 1 K 2 K 3 K 4 K 5 K 6 K 7 K 8 K 9 D1 D1 D1 D1 D1 D1 D2 D3 C1 C1 C1 C1 C1 C2 C3 C1 C1 4 A1 B2 5 A1 B3 6 A1 B1 7 A1 B1 8 A1 B1 1 A1 B1 2 A2 B1 3 A3 B1 9 A1 B1 1. Compare K 1 , K 2 , K 3 ? A* 2. Compare K 1 , K 4 , K 5 ? B* 3. Compare K 1 , K 6 , K 7 ? C* 4. Compare K 1 , K 8 , K 9 ? D* “Best design” is A*,B*,C*,D* Does not capture interaction between variables 12 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics One At a Time Change first factor, all others at base value If output is improved, keep new level for that factor Move on to next factor and repeat Factor Expt No. ABCD C1 D1 D1 D1 D1 D1 D1 D1 D2 D3 C1 C1 C1 C1 C2 C3 C* C* 4 A* B2 5 A* B3 6 A* B* 7 A* B* 8 A* B* 1 A1 B1 2 A2 B1 3 A3 B1 9 A* B* n factors l levels 1+n(l-1) evaluations 4 factors, 3 levels each: 1+n(l-1) = 1+4(3-1) = 9 expts Result depends on order of factors 13 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Parameter Study vs. One at a Time Parameter study: – Chances are you will not actually evaluate the “best design” as part of your original experiment – “Best design” is chosen by extrapolating each factor’s behavior, but interactions are not considered One at a Time: – The “best design” is a member of your matrix experiment – Some interactions are captured, even though the result depends on the order of the factors 14 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Latin Hypercubes Divide design space uniformly into l divisions for each factor Combine levels randomly – specify l points – use each level of a factor only once e.g. two factors, four levels each: A A1 A2 A3 A4 B B1 B2 B3 B4 Results not repeatable Can have poor coverage 15 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Orthogonal Arrays Specify levels for each factor Use arrays to choose a subset of the full- factorial experiment Subset selected to maintain orthogonality between factors n factors subset of l n evaluations l levels Does not capture all interactions, but is efficient Experiment is balanced 16 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Orthogonal Arrays D3C2B1A24 D1C3B2A25 D2C1B3A26 D2C3B1A37 D3C1B2A38 D1 D3 D2 D1 D Factor C3B3A13 C2B3A39 C2B2A12 C1B1A11 CBA Expt No. Factor Expt No. AB C 1 A1 B1 C1 2 A1 B2 C2 3 A2 B1 C2 4 A2 B2 C1 L 4 (2 3 ) 4 expts 3 factors L 9 (3 4 ) 9 expts 4 factors 3 levels 2 levels 17 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Orthogonality Notice that for any pair of columns, all combinations of factor levels occur and they occur an equal number of times. This is the balancing property. In general, the balancing property is sufficient for orthogonality. There is a formal statistical definition of orthogonality, but we will not go into it here. Factor D D1 D2 D3 4 A2 B1 C2 D3 5 A2 B2 C3 D1 6 A2 B3 C1 D2 7 A3 B1 C3 D2 8 A3 B2 C1 D3 D1 Expt No. AB C 1 A1 B1 C1 2 A1 B2 C2 3 A1 B3 C3 9 A3 B3 C2 All of the combinations (1-1, 1-2, 1-3, 2-1, 2-2, 2-3, 3-1, 3-2, 3-3) occur once for each pair of columns. L 9 (3 4 ) 18 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Effects Once the experiments have been performed, the results can be used to calculate effects. The effect of a factor is the change in the response as the level of the factor is changed. – Main effects: averaged individual measures of effects of factors – Interaction effects: the effect of a factor depends on the level of another factor Often, the effect is determined for a change from a minus level (-) to a plus level (+) (2-level experiments). 19 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Effects Consider the following experiment: – We are studying the effect of three factors on the price of an aircraft – The factors are the number of seats, range and aircraft manufacturer – Each factor can take two levels: Factor 1: Seats 100<S1<150 150<S2<200 Factor 2: Range (nm) 2000<R1<2800 2800<R2<3500 Factor 3: Manufacturer M1=Boeing M2=Airbus 20 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Main Effects Expt No. Seats (S) Range (R) Mfr (M) Price (observation) 1 S1 R1 M1 P 1 2 S1 R1 M2 P 2 3 S1 R2 M1 P 3 4 S1 R2 M2 P 4 5 S2 R1 M1 P 5 6 S2 R1 M2 P 6 7 S2 R2 M1 P 7 8 S2 R2 M2 P 8 L 8 (2 3 ) (full factorial design) The main effect of a factor is the effect of that factor on the output averaged across the levels of other factors. 21 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Main Effects Question: what is the main effect of manufacturer? i.e. from our experiments, can we predict how the price is affected by whether Boeing or Airbus makes the aircraft? Expt No. Seats (S) Range (R) Mfr (M) Price (observation) 1 S1 R1 M1 P 1 2 S1 R1 M2 P 2 3 S1 R2 M1 P 3 4 S1 R2 M2 P 4 5 S2 R1 M1 P 5 6 S2 R1 M2 P 6 7 S2 R2 M1 P 7 8 S2 R2 M2 P 8 21 43 65 87 (P -P )+ (P -P )+ (P -P )+ (P -P ) 4 main effect of manufacturer = expts 1 and 2 differ only in the manufacturer 22 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Main Effects – Another Interpretation 123 45678 + + + 8 PPPPPPPP m  overall mean response: 135 7 1 4 M PPPP m  avg over all expts when M=M1 : effect of mfr level M1 = 1M mm Effect of factor level can be defined for multiple levels effect of mfr level M2 = 2M mm main effect of mfr = 21MM mm Main effect of factor is defined as difference between two levels NOTE: The main effect should be interpreted individually only if the variable does not appear to interact with other variables 23 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Main Effect Example Expt No. Aircraft Seats (S) Range (R) Mfr (M) Price ($M) 1 717 A318-100 737-700 A319-100 737-900 A321-200 737-800 A320-200 S1 R1 M1 24.0 2 S1 R1 M2 29.3 3 S1 R2 M1 33.0 4 S1 R2 M2 35.0 5 S2 R1 M1 43.7 6 S2 R1 M2 48.0 7 S2 R2 M1 39.1 8 S2 R2 M2 38.0 Sources: Seats/Range data: Boeing Quick Looks Price data: Aircraft Value News Airline Monitor, May 2001 issue 100<S1<150 150<S2<200 2000<R1<2800 2800<R2<3500 M1=Boeing M2=Airbus 24 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Main Effect Example overall mean price = 1/8*(24.0+29.3+33.0+35.0+43.7+48.0+39.1+38.0) = 36.26 mean of experiments with M1 = 1/4*(24.0+33.0+43.7+39.1) = 34.95 mean of experiments with M2 = 1/4*(29.3+35.0+48.0+38.0) = 37.58 Main effect of Boeing (M1) = 34.95 – 36.26 = -1.3 Main effect of Airbus (M2) = 37.58 – 36.26 = 1.3 Main effect of manufacturer = 37.58 – 34.95 = 2.6 Interpretation? 25 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Interaction Effects We can also measure interaction effects between factors. Answers the question: does the effect of a factor depend on the level of another factor? e.g. Does the effect of manufacturer depend on whether we consider shorter range or longer range aircraft? The interaction between manufacturer and range is defined as half the difference between the average manufacturer effect with range 2 and the average manufacturer effect with range 1. mfr u range interaction avg mfr effect with range 1 avg mfr effect with range 2 = - 2 26 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Interaction Effects Expt No. Seats (S) Range (R) Mfr (M) Price ($M) 1 S1 R1 M1 24.0 2 S1 R1 M2 29.3 3 S1 R2 M1 33.0 4 S1 R2 M2 35.0 5 S2 R1 M1 43.7 6 S2 R1 M2 48.0 7 S2 R2 M1 39.1 8 S2 R2 M2 38.0 range R1 : expts 1,2,5,6 range R2 : expts 3,4,7,8 21 65 (-)+ (-) 2 PP PP avg mfr effect with range 1 (29.3-24.0)+ (48.0-43.7) 4.8 2 avg mfr effect with range 2 43 87 (-)+ (-) 2 PP PP (35.0-33.0)+ (38.0-39.1) 0.45 2 mfr u range interaction 0.45 4.8 2.2 2   Interpretation? 27 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Interpretation of Effects seats manufacturer Main effects are the difference between two averages range seats range mfr S2 S1 1 2 3 4 5 6 7 8 R1 R2 1 2 3 4 5 6 7 8 M2 M1 1 2 3 4 5 6 7 8 main effect of mfr Expt No. Seats (S) Range (R) Mfr (M) 1 S1 R1 M1 2 S1 R1 M2 3 S1 R2 M1 4 S1 R2 M2 5 S2 R1 M1 6 S2 R1 M2 7 S2 R2 M1 8 S2 R2 M2 8642 7531 (+++)-(+++) 4 PPPP PPPP Adapted from Fig 10.2 Box, Hunter & Hunter 28 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Interpretation of Effects Expt No. Seats (S) Range (R) Mfr (M) 1 S1 R1 M1 2 S1 R1 M2 3 S1 R2 M1 4 S1 R2 M2 5 S2 R1 M1 6 S2 R1 M2 7 S2 R2 M1 8 S2 R2 M2 seats u range mfr u seats Interaction effects are also the difference between two averages, but the planes are no longer parallel - seats range mfr 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 8541 7632 (+++)-(+++) 4 PPPP PPPP mfr u range interaction Adapted from Fig 10.2 Box, Hunter & Hunter mfr u range 1 2 3 4 5 6 7 8 29 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Design Experiment Objective: Maximize Airplane Glide Distance Design Variables: Weight Distribution Stabilizer Orientation Nose Length Wing Angle Three levels for each design variable. Experiment courtesy of Prof. Eppinger 30 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Design Experiment Full factorial design : 3 4 =81 experiments We will use an L 9 (3 4 ) orthogonal array: Expt No. Weight A Stabilizer B Nose C Wing D 1 A1 B1 C1 D1 D2 D3 4 A2 B1 C2 D3 5 A2 B2 C3 D1 6 A2 B3 C1 D2 7 A3 B1 C3 D2 8 A3 B2 C1 D3 D1 2 A1 B2 C2 3 A1 B3 C3 9 A3 B3 C2 31 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics Design Experiment Things to think about ... Given just 9 out of a possible 81 experiments, can we predict the optimal airplane? Do some design variables seem to have a larger effect on the objective than others (sensitivity)? Are there other factors affecting the results (noise)? 32 ? Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Engineering Systems Division and Dept. of Aeronautics and Astronautics References Phadke, : Quality Engineering Using Robust Design, Prentice Hall, 1995 Box, G.; Hunter, W. and Hunter, J.: Statistics for Experimenters, John Wiley & Sons, 1978.