Test Matrices
16.621
Prof. Earll Murman
2
Variables
z Independent Variable: A quantity that you will vary and
control
– E.G.; angle of attack, chamber pressure or temperature,
coefficient in algorithm, luminosity, gain, ...
z Parameter: A quantity that is set or otherwise determined,
which you will not vary but which needs to be recorded
– E.G.; atmospheric pressure, constant in algorithm,
battery voltage,...
z Dependent Variable: A measureable output quantity of
your experiment which is a function of the input variables
and parameters
– E.G.; reaction time, force, energy consumed,
temperature
3
Exercise
z With your partner, write down your expected variables and
parameters.
z Independent Variables
z Parameters
z Dependent variables
4
Test Matrices
z A graphical display of your experimental independent
variables to help:
– Covey the scope of your experiment to your audience
– Plan and execute your experiment
z Each cell represents a “data point” for your experiment for
which you will collect values for the dependent variables.
4000
RPM
0 mph
5 mph
10 mph
15 mph
1000 RPM 2000 RPM 3000 RPM0 RPM
Propeller RPM
Speed .
Courtesy of Cyndi Vongvanith and Lester McCoy
IVs
DVs
5
Multi-Variable Experiments
– Factor = Number of independent variables
? Four-Factor experiment has 4 independent variables
– Level = A given value of an independent variable
? Numerical – 200, 300, 400 …
? Qualitative – Brand x, Brand y, Brand z …
– Full-Factorial Experiment
? All factors at all levels
? May lead to a huge number of data points.
– Fractional-Factorial Experiment
? Expert judgement: carefully selected subset
? Adaptive: decide as you get some data
? Design of Experiments: Taguchi, orthogonal arrays
– Beyond scope of 16.62X
6
Presentation of Test Matrices: Full Factorial
– Test matrix used for graphical representation of test plan
? Define: A
n
= n
th
level of factor A
– One variable matrix
– Two variable matrix
– Three variable matrix
mm2m1m
n22212
n12111
BABABA
BABABA
BABABA
Λ
ΜΟΜΜ
Λ
Λ
A
1
B
1
C
1
A
4
B
3
C
2
IV
A
1
...
A
n
DV1 DV2 DV3
7
– N- variable matrix
? Creativity needed
? Stamina will probably also be required!
D
1
E
1
F
1
D
4
E
3
F
2
D
1
E
1
F
1
D
4
E
3
F
2
For A
1
B
1
:
For A
2
B
7
:
Presentation of Test Matrices: Full Factorial
8
Expert Judgement Approach
– Eliminate some combinations
of independent variables to
reduce the total number of data
points
– Often required to make
experiment feasible within time
and budget constraints
– Strategies for elimination
? Insight from previous
theory or experiments
? Wisdom from advisor or
other subject matter expert
? Logical thought about
interrelationship of
variables on the physics of
the problem
9
Adaptive Approach
– Preliminary Runs
? Use theory to bracket range
? 2 or 3 test cases to check set-up
? Compare with theory
– Production Runs
? Data range and spacing
? May not be uniform
– Cluster samples in
“interesting areas”
Run
Data
Run
Data
10
Additional Considerations
z Repeatability: Is there reason to believe that the
measurement accuracy will be increased if multiple “runs”
are made with the same independent variables and
parameters?
z Hysteresis: Is there reason to believe the physical effect
being studied may depend upon the sequence or rate in
which you vary the independent variable?
z Learning: Is the reason to believe your human subjects or
intelligent software will become more capable during the
experiment through learning?
z Fatigue: Will your subjects become less capable during the
test due to tiring?
Refer to backup slides for more information.
11
Hysteresis
z Hysteresis - “The lagging of an effect behind its cause, as
when the change in magnetism of a body lags behind
changes in the magnetic field.” http://www.dictionary.com/
– Feature of physical problem be studied
– Feature of measurement device (undesirable)
z Example - pitching vs fixed delta wing
α increasing
α decreasing
α fixed
Hysteresis depends
upon pitching rate
12
Learning
z “The act, process, or experience of gaining knowledge or
skill” http://www.dictionary.com/
z The response of a human subject changes as an experiment
proceeds because they gain skill or knowledge - the
experiment changes the subject!
z E.G. measuring a response of a human to a video game
experiment
– Test 1 and Test 2 are different, but subject learns how
to play the game in test 1 and can respond more quickly
in test 2.
z Typical mitigation strategies
– Train subjects to fix skill level
– Test many subjects and vary the order of the test
sequence to average out the learning effect.
13
Fatigue
z Fatigue: “Physical or mental weariness resulting from
exertion.” http://www.dictionary.com/
z This is different than learning
– Learning leads to a new skill level
– Fatigue is a temporary loss of capability
z Fatigue can effect both the subject and the experimenter
– Needs to be considered in the design of the execution of
the experiment
z Fatigue can also apply to physical materials