Interface System Supervisory Control Computer Display Control Sensors Direct Observation 16.422 Workload and Situation Awareness Prof. R. John Hansman Acknowledgements to Mica Ensley Interface System Supervisory Control Computer Display Control Sensors Direct Observation Workload ü What is workload? ü Why is it important? Interface System Supervisory Control Computer Display Control Sensors Direct Observation Driving Case: B757/767 2 or 3 person crew ? ü Prior to 767 somewhat arbitrary break at 100 seats o DC-9 (2 person crew - pilot, co-pilot) o B-727 (3 person crew - pilot, co-pilot, flight engineer) ü B-757/767 Designed for 2 person Crew o Use of automation and simplified systems so minimize systems management o Use of Advanced Cockpit to Increase SA and make primary flight tasks easier ü Safety concerns raised by Air Line Pilots Association (ALPA) o Workload o Off Nominal and Emergency Conditions (eg manual pressurization) o Job Protection issues ü Workload became political and regulatory issue Interface System Supervisory Control Computer Display Control Sensors Direct Observation Workload Definitions? ü Physical Workload o Traditional view of work for manual labor o Can be measured in physical terms (ergs, joules, ..) o Limited impact of skill to minimize (ie subject variability) ü “Mental” Workload o Often not related to physical work o Internal measure difficult to observe o Varies with task difficulty and complexity o Significant subject variability o No real consensus on what it is o Workload is a “dirty” word in Experimental Psychology ü Activity o Things that are done o Physical activity easy to measure ü Taskload o External measure of tasks which need to be done o Can be weighted for factors such as task difficulty or complexity Interface Yerks-Dotson Law System Supervisory Control Computer Display Control Sensors Direct Observation http://www.hf.faa.gov/Webtraining/Cognition/Workload/Mental3.htm Interface Curve System Supervisory Control Computer Display Control Sensors Direct Observation Typical Performance vs. Task Load Performance Task Load Helicopter Observation of Driver Example Interface System Supervisory Control Computer Display Control Sensors Direct Observation Off Nominal Considerations ü System design often driven by off-nominal conditions o Emergencies o System Failures o Failure of the Automation system ü Secondary task considerations ü Cockpit Example o Emergency diversion o Depressurization Interface System Supervisory Control Computer Display Control Sensors Direct Observation Workload Measurement Approaches ü Objective Performance Approaches o Primary Task (Yerks Dodson) o Secondary Task (works well to measure saturation threshold) u Concept of Spare Cognitive Capacity ü Objective Physiological Measures (weak) o Heart Rate Variability o Pupil Diameter o EEG P 300 o Skin Galvanic Response o New Imaging Methods ü Subjective Workload Assessment Techniques o Formal o Direct Query Interface System Supervisory Control Computer Display Control Sensors Direct Observation Subjective Assessment Techniques ü Simpson-Sheridan/ Cooper-Harper ü Bedford Scale ü Rate or Perceived Exertion (RPE) ü NASA Task Load Index (TLX) ü Defense Research Agency Workload Scale (DRAWS) ü Malvern Capacity Estimate (MCE) Interface System Supervisory Control Computer Display Control Sensors Direct Observation Simpson-Sheridan Scale ü Modified Cooper Harper Scale for Workload System Supervisory Control Computer Interface Display Control Sensors Direct Observation Cooper Harper Source: http://history.nasa.gov/SP-3300 Interface System Supervisory Control Computer Display Control Sensors Direct Observation Bedford Scale ü The Bedford Scale is a uni-dimensional rating scale designed to identify operator's spare mental capacity while completing a task. The single dimension is assessed using a hierarchical decision tree that guides the operator through a ten-point rating scale, each point of which is accompanied by a descriptor of the associated level of workload. It is simple, quick and easy to apply in situ to assess task load in high workload environments, but it does not have a diagnostic capability. ü Refs: Roscoe and Ellis, 199 Source: Eurocontrol http://www.eurocontrol.int/eatmp/hifa/hifa/HIFAdata_tools_workload.html Interface System Supervisory Control Computer Display Control Sensors Direct Observation Rate of Perceived Exertion Borg RPE Scale ü 6 No exertion at all ü 7 Extremely Light ü 8 ü 9 Very Light ü 10 ü 11 Light ü 12 ü 13 Somewhat Hard ü 14 ü 15 Hard (Heavy) ü 16 ü 17 Very Hard ü 18 ü 19 Extremely Hard ü 20 Maximal Exertion ü Borg Rate of Perceived Exertion Scale ü Originally developed for physical workload ü Intended to be ordinal scale ü Modified 0-10 version CR-10 Source: http://dticam.dtic.mil Interface NASA TLX Task Load Index System Supervisory Control Computer Display Control Sensors Direct Observation ü Sandy Hart ü 5 Element Structured Subjective Assessment ü Individual relative element calibration ü Requires Trained Users ü Often used but difficult to interpert http://www.hf.faa.gov/Webtraining/Cognition/Workload/Mental3.htm Interface System Supervisory Control Computer Display Control Sensors Direct Observation DRAWS ü DRAWS is a multi-dimensional tool (similar to NASA TLX) used to gain a subjective assessment of workload from operators. The rating scales are input demand (demand from the acquisition of information from external sources), central demand (demand from mental operations), output demand (demand from the responses required by the task), and time pressure (demand from the rate at which tasks must be performed). DRAWS offers ease of data collection and ratings can be obtained during task performance by asking respondent to call out ratings (from 0 to 100) to verbal prompts. This can also provide a workload profile through a task sequence. ü Refs: Farmer et al, 1995; Jordan et al, 1995. Source: Eurocontrol http://www.eurocontrol.int/eatmp/hifa/hifa/HIFAdata_tools_workload.html Interface System Supervisory Control Computer Display Control Sensors Direct Observation Malvern Capacity Estimate ü MACE is designed as a quick simple and direct measure of maximum capacity. It is designed to provide a direct measure of air traffic controllers' subjective estimates of their own aircraft handling capacity. MACE is applied at the end of a work sequence (e.g., simulation trial) and provides capacity estimates in aircraft per hour. Applications have typically been in simulation environments. ü Refs: Goillau and Kelly, 1996. Source: Eurocontrol http://www.eurocontrol.int/eatmp/hifa/hifa/HIFAdata_tools_workload.html Interface System Supervisory Control Computer Display Control Sensors Direct Observation Instant Self Assessment of Workload (ISA) ü ISA was developed as a tool that an operator could use to estimate their perceived workload during real-time simulations. The operator is prompted at regular intervals to give a rating of 1 to 5 of how busy he is (1 means under-utilized, 5 means excessively busy). These data can be used to compare operators' perceived workload, for example, with and without a particular tool, or between different systems. ü Refs: Jordan, 1992. Source: Eurocontrol http://www.eurocontrol.int/eatmp/hifa/hifa/HIFAdata_tools_workload.html Interface System Supervisory Control Computer Display Control Sensors Direct Observation Subjective Workload Assessment Techniques (SWAT) ü SWAT is a subjective scale of workload that can be administered easily in operation situations and is available as a PC-based software tool. It is multi-dimensional tool incorporating factors of temporal load, mental effort and psychological stress. SWAT has two stages: The respondent ranks the levels of the three workload scales in order from the lowest to highest workload prior to the trial, and rates each of the scales during the trial. It was originally designed to assess aircraft cockpit and other crew-station environments to assess the workload associated with the operators' activities. ü Refs: Reid and Nygren, 1988; Dean 1997 Source: Eurocontrol http://www.eurocontrol.int/eatmp/hifa/hifa/HIFAdata_tools_workload.html Interface System Supervisory Control Computer Display Control Sensors Direct Observation S i t u a t i o n Aw a r e n e s s ü Term originally defined for air combat ü Working Definition (Hansman) : Sufficiently detailed mental picture of the vehicle and environment (i.e. world model) to allow the operator to make well- informed (i.e., conditionally correct) decisions. ü Individual SA and Team SA ü Has become an extremely popular and powerful concept ü Mica Endsley: Situation vs Situational Awareness Interface System Supervisory Control Computer Display Control Sensors Direct Observation Endsley Situation Awareness Model (Image removed due to copyright considerations.) System Supervisory Control Computer Interface Display Control Sensors Direct Observation References: Endsley, 1995; Pawlak, 1996; Reynolds et al., 2002 3 Projection DECISION 3 - Projection 2 - Comprehension1- Perception Weather State Aircraft State Interaction SITUATIONAL AWARENESS Weather Phenomenology Aircraft Envelope Future Exposure Weather Forecast Aircraft Trajectory PERFORMANCE OF ACTIONS Implement Information System Aircraft Trajectory Control Pilot Monitoring Evaluating Planning PLAN Contingency Plans Nominal Plan Weather Mental Model Training Experience Procedures Adjusting Inform ation Information Request/Transmission Aircraft Situation Dynamics Interaction Weather PERCEP TION Model of Pilots’ Cognitive Constructs of Information Processing Interface System Supervisory Control Computer Display Control Sensors Direct Observation Enhancing SA ü Level 1 - Perception o Enhanced Perception Systems ( eg Enhanced Vision Systems) o Alerting Systems ü Level 2 - Comprehension o SA Displays (eg Moving Map Displays, EGPWS) ü Level 3 - Projection o Displays o Decision Support Tools Interface System Supervisory Control Computer Display Control Sensors Direct Observation Enhancing SA ü Level 1 - Perception o Enhanced Perception Systems ( eg Enhanced Vision Systems) o Alerting Systems ü Level 2 - Comprehension o SA Displays (eg Moving Map Displays, EGPWS) ü Level 3 - Projection o Displays o Decision Support Tools Interface System Supervisory Control Computer Display Control Sensors Direct Observation Enhanced Vision & Synthetic Vision Systems Enhanced Vision Synthetic Vision Interface System Supervisory Control Computer Display Control Sensors Direct Observation Enhanced Vision Picture of the outside world created by real-time weather and darkness penetrating on-board sensors (eg. Cameras, FLIR, MMW radar, and weather radar). Interface System Supervisory Control Computer Display Control Sensors Direct Observation Synthetic Vision Picture of the outside world created by combining precise navigation position with databases of comprehensive geographic, cultural and tactical information. Interface System Supervisory Control Computer Display Control Sensors Direct Observation Enhancing SA ü Level 1 - Perception o Enhanced Perception Systems ( eg Enhanced Vision Systems) o Alerting Systems ü Level 2 - Comprehension o SA Displays (eg Moving Map Displays, EGPWS) ü Level 3 - Projection o Displays o Decision Support Tools Interface System Supervisory Control Computer Display Control Sensors Direct Observation Enhancing SA ü Level 1 - Perception o Enhanced Perception Systems ( eg Enhanced Vision Systems) o Alerting Systems ü Level 2 - Comprehension o SA Displays (eg Moving Map Displays, EGPWS) ü Level 3 - Projection o Displays o Decision Support Tools Interface System Supervisory Control Computer Display Control Sensors Direct Observation New Weather Datalink Products ARNAV Avidyne Bendix/King FAA FISDL Control Vision Echo Flight Garmin UPS – AirCell Vigyan System Supervisory Control Computer Interface Display Control Sensors Direct Observation References: Endsley, 1995; Pawlak, 1996; Reynolds et al., 2002 3 Projection DECISION 3 - Projection 2 - Comprehension1- Perception Weather State Aircraft State Interaction SITUATIONAL AWARENESS Weather Phenomenology Aircraft Envelope Future Exposure Weather Forecast Aircraft Trajectory PERFORMANCE OF ACTIONS Implement Information System Aircraft Trajectory Control Pilot Monitoring Evaluating Planning PLAN Contingency Plans Nominal Plan Weather Mental Model Training Experience Procedures Adjusting Inform ation Information Request/Transmission Aircraft Situation Dynamics Interaction Weather PERCEP TION Model of Pilots’ Cognitive Constructs of Information Processing System Supervisory Control Computer Interface Display Control Sensors Direct Observation Temporal Representation of Pilots’ Functions Aircraft Situation Dynamics Interaction Weather Strategic Tactical Reactive TEMPORAL REGIMES OF PLANNING Execution In-Flight Planning Pre-Flight Planning Weather Information min hrs - mins day - hrs PILOTS’ FUNCTIONS Aircraft Information Go/ No-Go Aircraft Trajectory Control Information Information Request/Transmission Interface System Supervisory Control Computer Display Control Sensors Direct Observation Time constants dependent on: Temporal Regimes of Wx Predictability Unce row orecas orizon - Weather phenomena and phenomenology (e.g., convective weather, droplet size distribution, temperature) - Phase of weather phenomena (e.g., storm initiation versus storm decay) rtainty G th with F t H Uncertainty Persistence ? ? Weather Forecast Probabilistic Deterministic U(t) Time of Forecast Limit of Deterministic Weather Forecast Issuance Predictability Horizon Interface Temporal Regimes of Cognitive Projection System Supervisory Control Computer Display Control Sensors Direct Observation Weather representation Weather representation Weather Uncertainty Growth with Horizon of Projection based on observation over a based on deterministic representation at time period where conditions forecast of “acceptable” do not significantly change accuracy time in future beyond “predictability limit” Weather Projection Uncertainty ? ? Stochastic Deterministic Constant U(t) Reference Time of Limit of Deterministic Horizon of Cognitive Weather Mental Model Projection Projection Interface Temporal Framework of Decision-Making Representation of Cognitive Plan System Supervisory Control Computer Display Control Sensors Direct Observation Horizon of Cognitive Weather Projection Tactical Planning Dynamics Reactive Strategic Stochastic Deterministic Constant Pilots’ Planning Horizon Time of Information Production Time Time of Planning Interface System Supervisory Control Computer Display Control Sensors Direct Observation Representation of Cognitive Plan Examples Horizon of Tactical + 1 hr + 2 hr Landing Before + 0 hr Front +30 min + 1 day Reactive Strategic Stochastic Deterministic Constant Pilots’ Planning Horizon Cognitive Weather Projection Time of Information Production Time Initial Climb Around Front Front Passage Convective Microburst Volcanic Ash Time of Planning Interface Measurement of Situation Awareness System Supervisory Control Computer Display Control Sensors Direct Observation ü Situation Awareness General Assessment Technique (SAGAT) o Endsley o Requires interruptions o Invasive (queries may influence subsequent SA) o Time issue o Requires knowledge of required SA elements u Goal Directed Task Analysis ü Testable Response Approach o Pritchett and Hansman o Works for scenario based studies o Requires scenarios where differential SA implies differential action Interface System Supervisory Control Computer Display Control Sensors Direct Observation Datalink Shared Information Experiment (Traffic & Weather) Simulator Data Simulator Data via Internet via Internet Data link OFF ON ON Data link Weather Traffic Simulation Host Scenario Generation Pseudo-Pilot Station Secondary Traffic Weather Data TW443 220C DL102 170C NW589 335C QF004 360C SLB Advanced Cockpit OFF Plan View Display Simulator Voice Communication Link via Internet Pilot Air Traffic Controller Interface From the Cockpit System Supervisory Control Computer Display Control Sensors Direct Observation Data link OFF Data link ON Interface From the ATC Display System Supervisory Control Computer Display Control Sensors Direct Observation Data link OFF Data link ON Interface Pseudo-Pilot Station System Supervisory Control Computer Display Control Sensors Direct Observation Interface System Supervisory Control Computer Display Control Sensors Direct Observation Example Scenario ü 12-18 aircraft ü Convective weather ü Performed once without the shared information ü Repeated once with the shared information ü 6 subjects x 6 runs each = 36 runs total ü ~10 minutes in duration ü Averaged 80-90 voice transmissions per run ü Recorded data: o Situation awareness data o Aircraft trajectories o Voice data o Workload data o Subjective ratings Interface System Supervisory Control Computer Display Control Sensors Direct Observation Results: Situation Awareness ü Controllers’ situation awareness with respect to weather improves when weather information is shared ü Pilots’ situation awareness with respect to traffic improves when traffic information is shared Weather Situation Awareness Traffic Situation Awareness Data link OFF Data link ON Data link OFF Data link ON Aware 100% Aware 100% Aware Unaware 39% 61% Unaware 94% Aware 89% Aware 56% Aware 94% Aware 50% Aware Ambiguous Not aware Pilo t C ontro ller Pilo t C ontro ller Interface Results: Controllers’ Weather Awareness System Supervisory Control Computer Display Control Sensors Direct Observation Scenario 1 Scenario 2 Scenario 3 Subject 1 Subject 3 Subject 5 Interface Vertical Separation (ft) System Supervisory Control Computer Display Control Sensors Direct Observation Results: Separation Violations ü 5 operational errors observed in 36 scenario runs o All occurred in the non-datalinked configuration 0 200 400 600 800 1000 1 2 3 4 5 Vertical Separation (ft) 1 Conflict precipitated by a late deviation around weather 2 Several aircraft diverting through same hole in weather 3 A/C not handed off; conflict occurred outside the sector 4 Pilot blundered (turned in wrong direction) 5 Pilot blundered 0 1 2 3 4 5 Lateral Separation (nm) (wrong A/P mode for descent) Interface System Supervisory Control Computer Display Control Sensors Direct Observation s1 Results: Separation Violations : total separation < 100 feet