MIT ICAT Human and Automation Integration Considerations for UAV Systems Prof. R. John Hansman Roland Weibel MIT International Center for Air Transportation Department of Aeronautics & Astronautics MIT ICAT Possible Commercial UAV Applications - Motivation y Remote Sensing ? Meteorology ? Scientific Research ? Aerial Photography/ Mapping ? Pipeline Spotting ? Disaster Monitoring ? Agriculture y Surveillance ? Border Patrol ? Homeland Security/ Law Enforcement ? Traffic Monitoring ? Search and Rescue y Data Delivery ? Communications Relay ? Multimedia Broadcast y Cargo Transport MIT ICAT Possible Military UAV Missions - Motivation y Intelligence ? Reconnaissance ? Target Monitoring ? Forward Air Control ? Electronic Warfare ? Search and Rescue ? Battle Damage Assessment (BDA) y Offensive Operation ? Suppression of Enemy Air Defenses (SEAD) ? Close Air Support ? Deep Strike y Cargo Transport MIT ICAT Current Unmanned Aerial Vehicles Aerovironment Black Widow – 2.12 oz. BAE Systems Microstar – 3.0 oz. Sig Kadet II RC Trainer – 5 lb Aerovironment Pointer – 9.6 lb Boeing/ Insitu Scaneagle – 33 lb IAI Scout – 351 lb Boeing X-45A UCAV – 12,195 lb (est) Micro Mini Tactical High Alt / UCAVShort Range Bell Eagle Eye – 2,250 lb Allied Aero. LADF – 3.8 lb NOAA Weather Balloon 2-6 lb Gen. Atomics – Predator B – 7,000 lb Northrop-Grumman Global Hawk 25,600 lb UAV Weight (lb) 0 1 10 100 1,000 10,000 100,000 **Mass Range** Large range of UAV types as users of NAS -propulsion, configuration, capabilities, etc MIT ICAT Ceiling S hed M k 3 Fo x G nat G nat 2 P r edat or B Pr o w l e r I I He ro n S eas c a n R oboc opt er H unt er D r agon E y e C a m c opt er H e r m es 1500 Mi n i - V S u rv e y -Co p t e r 1 S hadow 200 He l i o s Po i n te r Az i m u t B i odr one Pe r s e u s P h oeni x E agl e E y e S heddon E agl e 2 Luna Al tu s II P r edat or S ear c h er S c out Sp e c tr e S o l a r B i rd R apt or G l obal H a w k F i r e S c out S e nder RM A X 0 10000 20000 30000 40000 50000 60000 70000 1 10 100 1000 10000 100000 Max TO Weight (lb) Ce i l i n g (ft) Electric Piston Turbocharged Turboprop Turboshaft Turbofan + * Micro Mini Tactical MALE HALE Rotary Legend FL 600 18,000 ft Class A Class G Classes B-E, G MIT ICAT Takeoff Method Hand-launched: Aerovironment Pointer Rocket-Assisted: Hunter UAV Rail-Launched: Sperwar Tilt-Rotor: Eagle Eye Runway Takeoff: X-45 UCAV MIT ICAT Basic Supervisory Control Architecture Human Operator Displays Controls Human Int. Computer Communications Channel Task Int. Computer Vehicle Sensors Controlled Process Adapted from Sheridan, Humans and Automation MIT ICAT UAV Operation Basic Functional Architecture UAV vehicle Environment (A) controls operator (vehicle) operator (sensors) commands Payload displays feedback transmission sensors sensors sensors Air Traffic Control surveillance reporting direct control commands reporting/ negotiation operations controller (dispatch) (customer) (field commander) displays Other A/C surveillance MIT ICAT Pointer UAV y Used for Short-Range Surveillance ? Battlefield commanders ? Law Enforcement y Vehicle Capabilities ? Manual Control ? Autopilot ? Sensor Integration and Display ? Loss of Link Return to Base y Bandwidth Requirements ? Transmission of Vehicle Commands ? Receipt of Sensor Intelligence, Vehicle State MIT ICAT Pointer UAV Tasking & Control pilot sensor operator speech ground station vehicle CDU MCP controls control surfaces UAV displays displays manual control state commands traj commands FMC auto- pilot Lost Link Procedure Aircraft Control Plan Actions Monitor experience, training tasking/interpretation Plan Actions Monitor- Interpret Control Camera Implement tasking Commander Interpret tasking experience, traininggoals Recording camera guidance sensor integration goals Camera sensors General Atomics Predator Medium Altitude, Endurance MIT ICAT Predator Air Vehicle Operator (AVO) Station from – M. Draper, Air Force Research Lab (2001) Northrop-Grumman Global Hawk HALE UAV MIT ICAT MIT ICAT Global Hawk Mission Control Elements y Navigation Plan y Communications Plan y Sensor Plan y Dissemination Plan y Dynamic Retasking y View Imagery y Monitor Sensor Status y Calibrate Sensors y Process & Disseminate Imagery Mission Planning Station Sensor Data and Processing Station y Interface with ATC y Uplink Mission Changes y Monitor Vehicle Health and Status y Monitor Threat Warning and Deception Air Vehicle Operator Station y Maintain Health and Status of Comm. Subsystems y Construct and Monitor Comm. Plan Communication and Control Station Command & Control ATC Battlefield Intelligence U.S. Air Force photo MIT ICAT Global Hawk MCE MIT ICAT Boeing X-45 UCAV Boeing X-45A Control Station from – DARPA Website (2003) MIT ICAT Multiple UAV Control Station for Simulated Scenario from – J. Nalepka, Air Force Research Lab (2003) MIT ICAT MIT ICAT MIT ICAT MIT ICAT UAV-Related Human Factors Issues - (Partial List) y Allocation/ Level of Autonomy y Bandwidth/ Latency y Situation Awareness y Cognitive Complexity Limitations ? Single & Multiple UAVs y Information Saturation/ Boredom y Simulator Sickness y Operator Orientation Confusion y Culture Resistance y Judgment ? Acceptable Risk ? Weapons Release Authorization MIT ICAT UAV Task Analysis y Situation (Battlespace) Awareness ? Perception ? Comprehension ? Projection y Diagnosis ? Environment ? Threat ? Targets y Strategic Planning/ Re-planning ? Goal Management ? Route planning y Tactical Decisions ? Weapons Authorization ? Avoidance of Hazards ? Systems Management y Control ? Navigation ? Aircraft Configuration ? Sensor Operation y Monitoring ? Vehicle Health ? External Environment ? Threats, Targets, Traffic ? Risk Assessment ? Communications Link ? Sensor Data y Communication ? Current State ? Intent ? Intelligence ? Tasking MIT ICAT AFRL Levels of Autonomy Remote Manual Control 1. Remotely Guided 2. Real Time Health Diagnosis 3. Adapt to Failures & Flight Conditions 4. Onboard Route Replan 5. Group Coordination 6. Group Tactical Replan 7. Group Tactical Goals 8. Distributed Control 9. Group Strategic Goals 10. Fully Autonomous Swarms Fully Autonomous, World Aware Currently Realized Supervisory Control Full Continuum MIT ICAT Level of Autonomy Trend Source: DOD UAV Roadmap, 2000 MIT ICAT MIT ICAT UAV Design Space - Military Level of Autonomy/ System Complexity Waypoint Designation Health Monitoring Tactical Replan Group Coord Pointer Pioneer Shadow Predator Global Hawk X-45 @add pictures@ Manual Pilotage Tactical Scout Battlefield Monitoring Multiship Coord Mission Complexity MIT ICAT Diagnosis Procedure Role Intelligence Goals Target Detection Diagnosis Procedure Experience Training UAV Platform Sensors Command Console Display Human Lethal Force Authorization Importance of Situation Awareness MIT ICAT Endsley Situation Awareness Model ?System Capability ?Interface Design ?Stress & Workload Performance Of Actions Decision Perception Of Elements In Current Situation Level 1 Comprehension Of Current Situation Level 2 Projection Of Future Status Level 3 ?Goals & Objectives ?Preconceptions (Expectations) Information Processing Mechanisms Long Term Memory Stores Automatically ?Abilities ?Experience ?Training State of the Environment Feedback Situation Awareness ?Complexity ?Automation Task/System Factors Individual Factors MIT ICAT Bandwidth Limits Human Operator Displays Controls Human Int. Computer Communications Channel Bandwidth Limits Task Int. Computer Vehicle Sensors Controlled Process Adapted from Sheridan, Humans and Automation MIT ICAT Bandwidth Limit Downlink y Video ? Forward View, Surveillance y Imagery ? Reconnaissance, Target Selection y Voice ? ATC Comm, Intelligence y Schematic Data ? System Health, Location Uplink y Voice ? ATC Comm, Comm to Ground y Manual Control y Commands ? Waypoint/ Tasking Commands MIT ICAT Task Performance & Bandwidth Frame Rate Constant Task Performance Color Depth Constant Bitrate Resolution Diagram from Sheridan, Teleoperation MIT ICAT Communications Latency Problems Air Traffic Control Other Traffic Operator + + + + UAV Satellite Link communication over time channel used channel free receipt delay transmission delay Satellite Latency Cycle Times : 2-5 sec PIO Issues due to lags. MIT ICAT Multiple Vehicle Control y Situation Awareness ? “Big Picture” Overview of Battlefield ? Orientation Confusion Multiple Reference Frames ? N Vehicle states ? N Vehicle status ? Kindergarten Model y Human/ Machine Allocation ? Level of Vehicle Autonomy ? Need for Higher Level of Abstraction (Macro vs Micro Management) ? Organizational vs Operator Model ? Directed vs Behavioral Automation ? Dynamic re-allocation y Cognitive Workload - Taskload ? How many vehicles can be reliably managed ? Cognitive Complexity Limitations ? ATC Analogy (Acceptable Level of Traffic) MIT ICAT AIR TRAFFIC SITUATION ATC OPERATIONAL CONTEXT AIR TRAFFIC CONTROLLER SITUATION AWARENESS LEVEL 1 Perception LEVEL 2 Comprehension LEVEL 3 Projection DECISION PROCESSES x Monitoring x Evaluating x Planning PERFORMANCE OF ACTIONS x Implementing “CURRENT PLAN” WORKING MENTAL MODEL Surveillance Path Command Path Complexity Concepts & Controller Process Model COGNITIVE COMPLEXITY PERCEIVED COMPLEXITY Adapted from Endsley Situation Awareness Model, Pawlak Key ATC Processes STRUCTURE SITUATION COMPLEXITY MIT ICAT Human-System Interface Issues y Interface Comparison - UAV vs Commercial ? DARPA USAF Boeing X-45 Example ? Boeing B-777 y Source: Build 2 Operational Simulation Overview Briefing ? Caveats: ? Prototype not operational system ? Briefing may not reflect actual system ? PC based interface MIT ICAT Interface Design Comparison PC vs Commercial Avionics Conventions MIT ICAT X-45 Primary Flight Display (PFD) Analogue vs Digital Indications Color Conventions Readability Hidden Info MIT ICAT Commercial B-777 Primary Flight Display (PFD) MIT ICAT Quiet Dark Philosophy ? Reduction of Clutter ? No indications for “normal” ? No “ON” indicators ? No indications for “do nothing” ? Indicate limits, not normal range Elements of quiet dark … So once the procedure for this failure is taken care of …. When the gear is safely up and locked … When the flaps are up … Only the engine indications remain. Maybe in the future we can eliminate most of them as well. MIT ICAT Mode Awareness Cognitive Models Operator Directed Process MIT ICAT Example: Flight Automation y Mode Awareness is becoming a serious issues in Complex Automation Systems ? automation executes an unexpected action (commission), or fails to execute an action (omission) that is anticipated or expected by one or more of the pilots y Multiple accidents and incidents ? Strasbourg A320 crash: incorrect vertical mode selection ? Orly A310 violent pitchup: flap overspeed ? B757 speed violations: early leveloff conditions y Pilot needs to ? Identify current state of automation ? Understand implications of current state ? Predict future states of automation Reference: Aviation Week & Space Technology. McGraw- Hill, January 30, 1995. MIT ICAT MIT ICAT MIT ICAT MIT ICAT MIT ICAT MIT ICAT MIT ICAT MIT ICAT MIT ICAT MIT ICAT MIT ICAT