BAE Systems is studying future fast jet cockpit technologies for enhanced combat effectiveness through advanced human-machine interfaces and increased automation, as PAUL E EDENreports.
Agile targeting, data fusion and data sharing, helmet-mounted displays and advanced combat systems; these are the stuff of modern fast jet cockpits. And pilots too, of course, although in an era of AI and loyal wingmen, should future cockpit designers be considering human factors and limitations at all?
Nellis AFB, home to the famous Red Flag exercise series is right next door to Las Vegas. If you’ve flown from Vegas to London, you’ll appreciate there are better ways to spend ten hours than sitting in an airline seat. Without consideration for G-forces, cockpit workload or being shot at, now imagine the ten hours from Nellis to the UK strapped into the ejection seat of a Eurofighter Typhoon. No meal service, no snacks, no toilets and inflight movie.
Deployments aside, the nature of global conflict over the past two or three decades has seen fast jet crews spend hours in the cockpit before reaching the area of regard, remain on station for two hours or more, then make the long flight back to base. Future fast jet cockpits must, through necessity, balance combat efficiency with pilot health and living space.
Information overload
Through its work on Tempest, the Future Combat Air System (FCAS), BAE Systems is exploring the gamut of future fast jet cockpit technologies.
Any aircraft emerging from FCAS will generate more data than today’s most advanced fighter – arguably the F-35 – from more powerful sensors. It will likely place greater physiological demands upon its pilots through higher performance and require that they operate within a team comprising crewed and uncrewed assets, perhaps while controlling one or more of the latter.
Casual conversations with senior RAF pilots early in Typhoon’s service revealed the risk of ‘helmet fires’, where pilots were overwhelmed by the volume of data generated by the jet’s sensors while attempting to fly and ‘fight it’ effectively. New systems, better data displays and world-class training have reduced the problem, but could advancing technologies see it reignite?
Jacob Greene, a human factors engineer at BAE Systems, says: “Pilot overload from the availability of increased data and sensor capability is a real threat. Simplifying data provision is a key human machine interface (HMI) design consideration integral to any future cockpit display and control system. Novel cockpit HMI technologies aim to provide greater display real-estate enabling pilots to view data through enhanced helmet mounted display (HMD) systems for heads-up viewing and a large area display (LAD) for increased display functionality and customisation, while offering improved methods of display interaction.
“Next generation aircraft are also likely to include enhanced autonomous capabilities, driven through an intelligent agent. A system capable of determining which information is valuable and how to present it in a manageable form while considering the operational context, should help reduce the burden on the human operator, providing there is trust in the system.”
The concept of increased cockpit autonomy raises important questions. Is autonomy’s role simply to relieve pilot workload? Does it enhance mission capabilities outside the pilot’s ability? Does it offload tasks, leaving pilots with capacity for others? And since autonomy implies AI and AI builds on experience, what are the fundamental starting points?
Regarding this latter question, most fast jet pilots have been male, but pilots who have worked in formations of male and female pilots, or in mixed crews, suggest that considerable benefit is gained from different male and female thought processes, especially in the heat of combat or crisis. And yet the evidence of how fast jets are flown is overwhelmingly from a male perspective – will cockpit AI inevitably ‘think like a man’?
Another of BAE Systems’ human factors engineers, Lucy Crabb responds: “Flexibility is a key factor for successfully incorporating AI into the cockpit. This means that to cater for how different pilots think and react to stimuli around them, we need to give them the facility to personalise the AI to best match their working style – including their AI’s gender. It is important when designing and integrating AI and autonomous systems that there is no negative effect on the dynamic between pilots in a formation. We’re combating this by identifying the key characteristics that make a good human crew and ensuring the AI displays the same characteristics to create an optimum human-machine team.”
Continuing the theme of advanced HMI, Greene notes: “Since a 6th-generation aircraft is likely to have enhanced autonomous capabilities, haptic body cues can provide additional spatial cues to provide the operator with attention-getting and warning indicators. Haptic cues can also be programmed to activate a specific number of sensor combinations at different intensities.”
BAE Systems has used haptic gaming vests in early assessments to determine how these new sensory inputs could enhance pilot capability. Greene continues: “It is well known that inattentional deafness can occur in high stress and overloading environments, leaving pilots unable to hear traditional audio warnings. Body haptic cues could have potential for alerting the pilot to prepare for any sudden changes in g-force, for example, likely used alongside traditional methods in a multi-modal array to enhance reaction and response times rather than adding to workload as a standalone additional input.”
Eye tracking is another technology promising closer human-machine integration. Greene explains: “Eye tracking captures a variety of metrics, including gaze data, pupil dilation and blink rates, in real-time. Their analysis can infer human cognitive, emotional and physiological states, including mental workload and fatigue. Eye tracking can also be deployed as an HMI input method, enabling cursor control through gaze point selection. The use of a dedicated button that could be integrated onto a hands on throttle and stick (HOTAS) system could be used as a simple, reliable method to support eye-based input. Technical advancements are being made to ensure reliable tracking under high ambient lighting conditions, including working outdoors in direct sunlight, which has been a limiting factor.”
Virtual WSO? Loyal wingman?
‘Maverick’ leads a formation of single-seat F/A-18E and two-seat F/A-18F Super Hornets against an ‘illegal uranium enrichment plant’ in the 2022 movie Top Gun: Maverick. The US Navy sees continuing benefits in two-seat combat aircraft, opting for a mixed Super Hornet fleet, where crews of pilot and weapon system officer (WSO) fly the F/A-18F.
However, there is no clear consensus; the French Air and Space Force operates a mix of single- and dual-seat Rafales, for example, while the Royal Air Force flies only single-seat Typhoons and F-35 operators have no choice but to fly with a single pilot.
Work under way in Australia, the UK, US and elsewhere is exploring concepts for mixed formations with autonomous or remotely piloted aircraft flying as ‘loyal wingmen’ alongside crewed fighter aircraft. Meanwhile, Boeing’s T-7A Red Hawk training system includes provision for students and instructors crewing advanced simulators on the ground to ‘fly’ alongside aircraft in the real world and Sikorsky has flown H-60 trials with a human pilot flying alongside a virtual colleague. Is there space in future fast jet cockpits for a virtual back-seater?
“Such a concept could provide important benefits including reducing platform size and weight, and the workload of a single operator,” Crabb believes. “Strong communication channels between operators would be needed for a remote crew partnership to work successfully, along with a clear division of roles to prevent confusion or vital tasks being missed. Effective training and operating procedures would also be vital. The second crew member could be displayed as an avatar that could be personalised by appearance and voice, for example, to enhance interaction.”
The ‘loyal wingman’ concept raises questions over AI, control and increasing cockpit workload. It is easy to overlook the value of discussion between pilots in a tactical formation, each with a unique viewpoint and differing experience. Following the RAF’s acclaimed participation in Operation Unified Protector over Libya in 2011, a senior RAF Tornado pilot told the author that crews regularly discussed how best to engage targets and, crucially, he identified those occasions on which the formation decided it should not release weapons as the most significant. How does a loyal wingman fit into this scenario and how might it be controlled?
Crabb continues: “Generally, control of uncrewed assets from a fast jet cockpit impacts on workload – we’re exploring and trying to address the issue. Thinking about how a pilot interacts with a human wingman, both are autonomous. Given a task or objective, they’ll use their training and experience to get the job done. Realistically, we need to achieve that kind of interaction with machine systems too, and even more so when the number of cooperating assets increases. Pilots simply won’t have the capacity to be involved in the detailed operation of those assets, so they’ll need to interact in a more goal orientated manner, supplying tasks and providing oversight as a flight lead might. The implication is that any loyal wingman assets would need to facilitate that level of capability in their technology too, so we must generate human-centred requirements for these systems if they’re going to be useful and useable.
Working from home?
Crabb’s vision for advanced human-machine interaction may be some way off, but the reality of controlling a loyal wingman remotely, perhaps with a pilot sitting in a ‘cockpit’ at home base, is far closer. Crabb says that may have implications for fundamental aspects of cockpit design and pilot role. “We envisage the pilot’s role developing as technology and the future battlespace continue to develop. If this leads to remotely piloted wingmen, these could be operated from ground control stations, allowing human pilots to operate the platform at a safer distance from the battlespace and in a more comfortable environment unaffected by the environmental or physiological constraints that normally impact operations, including G-forces and cockpit size.”
For now, though, pilots will remain in fast jet cockpits and BAE Systems is paying considerable attention to pilot welfare, for combat efficiency and more prosaic matters. Returning to the flight from Nellis, fast jet pilots have traditionally employed ‘tactical dehydration’ before long flights, yet dehydration has deleterious effects on performance. Greene explains: “Depending on the operational environment, the cockpit can reach temperatures that cause pilots to lose water and electrolytes excreted through sweat. Additional factors including limited aircrew relief systems and cockpit storage discourage the on take of fluids before and during flights, and even minor levels of dehydration can have negative effects on physical and mental performance.”
Measuring and understanding such effects is critical and Greene continues: “As human factors engineers, we work closely with the RAF Centre of Aviation Medicine, keeping informed on the latest findings regarding solutions for the reliable and robust measurement of pilot status. Separately, we’re exploring the possibility of monitoring and classifying human performance metrics including mental workload, fatigue and vigilance [which can be defined as sustained concentration] through psychophysiological monitoring. Wearable technologies, eye tracking, electrocardiograms and electroencephalograms among them, are being explored as possible solutions for capturing insights into pilot behaviour. Research is under way to understand how these technologies stand up during airborne testing and further our understanding of how susceptible the data is to noise and artefacts during changes in g and under high vibration.”
Future pilot
A WW1 fighter pilot travelling through time from a Sopwith Camel cockpit to a Eurofighter Typhoon ‘office’ would notice fundamental similarities in layout and controls. But would the same be true if he travelled into the future?
Greene concludes: “The ability to effectively process the vast amount of information available in the modern battlespace will be essential to maintaining combat advantage, while human performance is often the limiting factor to overall platform performance. Cockpit research is exploring ways to alleviate this concern.
“Understanding what flying controls remain necessary when automation is increased is a fundamental question. HOTAS will remain key for HMI functionality, although novel features are being explored to understand its benefits on a new platform. The multi-modal cockpit is another important consideration for a new platform, including an augmented reality HMD for increased display real-estate in the outside world. Virtual displays also enable improved heads-out viewing and increased display area, reducing the reliance on traditional glass cockpit displays. Combining these with eye tracking, 3D audio, body haptics and other wearable technology will improve situational awareness and speed up interaction time with the HMI, while tried and tested functionalities – pressing a HOTAS button – will ensure robust, reliable functionality.”
Our Camel pilot would not feel at home, but in an era of AI and loyal wingmen, BAE Systems is working to serve fast jet pilots better than ever before.
Paul E Eden
18 August 2023
