JoVE Journal > Behavior
Summary
Abstract
Introduction
Protocol
Results
Discussion
Disclosures
Acknowledgements
Materials
References
Automatic Translation
Please note that all translations are automatically generated. Click here for the English version.
Behavior

Evaluating Flight Performance and Eye Movement Patterns Using Virtual Reality Flight Simulator

Published: May 19, 2023
doi:
10.3791/65170
, , , , , , ,
1Psychology Department, School of Social Sciences,Tsinghua University, 2Faculty of Psychology,Beijing Normal University, 3National Key Laboratory of Human Factors Engineering,the China Astronaut Research and Training Center, 4Department of Military Medical Psychology,Air Force Medical University

Summary

A new virtual reality flight simulator was built, which enables efficient and low-cost evaluation of flight performance and eye movement patterns. It also provides a high-potential research tool for ergonomics and other research.

Abstract

Efficient and economical performance evaluation of pilots has become critical to the aviation industry. With the development of virtual reality (VR) and the combination of eye-tracking technology, solutions to meet these needs are becoming a reality. Previous studies have explored VR-based flight simulators, focusing mainly on technology validation and flight training. The current study developed a new VR flight simulator to evaluate pilots’ flight performance based on eye movement and flight indicators in a 3D immersive scene. During the experiment, 46 participants were recruited: 23 professional pilots and 23 college students without flight experience. The experiment results showed significant differences in flight performance between participants with and without flight experience, the former being higher than the latter. In contrast, those with flight experience showed more structured and efficient eye-movement patterns. These results of the differentiation of flight performance demonstrate the validity of the current VR flight simulator as a flight performance assessment method. The different eye-movement patterns with flight experience provide the basis for future flight selection. However, this VR-based flight simulator has shortcomings like motion feedback compared to traditional flight simulators. This flight simulator platform is highly flexible except for the apparent low cost. It can meet the diverse needs of researchers (e.g., measuring situation awareness, VR sickness, and workload by adding relevant scales).

Introduction

The European Aviation Safety Agency (2012) categorizes flight simulators as training facilities, flight and navigation program trainers, flight training equipment, and complete flight simulators1. To date, a range of flight simulators is available for training, from low-level tabletop systems to highly complicated motion-based full flight simulators2. The traditional simulator includes a flight dynamics model, a system simulation, a hardware cockpit, an external visualization, and an optional motion simulation3.

These traditional flight simulators have some advantages as effective flight training equipment. However, their cost is high and environmentally unfriendly, as the drive of each system requires substantial electrical energy, especially a full flight simulator, which requires high temperature and high-pressure fluid or air pressure, consumes much power and generates a lot of noise4.

However, a simple desktop simulator system is flexible and low-cost, with lower immersion and fewer interactions than a full flight simulator2. Therefore, it is essential to develop new flight simulators that combine the advantages of desktop systems and full flight simulators (in other words, the flexibility of a tabletop simulation and the immersion and interaction level close to a full flight simulator).

With the development of computer technology, especially virtual reality (VR) technology, a new type of flight simulator based on emerging VR technology is becoming a reality. The VR-based flight simulator is flexible, portable, low-cost, and has fewer space requirements than conventional flight simulators5. Researchers have created flight simulators based on VR technology over the past 20 years6,7,8,9,10,11; however, these VR flight simulators are mainly for flight training, and there are few for pilot selection. Still, with cost reduction and technology enhancement, VR-based simulators are changing and becoming feasible for personal selection. Some studies have used VR-based simulators for personal selection in different domains: Schijven et al.12 selected surgical trainees using a virtual reality simulator. Huang et al.13 developed a psychology selection instrument based on virtual reality technology for air force pilot recruitment. Wojciechowski and Wojtowicz14 assessed a candidate's capabilities as an unmanned aerial vehicle (UAV) pilot based on VR technology. Given that pilot selection is critical for the aviation industry, it is pressing to develop a new VR-based flight simulator focusing on pilot selection, as large-scale pilot selection is susceptible to the cost of the simulator and demands in the portability simulator system.

Eye movements provide cues for a pilot's performance. Different studies have found that the eye-scanning mode distinguishes the performance between expert and novice pilots. By comparing the scanning pattern between experts and novices, experts' efficient and structural eye movement behavior and the inadequate scanning methods of beginners could be differentiated. Several aviation studies have found that pilots' eye-scanning strategy highly relates to the level of expertise15,16,17,18,19,20,21,22,23,24. According to Bellenkes et al.25, the duration of experts' fixations is shorter, and the frequency of their fixations on instruments is higher than that of novices. Almost the same conclusion was drawn by Kasarskis et al.26, who discovered that expert pilots have more fixations combined with shorter durations than novices, suggested that expert pilots have a better visual mode than novices. In another study, Lorenz et al.27 found that experts spend more time looking outside the cockpit than novices. These results have great practical value in the selection of newcomers.

Flight performance assessment is another critical factor for pilot selection. However, the following problems exist in pilot flight performance evaluation: conflicting expert opinions, more selection norms, and a unified selection theory. In the driving field, Horrey et al.28 compared the absolute value of lane departure from the centerline for different experimental conditions to assess driving performance. Back to the aviation domain, the flight quick access recorder (QAR) records all sorts of pilot manipulation parameters, aircraft parameters, environments, and warning information during flight29. More specifically, as the QAR indicators, the pitch angle is the rotation angle around the left and right axes of the aircraft30, and the reference line (or the center reference line) is right in the middle of the red and green lines28; these two flight parameters are used to evaluate the flight performance of participants with or without experience in the current study. These QAR data can be used to evaluate flight performance, yet to our best knowledge, they have seldom been used for personal training and selection in scientific research31,32.

Measurements of eye movement patterns can be used to assess and predict flight performance and guide pilot training and selection. Gerathewohl33 stated that the eye is the most important sensory organ of the pilot, processing 80% of the flight information. Pilots must acquire visual information from instruments in the cockpit and integrate it into a coherent image to manage the flight22. Further, optimal scanning behavior is essential to accomplish better flight performance15. However, no affordable flight simulator currently integrates an eye tracker to facilitate quantitative studies of the relationship between eye movements and flight performance.

The current study developed a new VR flight simulator to assess if participants with flight experience had better flight performance than those without flight experience. The VR flight simulator integrates eye tracking and a flight dynamics system allowing eye-movement pattern analysis and flight performance evaluation. In particular, it is worth mentioning that the VR flight simulator uses a VR eye tracker34, not a glass-like or desktop eye tracker, to analyze the area of interest (AOI)-based eye movement without time-consuming frame counting.

Finally, the present work can lead to an omnibus measurement for pilot selection in the future, from eye scanning path to objective flying performance data. With the help of the virtual flight simulator, the cost of flight selection will be significantly reduced, and the norm of pilots can be formed based on extensive data gathering. The work fills a gap between conventional and desktop simulators for flight selection needs.

Protocol

All methods described here have been approved by the Institutional Review Board (IRB) of Tsinghua University, and informed consent was obtained from all participants. After completion, all participants were paid $12 (or a gift of equal value). 1. Participant selection Recruit participants according to a prior study of power analysis using G*Power software35 (see Table of Materials) to ensure the participant number meets the …

Representative Results

For the current experiment, 23 experts with flight experience and 23 novices without flight experience were chosen. The participants were between 25 and 58 years of age (experts: M = 32.52 years, SD = 7.28 years; novices: M = 29.57 years, SD = 5.74 years). The gender of all participants was male. All the novices were recruited from Tsinghua University (students or faculty), and all the experts were from China Eastern Airlines. Eye movement<b…

Discussion

The current study assessed if participants with flight experience had better flight performance than those without flight experience in a VR-based flight simulator. More importantly, it evaluated whether a more optimized eye movement pattern could be found in these participants with better flight performance. The results have significant differences between participants with and without flight experience in three key flying QAR indicators: pitch angle 1 s before landing, the mean distance to the reference line, a…

Disclosures

The authors have nothing to disclose.

Acknowledgements

The authors are incredibly grateful to Mr. Li Yan for his help in recruiting pilot participants and acknowledge Ms. Bu Lingyun for her work on drawing pictures. The research was supported by the National Natural Science Foundation of China (grant number T2192931, 72071185), the National Brain Project (grant number STI2030-Major Projects2022ZD0208500), the National Key Laboratory Project of Human Factors Engineering (grant number SYFD062003), the National Key Laboratory Project of Human Factors Engineering (grant number 6142222210201), and year 2022 Major Projects of Military Logistic Research Grant and Key Project of Air Force Equipment Comprehensive Research (grant number KJ2022A000415).

Materials

3D engine SDK Epic Games Unreal Engine 4
GameAnalytics Unreal SDK		 This SDK is a powerful yet flexible free analytics tool designed for games.
CPU Intel IntelCore i9 One of the most powerful CPU on the mainstream market.
Eye tracking SDK Tobii Tobii XR SDK This SDK provide device agnostic access to eye tracking data to allow development for headsets from many different hardware vendors and is not limited to devices using Tobii Eye Tracking hardware.
Eye tracking software Developed by the research team A program that tracks the movement of a person's eyes while they are using a virtual reality HMD.
FlySimulator program Developed by the research team A software that simulates flying experiences in a virtual environment, using VR HMD and hand-held controllers.
Graphics card NVIDIA GeForce RTX 3090
10496 NVIDIA CUDA Cores
1.70 GHz Boost Clock  
24 GB Memory Size
GDDR6X Memory Type		 One of the most powerful graphics card on the mainstream market.
Operating system (OS) Microsoft Windows XP An operating system (OS) developed and exclusively distributed by Microsoft Corporation
Replica control panel THRUSTMASTER 2960720 2971004 2962072 2960748 2960769 U.S. Air Force A-10C attack aircraft HOTAS
Replica joystick THRUSTMASTER 2960720 U.S. Air Force A-10C attack aircraft HOTAS
Replica pedal THRUSTMASTER TPR pendular rudder
Replica throttle THRUSTMASTER U.S. Air Force A-10C attack aircraft HOTAS
Screen connected to PC Redmi RMMNT27NF, 27-inch, 1920 X 1080 resolution ratio Screen allows the experimenter to simultaneously view what is happening in the VR HMD
Screen recording software OBS Project OBS Studio Version 28.0 A free and open source software for video recording and live streaming
Statistical power analysis software Open-Source G*power Version 3.1.9.6 A free and user-friendly tool for estimating statistical power and sample size.
Statistical software IBM SPSS Version 24.0 A powerful statistical software platform
Versatile statistics tool GraphPad Software GraphPad Prism Version 9.4.0 A versatile statistics tool purpose-built for scientists-not statisticians
VR app store HTC Corporation VIVE Software 2.0.17.6 / 2.1.17.6 An app store for virtual reality where customers can explore, create, connect, and experience the content they love and need.
VR head-mounted display (HMD) HTC Corporation VIVE Pro Eye A VR headset with precision eye tracking
VR software Steam Steam VR Version 1.23 A tool for experiencing VR content on the hardware
DOWNLOAD MATERIALS LIST

References

  1. Oberhauser, M., Dreyer, D., Braunstingl, R., Koglbauer, I. What’s real about virtual reality flight simulation. Aviation Psychology and Applied Human Factors. 8 (1), 22-34 (2018).
  2. Oberhauser, M., Dreyer, D. A virtual reality flight simulator for human factors engineering. Cognition, Technology & Work. 19 (2-3), 263-277 (2017).
  3. Rolfe, J. M., Staples, K. J. . Flight Simulation. , (1986).
  4. Robinson, A., Mania, K., Perey, P. Flight simulation: Research challenges and user assessments of fidelity. Proceedings of the 2004 ACM SIGGRAPH International Conference on Virtual Reality Continuum and its Applications in Industry. , 261-268 (2004).
  5. Moroney, W. F., Moreney, B. W. Flight Simulation. Handbook of Aviation Human Factors. , 261-268 (1999).
  6. McCarty, W. D., Sheasby, S., Amburn, P., Stytz, M. R., Switzer, C. A virtual cockpit for a distributed interactive simulation. IEEE Computer Graphics and Applications. 14 (1), 49-54 (1994).
  7. Dorr, K. U., Schiefel, J., Kubbat, I. Virtual cockpit simulation for pilot training. In . The Hague, The Netherlands. What is Essential for Virtual Reality Systems to Meet Military Human Performance Goals? RTO human factors and medicine panel (HEM) workshop. , (2001).
  8. Bauer, M., Klingauf, U. Virtual-reality as a future training medium for civilian flight procedure training. AIAA Modeling and Simulation Technologies Conference and Exhibit. , 18-21 (2008).
  9. Yavrucuk, I., Kubali, E., Tarimci, O. A low cost flight simulator using virtual reality tools. IEEE Aerospace and Electronic Systems Magazine. 26 (4), 10-14 (2011).
  10. Aslandere, T., Dreyer, D., Pankratz, F., Schubotz, R. A generic virtual reality flight simulator. Virtuelle und Erweiterte Realität, 11. Workshop der GI-Fachgruppe VR/AR. , 1-13 (2014).
  11. Joyce, R. D., Robinson, S. K. The rapidly reconfigurable research cockpit. AIAA Modeling and Simulation Technologies Conference. , 22-26 (2015).
  12. Schijven, M. P., Jakimowicz, J. J., Carter, F. J. How to select aspirant laparoscopic surgical trainees: Establishing concurrent validity comparing Xitact LS500 index performance scores with standardized psychomotor aptitude test battery scores. The Journal of Surgical Research. 121 (1), 112-119 (2004).
  13. Huang, P., Zhu, X., Liu, X., Xiao, W., Wu, S. . Psychology selecting device for air force pilot recruitment based on virtual reality technology, has industrial personal computer connected with memory, where industrial control computer is connected with image display device. , (2020).
  14. Wojciechowski, P., Wojtowicz, K. Simulator sickness and cybersickness as significant indicators in a primary selection of candidates for FPV drone piloting. 2022 IEEE 9th International Workshop on Metrology for AeroSpace (MetroAeroSpace). , (2022).
  15. Ziv, G. Gaze behavior and visual attention: A review of eye tracking studies in aviation. The International Journal of Aviation Psychology. 26 (3-4), 75-104 (2016).
  16. Lai, M. L., et al. A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educational Research Review. 10, 90-115 (2013).
  17. Robinski, M., Stein, M. Tracking visual scanning techniques in training simulation for helicopter landing. Journal of Eye Movement Research. 6 (2), 1-17 (2013).
  18. Yang, J. H., Kennedy, Q., Sullivan, J., Fricker, R. D. Pilot performance: Assessing how scan patterns & navigational assessments vary by flight expertise. Aviation Space and Environmental Medecine. 84 (2), 116-124 (2013).
  19. Yu, C. S., Wang, E. M. Y., Li, W. C., Braithwaite, G., Greaves, M. Pilots’ visual scan patterns and attention distribution during the pursuit of a dynamic target. Aerospace Medicine and Human Performance. 87 (1), 40-47 (2016).
  20. Haslbeck, A., Zhang, B. I spy with my little eye: Analysis of airline pilots’ gaze patterns in a manual instrument flight scenario. Applied Ergonomics. 63, 62-71 (2017).
  21. Brams, S., et al. Does effective gaze behavior lead to enhanced performance in a complex error-detection cockpit task. PLoS One. 13 (11), e0207439 (2018).
  22. Peißl, S., Wickens, C. D., Baruah, R. Eye-tracking measures in aviation: A selective literature review. The International Journal of Aerospace Psychology. 28 (3-4), 98-112 (2018).
  23. Jin, H., et al. Study on how expert and novice pilots can distribute their visual attention to improve flight performance. IEEE Access. 9, 44757-44769 (2021).
  24. Lounis, C., Peysakhovich, V., Causse, M. Visual scanning strategies in the cockpit are modulated by pilots’ expertise: A flight simulator study. PLoS One. 16 (2), e0247061 (2021).
  25. Bellenkes, A. H., Wickens, C. D., Kramer, A. F. Visual scanning and pilot expertise: The role of attentional flexibility and mental model development. Aviation Space and Environmental. 68 (7), 569-579 (1997).
  26. Kasarskis, P., Stehwien, J., Hickox, J., Aretz, A., Wickens, C. Comparison of expert and novice scan behaviors during VFR flight. Proceedings of the 11th International Symposium on Aviation Psychology. , (2001).
  27. Lorenz, B., et al. Performance, situation awareness, and visual scanning of pilots receiving onboard taxi navigation support during simulated airport surface operation. Human Factors and Aerospace Safety. 6 (2), 135-154 (2006).
  28. Horrey, W. J., Alexander, A. L., Wickens, C. D. Does workload modulate the effects of in-vehicle display location on concurrent driving and side task performance. Driving Simulator Conference North America Proceedings. , (2013).
  29. Wang, L., Ren, Y., Sun, H., Dong, C. A landing operation performance evaluation method and device based on flight data. In Engineering Psychology and Cognitive Ergonomics: Cognition and Design. , 297-305 (2017).
  30. Wang, L., Ren, Y., Wu, C. Effects of flare operation on landing safety: A study based on ANOVA of real flight data. Safety Science. 102, 14-25 (2018).
  31. Huang, R., Sun, H., Wu, C., Wang, C., Lu, B. Estimating eddy dissipation rate with QAR flight big data. Applied Sciences. 9 (23), 5192 (2019).
  32. Wang, L., Zhang, J., Dong, C., Sun, H., Ren, Y. A method of applying flight data to evaluate landing operation performance. Ergonomics. 62 (2), 171-180 (2019).
  33. Gerathewohl, S. J. Leitfaden der Militärischen Flugpsychologie. Verlag für Wehrwissenschaften. , (1987).
  34. Ugwitz, P., Kvarda, O., Juříková, Z., Šašinka, &. #. 2. 6. 8. ;., Tamm, S. Eye-tracking in interactive virtual environments: implementation and evaluation. Applied Sciences. 12 (3), 1027 (2022).
  35. Faul, F., Erdfelder, E., Lang, A. -. G., Buchner, A. G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods. 39 (2), 175-191 (2007).
  36. Boslaugh, S. E. . Snellen Chart. , (2018).
  37. He, J., Becic, E., Lee, Y. -. C., McCarley, J. S. Mind wandering behind the wheel. Human Factors: The Journal of the Human Factors and Ergonomics Society. 53 (1), 13-21 (2011).
  38. Tanveer Alam, . GitHub – tanvcodes/qar_analytics: Scripts for working with publicly available Quick Access Recorder (QAR) data from a fleet of 35 BAE-146 aircraft. GitHub. , (2022).
  39. Shapiro, S. S., Wilk, M. B. An analysis of variance test for normality (complete samples). Biometrika. 52 (3-4), 591-611 (1965).
  40. Hintze, J. L., Nelson, R. D. Violin plots: A box plot-density trace synergism. The American Statistician. 52 (2), 181-184 (1998).
  41. Cohen, J. . Statistical Power Analysis for the Behavioral Sciences (2nd ed.). , (1988).
  42. Sawilowsky, S. S. New effect size rules of thumb. Journal of Modern Applied Statistical Methods. 8 (2), 26 (2009).
  43. Bateman, T. S., Crant, J. M. The proactive component of organizational behavior: A measure and correlates. Journal of Organizational Behavior. 14 (2), 103-118 (1993).
  44. Endsley, M. R. Measurement of situation awareness in dynamic systems. Human Factors. 37 (1), 65-84 (1995).
  45. Hunter, D. R. Measuring general aviation pilot judgment using a situational judgment technique. The International Journal of Aviation Psychology. 13 (4), 373-386 (2003).
  46. Kim, H. K., Park, J., Choi, Y., Choe, M. Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment. Applied Ergonomics. 69, 66-73 (2018).
  47. Hart, S. G. . NASA Task Load Index (TLX). , (1986).

Tags

Virtual RealityFlight SimulatorEye MovementPilot PerformanceFlight Performance IndicatorsVR Environment SetupFlight Control ModuleScreen RecordingFormal Flight ExperimentsEye Movement DataFlight Performance Data
check_url/65170?article_type=t

Play Video
PDF
DOI
DOWNLOAD MATERIALS LIST
Cite This Article
Ke, L., Zhang, Z., Ma, Y., Xiao, Y., Wu, S., Wang, X., Liu, X., He, J. Evaluating Flight Performance and Eye Movement Patterns Using Virtual Reality Flight Simulator. J. Vis. Exp. (195), e65170, doi:10.3791/65170 (2023).
Download Citation
Reprints and Permissions

View Video

✖

To prove you're not a robot, please enter the text in the image below

CAPTCHA Image
Play CAPTCHA Audio
Refresh Image