International Transportation (68) 1 | 2016 36 Emotion-sensitive automation of air traic control Adapting air traic control automation to user emotions ATC, emotions, visualisation, HMIs, automation A human being is more lexible and adaptive than any technology. Nevertheless, the right support systems at the right time can bring large beneits. But how can we know when someone could beneit from technological support? The StayCentered project is working on the idea of collecting physiological data to assess the mental state of an air traic controller. Information about the controller’s mental state would allow for adaptive assistance and additional measures in cases where work overload is anticipated. Such measures could, for example, limit the number of aircraft in the airspace being controlled or provide relief using adjacent sectors. Authors: Jörg Buxbaum, Nicholas Hugo Müller, Peter Ohler, Linda Pfeifer, Paul Rosenthal, Georg Valtin A bout 28,000 commercial lights are conducted in Europe every day, and air traic controllers are responsible for providing separation between these lights. Air traic controllers work in control towers or area control centres. They give instructions to light crews via radiotelephony to ensure safe lights. While tower controllers generally monitor and coordinate aircraft in the vicinity of the airport with which they have visual contact, area and approach controllers rely on the radar information displayed on their screens (Figure 1). In the past decades there has been a signiicant increase in the degree of automation used in air traic control, especially at area control centres. The demands placed on air traic controllers have been reduced due to a wide range of technology. This includes alerting systems, more reliable predictions of future light trajectories, the iltered display of only the targets relevant for a certain working position, and the implementation of ergonomic colour concepts to increase awareness. This reduction of workload has led to an increase in airspace capacity and productivity - meaning the number of light hours that are handled in an air traic controller hour. Increasing the degree of automation at controller working positions remains irmly an R&D topic. The planning of implementation focuses primarily on technical issues. Recording the human workload in situ in this increasingly automated system only plays a subsidiary role. New controller assistance systems will lead to adjustments being made to current workload models, such as adjusting the time controllers will spend in position or on breaks when processing a speciic airspace. However, they will not Tower Bremen. Photo: DFS SCIENCE & RESEARCH Aviation International Transportation (68) 1 | 2016 37 Aviation SCIENCE & RESEARCH lead to changes to the mechanisms used to reduce workload itself. The same measures based on traic igures and subjective parameters will continue to be used to modulate the actual workload of air traic controllers and to reduce their actual workload by coordinating with adjacent sectors. This mechanism is not a closed control loop, particularly because it does not predict the future demands that will be placed on the human operator. As a counterpoint to this direction, a theory has arisen in which the actual cognitive workload of operators can play a growing role when there is a higher degree of automation. Such knowledge could be used, for example, to trigger a change to the distribution of workload between human being and machine to it the situation or to trigger a speciic type of display. To be able to adapt automation to actual or anticipated requirements, the system needs knowledge of the workload state of the air traic controller based on relevant, understandable, and minimally invasive measurable indicators. Even more beneicial would be the ability to predict such states reliably for a relevant period of time. This prerequisite cannot be met at the moment. Today, the team determines the level of the air traic controller’s workload. For en-route control, there are always two air traic controllers simultaneously responsible for a speciic airspace. This means they can assess and evaluate each other. At some control centres, it is possible to enter the workload level directly into the ATM system. This informs adjacent teams so that, if required, they can be asked for active support. This is a well-tested procedure but does not necessarily have to remain unchanged in the future. StayCentered project setup The StayCentered project at the Technische Universität Chemnitz (sponsored by the German Federal Ministry of Education and Research) is pursuing the goal of collecting and comparing the physiological and cognitive workload of air traic controllers using various sensory data. The goal is to create a real-time simulation of the controller workload using these data as the basis. Then recommendations should be made as to which actions should be taken or adaptations of the visual display of the interfaces should be made. The project also aims to provide a capacity forecast for sector planning at DFS control centres. On the basis of air situation displays of traic already handled, recommendations can be made about the need to raise staing levels, or to temporarily reconigure airspaces to provide better control, for example of evening air traic. Furthermore, the two-person team of controllers will be monitored more closely to obtain feedback about what aspects of verbal and non-verbal communication are necessary for the job. Overall, the goal is about optimising working conditions. A real-time simulation environment of area control working positions was chosen for the study (Figure 2). This is located on the premises of the DFS Research and Development Centre in Langen (near Frankfurt, Germany). The simulator is used to validate the research into controller assistance tools (CATO) as part of the SESAR project “Separation Task in En-Route Trajectory based environment” with the involvement of air traic controllers from the DFS Bremen control centre [2]. The project encompasses the development of a controller assistance system that displays conlict-free light levels and headings to increase airspace capacity. The simulated traic scenarios are characterised by a very high traic density that can lead to air traic controllers reaching their workload limits. This efect was one of the decisive requirements of the StayCentered project as the system can provide support for just such situations. Taking measurements in live operations of air traic control was not considered for a number of reasons, including the fact that it could not be ruled out that controllers would be distracted by the measurement instruments. In addition, it is not legal to record radiotelephony for R&D purposes. For these reasons, the tests are carried out with simulated air traic control as previously mentioned. To accomplish this, all the necessary parameters have to be readjusted as precisely as possible. In addition to the air traic controllers being tested, additional controllers and pilots support the tests by simulating traic in the adjacent sectors. The traic scenario was developed by DFS experts and fed into the simulator. Speciic conditions such as especially dense traic situations, emergency situations in the cockpit or lights without light plan can also be added to the simulation. The participants in the experiment are always informed about the measurements to be taken and the general context is explained to them. Furthermore, representative bodies, such as the staf council, were included in the planning of the experiments so that the requirements of staf representation could always be complied with. In addition to measurements taken at the simulator, observation studies are to be carried out during live operations and air traic controllers will be interviewed and surveyed. These methods are being used to attain two main goals. Firstly, factors will be identiied that are diferent during live operations than in the simulator. Secondly, interviews and questionnaires ofer insights into variables that cannot be directly observed but might be stress-inducing for air traic controllers. A pooling of all the methods is used to analyse the existing systems as well as to estimate the potential of the planned adaptive user interfaces. Recording relevant parameters To keep records of the cognitive workload of air traic controllers, the StayCentered project pursues a comprehensive approach to acquiring data in the simulations. A stereoscopic camera is used to record the posture and movements of the controllers’ upper body in three dimensions. This makes it possible to diferentiate between relaxed postures and highly concentrated ones. It is also used to determine in which situations controllers use verbal and non-verbal com- Figure 1: Air traic controllers in an area control center Photo: DFS International Transportation (68) 1 | 2016 38 SCIENCE & RESEARCH Aviation munication to get their message across. By monitoring skin conductance, heart rate and skin temperature, the physiological state in relation to the emotional and cognitive state can be constantly measured. Conclusions can then be drawn about particularly demanding traic situations as well as phases of underload. In the same way, by recording eye movements, elements are identiied that could potentially lead to a critical situation. The dilation of pupils is also an indication of concrete cognitive overload. This is possible due to a pair of eyeglass frames equipped with an integrated frontal camera that records the ield of vision as well as an infrared camera inside the frame to record the movement of the eyes. In addition, both radio communication with the pilots and the spoken communication between the two controllers is recorded using a number of independent microphones. The relevant parameters such as pitch, speed of speaking and other characteristics are correlated to the air traic situation. The movement of the controllers’ facial muscles is also recorded and analysed with the help of the facial action coding system (FACS) to determine the controllers’ emotions. Speciic groups of facial muscles are assigned action units. Speciic combinations of these correlate to diferent basic emotions. All these data are integrated into an overview by means of sensor data fusion. Various methods are applied to compare the data against each other. This makes it possible for more than one sensor to be used to measure a speciic parameter. The additional sensors are intended to conirm the correlation. For example, an increase in skin conductivity may be connected to a change in the duration eyes are ixated on something or to a change in how the person is sitting. The system has many valid indicators that can show that the workload of the controller has increased in this situation. The plausibility of the data supplied by the system is ensured by comparing them with objective traic data (number of aircraft or aircraft movements in the sector) at the point in time in question. This mechanism allows the system to make projections whether or when a critical situation could arise due to an unfavourable combination of controller emotions and air traic in the relevant sector. The system then transmits an alert about the critical situation. To create a model for calculating the emotional state and the cognitive workload, the controllers record their own subjective appraisal of how high the level of their workload is, how much pressure they feel, and how good their overview of the air situation is. Both the objectively measured data of the controllers and the light situation as well as the subjective aspects are used for the creation of a model. This cross-validation increases the reliability of correctly interpreting the data. Preliminary results In addition to the data obtained by the measurements during the experiments, further measurements of concentration and performance diagnostics were carried out. These showed that air traic controllers possess an above-average ability to concentrate and direct their attention. Although these tests were designed and validated to be practically impossible to complete within the allotted time, some of the air traic controllers as test persons were able to complete the tasks. This shows that normal psychological tests are insuicient to ensure valid recording of the actual abilities of air traic controllers. New approaches must be developed as most performance tests are designed to diagnose people with “normal” abilities. In addition to the bodily indicators for stress, other relevant variables were identiied in interviews with active air traic controllers about the general characteristics of their job. The major stressors for air traic controllers are high traic volume, particularly with a high number of vertical movements, unexpected events such as aircraft without light plans entering their sector, or failures and errors of the functionalities of equipment despite good back-up systems. In addition, long periods of absence from the job or other personal factors negatively inluence the perception of work overload/ performance of controllers. When it comes to user interfaces, the display of information about the workload of the air traic controller is very important. This concerns information about the workload situation of controllers in adjacent sectors when high traic levels make it necessary to shift traic to these sectors. However, it also concerns the prediction about the controller’s own situation as this can be the basis for decisions such as issuing pilots direct routings. Further adaptations in the presentation of information about the controller’s situation are necessary as not every piece of information is equally relevant for every situation. Well-designed user interfaces take into account and relect the social environment in which they are implemented. They also take into account the existing mental mod- Figure 2: Real-time simulation environment of area control working positions at DFS Research and Development Centre in Langen Photo: DFS International Transportation (68) 1 | 2016 39 Aviation Science & ReSeaRch els of the users, and support their procedures. As cooperation within the team of controllers is essential for air traic control, all the interfaces the controller uses must also support this cooperation by making actions transparent. In terms of the controller’s mental processes, it was determined that the mental representation of the light situation is not necessarily three-dimensional. As a result, the two-dimensional display has become the preferred way of displaying the air situation. Special attention is paid to altitude information as this has been called the most important piece of information for controllers. For further discussion of the initial results concerning working methods of air traic controllers and the implications of these for the emotion model and user interfaces see [1]. Outlook The project’s long-term potential extends far beyond the speciic scenario of air traic control. In fact, it lays a cornerstone for the hypothetical use of assistance systems in many ields where the human factor could potentially be the cause of devastating safety problems. This includes professions such as pilots, train drivers, and even safety staf in nuclear power plants. Although there are detailed provisions and procedures designed to provide maximum safety in these ields, just as there are in air traic control, not all possibilities can be accounted for, as the human factor is a volatile variable. Assistance systems can provide support when the technology is limited to a minimally invasive level that will not inluence actual work processes. In the coming years, solutions can be expected due to the rapid development of technology, particularly in the ield of wearable devices that collect physiological data. The major advantage here is that such a system is based on objective indicators, whereas human interpretations regarding one’s own cognitive and emotional state are always subjective and, consequently, can be distorted under demanding conditions. An assistance system should be free of such problems such that bad decisions are prevented and critical situations avoided. The real-time collection of video, audio and physiological data of staf raises the question as to how these data should be handled, how to prevent their misuse, as well as data protection in general. The related challenges that need to be met seem small when compared to the beneits such systems could bring. ■ Literature [1] L. Pfeifer, G. Valtin, N. H. Müller, and P. Rosenthal, “Aircraft in Your Head: How Air Traic Controllers Mentally Organize Air Traic,” in HUSO 2015 - The First International Conference on Human and Social Analytics, 2015, pp. 19-24. [2] S. Herr, M. Poppe, KS. Herr, M. Poppe, K. Reinhardt, and G. Achatz, “Erste Validierungsergebnisse der Controller Assistance Tools an der iCAS-SESAR- Plattform”, in “Innovation im Fokus 02/ 2015”, DFS, 2015 Paul Rosenthal, Jun.-Prof. Dr. Assistant Professor for Visual Computing, Visual Computing Laboratory, Technische Universität Chemnitz paul.rosenthal@ informatik.tu-chemnitz.de Georg Valtin Research Associate, Institute for Media Research, Technische Universität Chemnitz georg.valtin@phil.tu-chemnitz.de Linda Pfeifer Research Associate, Visual Computing Laboratory, Technische Universität Chemnitz linda.pfeifer@ informatik.tu-chemnitz.de Peter Ohler, Prof. Dr. Professor for Media Psychology, Institute for Media Research, Technische Universität Chemnitz peter.ohler@phil.tu-chemnitz.de nicholas hugo Müller, Dr. habil. Postdoctoral Researcher, Institute for Media Research, Technische Universität Chemnitz nicholas.mueller@ phil.tu-chemnitz.de Jörg Buxbaum Head R&D Team Air Traic Management, DFS Deutsche Flugsicherung GmbH, Langen joerg.buxbaum@dfs.de Branchenübergreifende Informationen zur aktiven Gestaltung der Stadt von morgen Ein Magazin von T RIALOG P UBLISHERS  Online-Wissensplattform  Newsletter  Fachmagazin als E-Paper und Print-Ausgabe Das neue Medium für Fach- und Führungskräfte w w w . t r a n s f o r m i n g c i t i e s . d e URBANE S YS TEME IM WANDEL