Software SCIENCE & RESEARCH International Transportation (71) 1 | 2019 29 Dominion A realtime middleware for connecting functions in-highly automated vehicles Middleware, Traffic, Automotive, Vehicle simulator, Automated vehicle The Institute for Transportation Systems (TS) at the German Aerospace Center (DLR) develops Dominion as the connecting software for all its automotive and some other research platforms. Dominion’s development began more than ten years ago. Since then, changing research topics required to increase Dominion’s flexibility to meet current and future projects’ demands. This article describes Dominion’s basic features and how we updated and extended them to keep Dominion a usable research tool for the future. Björn Hendriks, Christian Harms, Michael Kürschner T he Institute of Transportation Systems (TS) at German Aerospace Center (DLR) researches a broad range of topics for intelligent mobility. For automotive-based research the institute runs several research platforms to conduct studies with real or simulated vehicles (see figure 1 and figure 2) in different traffic scenarios. Currently, the simulation platforms are extended to include testands riding a bicycle or participating as pedestrian. The simulators as well as the real vehicles contain a broad range of software modules to read the data of sensors, control the actuators, or interact with the testands. Additionally, there are software modules for data processing, for example, to detect relevant objects in camera views or laser scanners, to create the output for advanced driver-assistance systems, or to implement vehicle automation. Many of these software modules were developed by researchers according to the requirements of their respective project demands. To connect all these software modules TS has developed the middleware Dominion as an abstract platform for the data exchange of these software modules [1, 2]. Its main objectives are • real-time capability, • distributable over multiple hosts with different operating systems (Linux, Windows, ...), • encapsulating research platform complexity, • easy to adapt to project-specific requirements. In Dominion’s terminology software modules running on Dominion are called Dominion Application. Examples for Dominion Applications are • read steering wheel or pedal positions, • display outputs to the driver (speedometer, signal lamps, …), • compute driving dynamics, • read driver state sensors (heart rate, eye tracker, …), • set acc-/ deceleration of an automated vehicle, • compute assistance system results, • read state of traffic lights, or • compute state of simulated traffic lights. The last two examples illustrate the separation of abstract functionality from the concrete research platform executing it. The abstract function “get traffic light state” is independent of the source of the traffic light state on the concrete platform. There is one Dominion Application implementing the functionality in a driving simulator and another Dominion Application implementing it on a real vehicle. All of these Dominion Applications produce the same output that is then provided to other Dominion Applications regardless of its concrete source, thus all Dominion Applications consuming the traffic light state can be the same on all platforms. Often driving studies need to alter specific aspects of the research platform while the remaining aspects of driving should work as usual. For example, the investigation of a new advanced driver-assistance system in a driving study may require to process already existing vehicle data and display the result on the dashboard. Another example is a study to investigate a new physiological sensor to detect driver tiredness. Because Dominion Applications usually control only a single aspect of the research platform the adaptations required for such studies can often be limited to add or adapt only a few Figure 1: Dynamic simulator at TS © DLR SCIENCE & RESEARCH Software International Transportation (71) 1 | 2019 30 Dominion Applications. That modularity enables researchers to develop project-specific functions as a Dominion Application without the need to know much about the technical details of the rest of the research platform. Instead, they only need to know how to produce or consume their project-specific data. This is an important precondition for an efficient research with our complex platforms. Dominion’s first major version was finished more than ten years ago, but since then, changes in research topics constantly created new requirements for Dominion. The next section describes Dominion’s basic architecture while the following sections explain how some of the new demands were met. Dominion’s architecture Dominion is split into a general part called Dominion Environment and an installation-specific part called Dominion User. Dominion User contains the Dominion Applications and all data structures they exchange. Dominion Environment contains the general libraries to execute the data exchange as well as functions and classes for typical tasks of Dominion Applications and several supplementary tools. All Dominion software (except for some supplementary tools) is implemented in C++, which allows to develop efficient code. The separation of Dominion Environment and Dominion User allows to apply Dominion Environment as a general middleware in other domains as well. At TS it is used for railway research too. Other DLR institutes apply Dominion Environment on their non-traffic research platforms to achieve a high-performance realtime data exchange. Dominion Data Core At the heart of Dominion User is a central data model called Dominion Data Core. Dominion Data Core contains a nested tree of data structures with all data eventually required by one of the Dominion Applications. At TS we use a joint Dominion Data Core for all automotive Dominion Applications. Having the same Dominion Data Core in the automotive simulators and the real vehicles makes transitions of Dominion Applications between these platforms easy. Dominion installations in other domains have their own Dominion User space and thus their own Dominion Data Core. The central part of the Dominion Data Core is a tree of data to pass between the Dominion Applications. The leaves of the tree are numbers of all kinds, characters, or enumerations. These are aggregated into a hierarchy of nested structures. In later versions of Dominion we added unions and ringbuffers beside the structures. A- structure contains all of its elements simultaneously, while unions only contain one element at a time. The Dominion Applications exchanging union data must provide a mechanism to select the active union element at runtime, because this selection cannot be defined generically. Ringbuffers are for cases where the amount of data produced by a Dominion Application per call cycle is not fixed. They provide a buffer which becomes automatically overwritten when it is full. All levels of the tree can have a multiplicity such that they are arrays of numbers, structures, etc. Besides the data tree, the Dominion Data Core contains meta info for all Dominion Applications and other components, particularly which part of the data tree a Dominion Application reads as input or writes as output. Dominion Applications only have access to their specified inputs and outputs. Code generation A main feature of Dominion Environment is the code generation for the Dominion Applications, which is based on the Eclipse Modeling Framework (EMF) [3]. EMF is an implementation of the Meta Object Facility (MOF) of the Object Management Group (OMG) specification consortium (see [4] and its referenced special MOF standards). The Dominion Data Core is implemented as an EMF Ecore model. A model-to-model transformation implemented in QVTo (Query/ View/ Transformation Operational Mapping, part of the MOF standards) transforms it to another Ecore model (application model), which has a structure oriented towards code generation demands. The application model is used as input to many code generation templates implemented in Acceleo [5]. Acceleo is an implementation of the OMG MTL standard [6] with some extensions. The Acceleo templates generate C++ code for each selected Dominion Application. Besides other functionality, the generated code contains a few empty interface functions to be implemented for the specific Dominion Application. In addition, it contains the trees of the Dominion Application’s input and output as nested C++ structs, such that input and output elements can be accessed as C++ variables. The generated code takes care to call the interface functions, particularly it calls the so-called run function regularly within a configured time frame. Figure 2: Research vehicles at TS © DLR Software SCIENCE & RESEARCH International Transportation (71) 1 | 2019 31 Data exchange All Dominion Applications running on the same host exchange their data by using a shared memory. To organize the shared memory each Dominion Application registers on initialization its specific inputs and outputs by sending their structure to the so-called Dominion Server. The Dominion Server merges the structures received from all Dominion Applications, maps them to addresses in the shared memory, and returns the respective addresses to the Dominion Applications. The generated code of each Dominion Application takes care to copy the relevant parts of the shared memory to its internal input C++ struct, call the Dominion Application-specific code, and finally copy the output back to the assigned shared memory addresses. The access of the shared memory is protected by a global mutex such that only one Dominion Application at a time can access the shared memory. Note, that the Dominion Server only organizes the data exchange in the initialization phase; once all Dominion Applications are initialized they communicate directly without the Dominion Server. If Dominion Applications run on different hosts the generated code takes care to synchronize the relevant parts of the shared memories on the hosts by sending UDP or TCP broadcasts over LAN. With a growing number of hosts the overhead of broadcasts sending Dominion Data Core updates unconditionally to every host limits the data throughput. To improve Dominion’s scalability, we are currently adding unicasts and multicasts to send data only to hosts which need it. Figure 3 shows an example of a possible data flow through Dominion based on an AGLOSA functionality [7]. A sensor, in this example a camera viewing a traffic light, is read by a special Dominion Application (Camera App). The Dominion Application extracts the traffic light state from the view and writes it into the shared memory of the host it runs on. Alternatively, the traffic light could be equipped with a V2X (vehicle-to-everything) sender read by a matching V2X Dominion Application [8]. Next, a Dominion Application (Processing App) running on the same host reads the data from the shared memory to compute a speed recommendation depending on the traffic light state. It stores the result in the shared memory and simultaneously sends it to the shared memory of Host 2, because it knows from the Dominion Server that on Host 2 another Dominion Application runs (Speed-Ctrl App) consuming the speed recommendation. On Host 1 the Dashboard Dominion Application reads the recommended speed from the shared memory and sends it to the vehicle’s dashboard. At the same time on Host 2 the Speed-Ctrl Dominion Application reads the recommended speed from the shared memory of Host 2 and sends it to the speed control of an automated vehicle. The described example shows that even a simple detection and distribution of traffic light states creates complex data dissemination requests, for which Dominion is optimized considering the complexity of different types of research platforms. Additionally, Dominion’s architecture based on shared memory access achieves a high-performance data exchange within a host, which is especially a requirement for highly automated vehicles. Tests on a workstation have shown that two Dominion Applications could exchange data via shared memory at a rate of 10 kHz. Connecting Dominion to other middlewares Automotive research evolves more and more to technical cooperation among vehicles and infrastructure, particularly for automated vehicles. Often cooperation experiments in this field are conducted together with external research groups who have their own research platforms. These need to cooperate with our research platforms running Dominion, such that data will be exchanged Figure 3: Example for a possible Dominion data flow. The numbers denote the order of data flow. © DLR SCIENCE & RESEARCH Software International Transportation (71) 1 | 2019 32 between Dominion and the middlewares deployed on the platforms run by the partners. Another reason to connect Dominion to other middlewares are software modules available for those. To integrate such an external software module it is often simpler to exchange Dominion’s data with the middleware belonging to the requested module than to integrate that module into Dominion. Examples for middlewares Dominion needs to connect to are Robot Operating System (ROS, see www.ros. org), AUTomotive Open System ARchitecture (AUTO- SAR, see www.autosar.org), MQ Telemetry Transport (MQTT, see www.mqtt.org), and Scalable service-Oriented MiddlewarE over IP (SOME/ IP, see www.some-ip. com). Until now, in such cases we have developed a specific communication Dominion Application for each individual project’s data-exchange requirements. But that required to adapt each of those specific Dominion Applications when there was a change of a concerned part of the data tree in the Dominion Data Core. To avoid that in the future, we are currently adding a new level of code generation. Each Dominion Application can have individual text generation templates (appspecific templates) implemented in Acceleo. There are two categories of such templates either for source files or for non-source files. Source file templates must generate valid C++ code, which will be automatically compiled and linked to its Dominion Application by Dominion Environment. Generated non-source files can contain arbitrary text, for example configuration data for an external tool or generated documentation. They will not be further processed by Dominion Environment. App-specific templates have access to the whole Dominion Data Core including the whole data tree as well as to the meta data of all Dominion Applications. To make navigation through the Dominion Data Core model easier, a QVTo mapping transforms the Dominion Data Core model to a special code-generation Ecore model (see figure 4). Opposed to the Dominion Data Core model the code generation model contains back references, for example, from the data entries to the Dominion Applications producing or consuming the data, so it can be navigated directly from each model element to all related model elements. The app-specific code generation enables the development of generic Dominion Applications to exchange data between Dominion and another middleware. The code to synchronize the Dominion Data Core tree with the other middleware can be generated as well as all necessary configuration files. Build system During the long development of Dominion its build system has evolved to a mixture of different build tools. The code generation (including model to model transformation) was handled by Java’s build tool Ant with the extensions Ant-Contrib and Antelope and some self-developed Dominion DataElement Application Library Entry Structure RingBuffer ModelElement Container CodeElement EntryInteger EntryFloat Union [0..1] parent [0..*] dataElements [0..*] outputs [0..*] producers [0..*] inputs [0..*] consumers [0..*] dataElements [0..*] codeElements [0..*] dependencies [0..*] dependencies Figure 4: Simplified code generation Ecore model © DLR Software SCIENCE & RESEARCH International Transportation (71) 1 | 2019 33 Ant tasks to call the code generation tools. On Linux Ant additionally handled the compilation of the C++ code with the Ant extension cpptasks. On Windows the code generation created Visual Studio project files which included a hierarchy of Visual Studio property files. On both platforms a Dominion Application developer had to build the Dominion Environment first and then each Dominion Application that was needed. Eventually additional libraries had to be built in between. This build system had several difficulties: • A Dominion Application developer needed to build the Dominion Environment, the Dominion Applications, and all other required components individually in the correct order. • Ant is mainly designed to compile Java code. Its extension cpptasks to compile C++ has some limitations. • Many build configurations have to be made in Ant and Visual Studio files in parallel with the risk to implement them slightly different or to fix OS-independent bugs only for either Linux or Windows. To address all these difficulties and refactor the build system we have reimplemented the build process with the modern build tool CMake (see www.cmake.org). CMake natively supports C++ but is flexible enough to integrate other compilers as well. CMake is a meta build system generating native build files for a variety of platforms and compilers, including GNU Makefiles for GCC on Linux as well as Visual Studio project and solution files on Windows. Thus the build code in CMake is in general the same for all platforms and compilers, although it is not always avoidable to have CMake code sections for a special compiler or platform (for example there are some special compiler flags set for Visual C++ or some code to install required DLLs on Windows next to the executables). To mitigate potential problems arising from the special sections we keep them as short as possible by applying CMake’s platformand compileragnostic commands and variables wherever possible. As a side benefit we can now compile Dominion on new platforms, like, for example, the ARM-based Raspberry Pi, without the need to do any additional adaptations. This setup reduces the compilation of Dominion Environment and all required Dominion Applications to a single call of CMake followed by a call of the installation-specific compiler. This also includes the building of the integrated documentation. So we do not have to train each Dominion Application developer to know the details of the build process any more. Currently, we are working on the automatic integration of commonly used external libraries by implementing CMake modules to build and link those generically. Summary and outlook Changing research interests over time create changing demands for the features of Dominion as the middleware of the research platforms at TS. We have shown how we met the demand to interact with other middlewares. Next, we have shown how refactoring the build system solved many difficulties accumulated in Dominion’s long-term development and made it fit for the future by increasing usability and manageability as well as integrating new platforms, compilers, and development environments. Besides these, we constantly improve Dominion in many more aspects. Currently we plan to compare Dominion’s abilities to other available middlewares to find out more about its strengths and weaknesses. Based on the results we will evaluate the effort of further Dominion development compared to replacing it with another middleware. But we are confident that at least Dominion’s shared memory communication is doing well compared to the communication methods of other middlewares, particlularly that Dominion is the right tool for the high real-time demands of automated driving. Besides, replacing Dominion would cause a lot of extra work to adapt all the Dominion Applications currently in use, so that an alternative middleware must at least compensate for that with its benefits. Overall, keeping Dominion is currently the most promising alternative and the comparison with other middlewares will guide Dominion’s future development direction towards a more beneficial research tool. ■ REFERENCES [1] Jan Gacnik, Oliver Häger, and Marco Hannibal. A service-oriented system architecture for the human centered design of intelligent transportation systems. In: European conference on human centred design for intelligent transport systems. Apr. 2008 [2] Jan Gacnik, Oliver Häger, Marco Hannibal, and Frank Köster. Service-oriented architecture for future driver assistance systems. In: Fisita 2008 automotive world congress. Sept.-2008 [3] D. Steinberg, F. Budinsky, and E. Merks. EMF: Eclipse Modeling Framework. Eclipse (Addison-Wesley). Addison-Wesley, 2009. ISBN: 9780321331885 [4] Object Management Group (OMG). Meta-Object Facility (MOF) Specification. https: / / www. omg.org/ spec/ MOF [5] Obeo. Acceleo. https: / / www.eclipse.org/ acceleo [6] Object Management Group (OMG). MOF Model to Text Transformation Language Specification. https: / / www.omg.org/ spec/ MOFM2T [7] Jakob Erdmann. Combining adaptive junction control with simultaneous green-lightoptimal-speed-advisory. In: Wivec 2013. June 2013 [8] Tobias Frankiewicz and Alexander Burmeister. AIM reference track— test site for V2X communication systems and cooperative ITS services. In: Journal of large-scale research facilities JLSRF 3.A106 (Apr. 2017) Christian Harms Institute of Transportation Systems, German Aerospace Center (DLR), Braunschweig (DE) christian.harms@dlr.de Michael Kürschner Institute of Transportation Systems, German Aerospace Center (DLR), Braunschweig (DE) michael.kuerschner@dlr.de Björn Hendriks, Dr Institute of Transportation Systems, German Aerospace Center (DLR), Braunschweig (DE) bjoern.hendriks@dlr.de