International Transportation (71) 1 | 2019 18 PRODUCTS & SOLUTIONS Maintenance Mobile system for road inspection and 3D modelling Introducing novel technology within the project “Digital-Roads New Zealand” Road inspection, Sensor systems, Optical navigation, Computer vision, 3D modelling of roadsides Regular inspections and the maintenance of roads support traffic safety. Inspection technologies may benefit from latest developments in sensor systems, camera technology and computer vision. The paper discusses the application of novel mobile technologies, including stereo vision and visual odometry, for modelling and analyzing extensive segments of roads. Applications of the developed system have been evaluated at test sites in New Zealand within an international collaboration project entitled “Digital Roads New Zealand”. Sergey Zuev, Anko Börner, Hongmou Zhang, Ines Ernst, Martin Knoche, Reinhard Klette T he inspection of road infrastructure or of road surfaces requires exact localization of detected road defects on a 2D map or in a 3D model. Developments in advanced driver assistance [1] or autonomous driving [2] depend on accurate 3D road and roadside models, for example for sensor testing or environment-based vehicle guidance. Such models can be generated by different sensor technologies which can be placed on various platforms. Lidar, radar or camera systems mounted on satellites, airplanes, UAV’s or cars gather data which are mapped automatically into environment models. These models can be used for information, assistance and control systems. Model parameters (e.g. online vs. offline generation, spatial resolution, coverage) depend on the given application. The research project “Digital Roads New Zealand” has been defined for meeting and understanding such needs in the context of developing a transport technology test site near Whangarei (New Zealand). This paper reports about current research in this project which is undertaken by three partners. The Institute for Optical Sensor Systems [3] at the German Aerospace Centre (DLR) developed an integrated positioning system (IPS) which was installed on a car for providing precise trajectories of the vehicle [4]. Data were recorded by a stereo vision system and an inertial measurement unit (IMU). Additionally, the data was used for modelling sections of the road from 3D point clouds generated by the IPS. The Centre for Robotics & Vision (CeRV) at Auckland University of Technology contributes to computer vision solutions and dynamic visualization [5]. The Northland Transport Technology Testbed (N3T) is developing road safety solutions for modern trucks [6] on rural roads and is preparing for driver assisting systems. In this paper, these three partners demonstrate the analysis of data collected with the IPS system during a recent measurement campaign under real-world conditions in Northland, New Zealand. Case studies have been defined for urban environments (town basin of Whangarei), a new subdivision (Marsden City) and a rural road (Otaika Valley Road). In this project we were able to perform large-scale 3D roadside modelling (for networks of roads or for 10 to 15 km long road segments) and also to demonstrate opportunities for road inspection. Integrated Positioning System The IPS is a low-cost stereo-vision-aided inertial navigation system (figure 1, left) that has been developed at DLR [7]. The IPS measures seamlessly a motion trajectory in unknown indoor or outdoor environment without any previous knowledge of the environment. Based on the measured trajectory and on computer-vision methods, following applications like 3D scene reconstruction, map-building and object localization can be achieved. The IPS consists of a stereo-camera system (Prosilica GC1380H) with a 45 cm base line and an inertial measurement unit IMU (ADIS-16488). The cameras can record Figure 1: IPS system installed on the car (left). Measurement vehicle (middle). Oncoming truck traffic during measurement campaign on rural road (right). All figures: DLR International Transportation (71) 1 | 2019 19 Maintenance PRODUCTS & SOLUTIONS CCD-progressive 1,360 x 1,024 stereo images at up to 30 frames per second (here used at 10 fps), with a focal length of 8.2 mm. For the IMU, both gyroscope and accelerometer have a bandwidth of 330 Hz, with a bias stability of 6.25 °/ h and 0.1 mg, respectively. The system includes also a low cost GNSS sensor. For more details about the IPS see [8] and [9]. Trajectory estimation The trajectory of the moving IPS is calculated by fusing IMU measurements with visual odometry, i.e. by using the recorded stereo camera data for detecting pose changes over time. The IMU used is a robust, self-contained sensor which means that the measurement does not relay on any (additional) outside information. The high sampling frequency of the IMU allows high spatial resolution when it comes to road defect detection. The IMU outputs six degrees of freedom motion information between two subsequent IMU frames, which includes acceleration and angular velocity. The estimated motion trajectory of the system, based on IMU integration only, is erroneous due to random walk noise in the measurements; this can cause that the error of integrated IMU measurements grows unboundly. In practical applications, an IMU is needed to be combined with other sensors (like cameras) to form a complementary system. A camera system (or visual odometry) also has its challenges. Solely camera-based practical applications are often limited by low resolution, texture issues, and sensitivity with respect to environment illumination. Visual odometry (VO) alone cannot work accurately in highly dynamic working environments (e.g. caused by incorrect feature matching due to motion-blurred images). Therefore, it is common to combine cameras with IMU aiming at a significant improvement of performance by an integrated system. In the IPS system, pose measurements from VO and from the IMU system are fused by using an error-state Kalman filter [10]. The output of the Kalman filter is finally considered as being the motion trajectory of the IPS system, and is then used in our point cloud generation step. In the IPS system, the IMU operates at a very high frequency and is able to reflect the state of the system. Therefore, the IMU measurements (strapdown) are considered as the basic reference of the system state. In addition to the IMU, other sensors synchronously measure the motion of the system but in different coordinates. To fuse the measurements of different sensors, all measurements are transformed into global Earth-centered, Earth-fixed (ECEF) coordinates as show in figure 2. Next, by taking differences between VO and IMU, or GPS and IMU as input, the error state of the system can be updated by the error-state Kalman filter. Finally, by adding the filtered error state to the strapdown output, the optimized system pose is obtained. At the end, the coordinates of the system pose is transformed back into geodetic Latitude, Longitude, Altitude (LLA) coordinates as the final moving trajectory of the system. 3D point cloud generation The IPS with its time-synchronized stereo camera system records two images at exactly the same time. They are used for reliable VO based on sparse sets of 3D points; they also serve for the generation of dense depth maps and resulting local 3D point sets for each recorded image pair. Numerous 3D point sets from image pair sequences can be fused using the calculated IPS attitude data. Subsequently, the point clouds from all image pairs can be merged into a high-density cloud and filtered into a voxel grid of an appropriate resolution and size. A large-scale 3D cloud of points for the entire observed area can be generated; see figure 3. Obviously, this approach strongly depends on a very high accuracy of all contributing modules. A deep understanding of all system components and accurate calibration are essential; we developed a chessboard-based robust calibration approach that is described in detail in [11]. For a fixed camera set-up, the depth resolution and local accuracy of 3D points is mostly determined by the stereo camera parameters, especially the pixel resolution and base length, and the minimum distance of the shown objects to the camera while passing. However, the global point cloud accuracy is directly correlated with the trajectory accuracy. In case of low light conditions with dark and especially very noisy images extracting dense depth maps may require some additional effort, as described in [12]. 3D point cloud generation is usually done in a time-consuming post-processing step with several specialized software packages. The IPS system approach allows 3D point generation in real time. Therefore we adapt the frame rate for point cloud generation dynamically to the camera movement. The computationally most expensive algorithm for dense stereo matching is optimized for the execution on a graphics processor unit (GPU), written in the platform and vendor independent programming language OpenCL. Based on former developments [13] and experiences with various cost functions [14] we use a semi-global matching algorithm with a census cost function. All steps, from the trajectory estimation up to the 3D point filtering to a voxel grid, + - VO IMU Strapdown Error State Filter Pose Estimation Pose in ECEF GPS LLA to ECEF Local to ECEF Local to ECEF - ECEF to LLA Pose in LLA Figure 2: Block diagram for IMU, VO and GPS integration International Transportation (71) 1 | 2019 20 PRODUCTS & SOLUTIONS Maintenance can be carried out on a capable laptop PC in real time. This provides the opportunity to generate and possibly view dense 3D point clouds already during an ongoing measurement and for the entire area of interest, e.g. for very long road paths or large-scale areas. In subsequent steps, e.g. after applying semantic segmentation these 3D points clouds can be used to derive 3D models of observed objects and help answering questions raised in dedicated applications. Using multiple measurements (i.e. repeated by driving in opposite directions or regularly the same road) over extended periods of time, occluded areas can be filled-in and long term changes in state parameters can be detected. Best practice: using of IPS technology in New Zealand Within the project “Digital Roads New Zealand”, several specified case studies with different environments using IPS technology have been applied in test areas in Northland, for the town basin of Whangarei, for the subdivision of Marsden City and for several rural roads. In all areas the measurement runs were from 3 to 10 km long. As a first step, the accurate trajectories have been calculated followed by a generation of largescale 3D point clouds of road environments. Figures 3, 5 and 6 show an overview of 3D clouds for dedicated test areas with a maximum resolution of a 1.5 cm grid. This high resolution of 3D information allowed us a detection of even very small details of road infrastructure. These 3D data are geo-referenced, with a position accuracy of several decimeters over the distance of travelling. The conditions of road infrastructure (e.g. bridges, guard railing, traffic road signs) can be automatically evaluated, recorded and monitored for changes. Generated 3D data allowed us to extract road geometry parameters such as the inclination along and across the road, or the state of barriers or of road markings. The 3D point data can be used for road inspection tasks and also as input for driver-assistance systems by describing the 3D road model at current time and place. The rural test area along Otaika Valley Road (OVR) is 12 km long; this is a hilly and windy rural road with high-frequency traffic of logging trucks. This road is a hot spot of traffic accidents, with about 90 crashes during the last 10 years, most of them involving logging trucks. The application of novel technologies for maintenance, modelling of road surfaces and for supporting drivers with assistance systems are important steps towards safer rural roads. Figure 4 shows calculated trajectory and figure 3 illustrates a resulting large-scale 3D point cloud based on just one run of our measuring car (in just one direction on the OVR). The urban area of town basin of Whangarei (of about 2x2 km) represents a typical city environment with high-frequently traffic; it is suitable for testing of driver-assistance systems. It is challenging to develop computer vision algorithms that filter out moving or otherwise disturbing objects while aiming at an accurate 3D model. Figure 5 shows a resulting overall 3D model and details of road infrastructure in the Whangarei town basin. The Marsden City subdivision area, with 5 km roads in total, is designed for future urban development with full road infrastructure components; it still has low traffic utilization prior to further development. This area is used for testing self-driving vehicle technology. Figure 6 shows a calculated trajectory of our measuring car and the overall 3D model. Road defect detection (e.g. potholes) is a subject for road-surface inspection. The ste- Figure 3: Case study “Otaika Valley Road” - examples of created 3D models Figure 4: Case study “Otaika Valley Road” - IPS measured trajectory in Google map Figure 5: Case study “Town Basin Whangarei” - Details of 3D model (left) and IPS original image-(right) International Transportation (71) 1 | 2019 21 Maintenance PRODUCTS & SOLUTIONS reo image data, provided by the IPS, can be used for efficient detection of road surface distress. IPS-based stereo vision [15] provides multi-frame depth data integration into a digital elevation model [16], supporting robust and efficient detection of potholes, and road surface inspection in general. Conclusions By our IPS measurement campaign, part of “Digital Roads NZ” project, we demonstrated a combination of real time trajectory estimation and 3D environment modelling; results are accurate and reliable over long time periods or trajectory lengths, with real time 3D point cloud generation (at 10 fps). This is a promising technological basis for various applications. The presented costeffective approach for data acquisition enables a high repetition rate. It is therefore not only well-suited for road inspection but also for change detection since time series of point clouds are exactly temporally and spatially referenced. ■ REFERENCES [1] R. Klette, “Vision-Based Driver Assistance”, in Wiley Encyclopedia of Electrical and Electronics Engineering, American Cancer Society, 2015, pp. 1-15 [2] K. Bimbraw, “Autonomous Cars: Past, Present and Future - A Review of the Developments in the Last Century”, in 12th International Conference on Informatics in Control, Automation and Robotic, 2015 [3] DLR Institute of Optical Sensors, 2019. [Online] Available: www.dlr. de/ os/ en [4] A. Börner, D. Baumbach, M. Buder, A. Choinowski, I. Ernst, E. Funk, D. Grießbach, A. Schischmanow, J. Wohlfeil and S. Zuev, “IPS - a vision aided navigation system”, Advanced Optical Technologies, pp. 121-129, 2017 [5] CeRV, Centre for Robotic & Visison, 2019. [Online] Available: www. cerv.aut.ac.nz [6] N3T, The Northland Transport Technology testbed, 2019. [Online] Available: http: / / www.n3t.kiwi [7] D. Grießbach, D. Baumbach and S. Zuev, “Stereo-Vision-Aided Inertial Navigation for Unknown Indoor and Outdoor Environments”, in International Conference on Indoor Positioning and Indoor Navigation, 2014 [8] H. Zhang, J. Wohlfeil, D. Grießbach and A. Börner, “Eligible Features Segregation for Real-time Visual Odometry”, in Conf. 3d-Nordost, Berlin, 2017 [9] H. Zhang, D. Grießbach, J. Wohlfeil and A. Börner, “Uncertainty Model for Template Feature Matching”, in The Pacific-Rim Symposium on Image and Video Technology, 2017 [10] H. Zhang, “Optical Navigation for Mobile Platforms Based on Camera Data”, Ph.D thesis, Technical University of Berlin, 2018 [11] J. Wohlfeil, D. Grießbach, I. Ernst, D. Baumbach and D. Dahlke, “Automatic camera system calibration with a chessboard enabling full image coverage”, in ISPRS Annals, in print, 2019 [12] H. Zhang, I. Ernst, S. Zuev, A. Börner, M. Knoche and R. Klette, “Visual Odometry and 3D Point Clouds Under Low-Light Conditions”, in International Conference on Image and Vision Computing New Zealand, Auckland, New Zealand, 2018 [13] H. Hirschmüller, M. Buder and I. Ernst, “Memory Efficient Semi- Global Matching”, in ISPRS Annals, 2012 [14] H. Hirschmüller and D. Scharstein, “Evaluation of Stereo Matching Costs on Images with Radiometric Differences”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, pp. 1582-1599, 9 2009 [15] A. Dhiman, H.-J. Chien, and R. Klette, “Road surface distress detection in disparity space”, in Proc. Int. Conf. Image Vision Computing New Zealand, IEEE online, Christchurch, 2017 [16] A. Dhiman, H.-J. Chien, and R. Klette, “A multi-frame Stereo Visonbased Road Profiling Technique for Distress Analysis”, in Proc. Int. Symposium on Pervasive Systems, Algorithms and Networks, IEEE online, Yichang, China, 2018 Martin Knoche, Dipl.-Phys., MBA CEO Northland Innovation Centre N3T, Whangarei (NZ) martin@n3t.kiwi Reinhard Klette, Prof. Dr. sc. nat. Director Centre for Robotic and Vision (CeRV), Auckland University of Technology, Auckland (NZ) rklette@aut.ac.nz Ines Ernst, Dipl.-Math. Research associate, DLR Institute of Optical Sensor Systems, Berlin (DE) ines.ernst@dlr.de Anko Börner, Dr.-Ing. Head of Department Real Time Data Processing, DLR Institute of Optical Sensor Systems, Berlin (DE) anko.boerner@dlr.de Hongmou Zhang, Dr.-Ing. Research associate, DLR Institute of Optical Sensor Systems, Berlin (DE) hongmou.zhang@dlr.de Sergey Zuev, Dr.-Ing. Research associate, DLR Institute of Optical Sensor Systems, Berlin (DE) sergey.zuev@dlr.de Figure 6: Case study “Marsden City” - IPS measured trajectory in Google map (left) and 3D-model of roads (right) Trialog Publishers Verlagsgesellschaft | Schliffkopfstrasse 22 | D-72270 Baiersbronn Tel.: +49 7449 91386.36 | Fax: +49 7449 91386.37 | office@trialog.de | www.trialog-publishers.de Let’s keep in touch editorsdesk@international-transportation.com advertising@international-transportation.com