Civil Engineering – Promoting Real-World Applications of AI Technology for Next-Generation Infrastructure Maintenance-

In Japan, a large number of social infrastructures constructed during high economic growth are aging, and there are concerns about the increase in serious accidents and maintenance costs. In this context, there is an urgent need for new infrastructure management techniques that use science, technology, and innovation to ensure the functionality and integrity of social infrastructure and reduce the risk of accidents and disasters. To realize the next generation of infrastructure maintenance, Laboratory of Media Dynamics is promoting research aimed at improving the efficiency of maintenance work and effectively transferring skilled techniques to young engineers by applying state-of-the-art (AI) technology.

AI-based High-precision Deformation Classification

Infrastructure is the subject to a wide variety of deformations such as cracks and corrosion (see figure below), and accurate classification of these deformations is required to support efficient maintenance management. In general, image classification requires thousands to tens of thousands of training images per class; however, in actual inspection sites, the number of images is limited since engineers take images manually.

In Laboratory of Media Dynamics, we have developed an AI system, which enables high-accuracy deformation classification from a small number of training images. In this study, we introduce Convolutional Sparse Coding to deep learning, which enables the AI to learn the characteristics of various deformations from a small number of training images (see the figure below). The classification accuracy achieves 92.5% with 400 training images per class ( accuracy of previous deep learning method: 78% ).

Crack feature learning process by AI constructed in this study.

Visualization of the Basis for AI Decisions

The infrastructure maintenance requires an extremely high level of safety. Therefore, in addition to high classification accuracy, it cannot be said that efficient infrastructure maintenance management using AI has been achieved without explaining the reasons that led to the classification results.
In Laboratory of Media Dynamics, in addition to high deformation classification accuracy, we are conducting research on improving explainability of classification results. We have developed ExplainableAI (XAI), which can explain the classification results by visualizing the regions that the AI focuses on. XAI is attracting a lot of attention in the current AI field, and the application of XAI to deformation images is a unique attempt in the world.
The red area in the figure below represents the area that the AI focused on. The regions close to the deformed regions are shown in red, indicating a high degree of confidence in the classification results.

Tacit Knowledge Inheritance of Skilled Engineers

There are concerns that the number of skilled engineers who maintain and manage infrastructures is decreasing due to the aging of the workforce and the declining number of young engineers. Therefore, there is an urgent need to establish technologies that can efficiently transfer the knowledge of skilled engineers.
In Laboratory of Media Dynamics, we have developed a technique to analyze the relationship between know-how and biometric signals obtained from skilled technicians wearing various sensors (such as gaze sensors, motion and life sensors) during infrastructure inspections. We found that the behavior peculiar to skilled technicians during sound and visual inspections are highly correlated with “eye movement” and “head and arm movement.”

Initiatives for Social Implementation ( link )

Deformation Evaluation Support System

In Laboratory of Media Dynamics, we are not only constructing theory of advanced AI technology using social infrastructure data, but also developing and verifying a “Deformation Evaluation Support System” (see below) using the image analysis technology we have constructed. The results of this joint research with East Nippon Expressway Company Limited were also introduced by Microsoft Japan (Published in July, 2019).
When the type of deformation and the degree of deterioration are determined, the target image is input into the system, and similar images from the past and their determination results are retrieved. This allows for efficient infrastructure maintenance.

Example of the deformation evaluation support system

Accepted by Top Journal of “Information Science x Civil Engineering ( link )

By applying state-of-the-art multimedia technology to the field of civil engineering and developing new theories to solve problems specific to the field, our paper has been accepted by Computer-Aided Civil and Infrastructure Engineering (CACAIE)(2020 IF=11.775).

Analyze a diverse data set in collaboration with the field ( link )

We are developing analysis technologies for a wide variety of data related to infrastructure maintenance and management, including not only image and video analysis but also biological signals obtained from technicians during inspections. The Laboratory of Media Dynamics, which specializes in multimedia analysis, is a pioneer in this field.

Cooperative Institutions

  • East Nippon Expressway Company Limited.
  • Tokyo Metro Co., Ltd.
  • Tokyo Electric Power Company Holdings, Inc.
  • Hazama Ando Coporation.
  • Sapporo River Office, Sapporo Development and Construction Department, Hokkaido Development Bureau, Ministry of Land, Infrastructure, Transport and Tourism.