Visualization of Factory Operations Using AI — Development of MIMAMORI AI
Applying deep learning to equalize workloads and improve work quality in factories
IHI Corporation
The IHI Group has begun trial operation of an AI system that supports activities to improve work efficiency in factories. The AI system automatically visualizes factory workers’ operating hours and work content based on image recognition employing deep learning (a technology that recognizes and analyzes images using pre-trained AI models). This contributes to the digitalization of factories in other words, productivity improvement using the digital technology.
Introduction
The IHI Group factories have been promoting activities to improve worker efficiency. Until now, at the factories, in order to clarify workers’ operating hours and work content, the operating hours were measured with stopwatches and the results were aggregated, or the work content was visually checked. However, grasping the actual status of such operations at the factories was time-consuming and labor-intensive because it was done manually. In addition, it has been difficult to assign manpower to analyzing trends, such as whether accumulating experience reduces operating hours, and to continue measuring and aggregating operating hours over a long period. In addition, if operating hours or work content were not recorded, the worker had to rely on his or her memory, which might deviate from the actual work situation, and in some cases it was difficult to make essential improvements.
Recent advances in AI have made it possible to detect people and objects in images as well as to automatically classify the content of people’s actions. These technologies, called object detection and image classification, have the potential to automate the process of grasping operating hours and work content, which has been performed manually thus far.
The IHI Group has built our deep learning model trained based on standard datasets for object detection in image recognition and on training data created from images and videos in the factories. A system called “MIMAMORI (Monitoring) AI,” which uses this model to automatically detect workers and recognize their work content, is currently in trial operation. This article introduces MIMAMORI AI’s functions and its future development.
Functions of MIMAMORI AI
MIMAMORI AI is a system with two functions: a human detection function that automatically counts the number of workers and their operating hours using images acquired from cameras installed in the factory, and a work classification function that automatically recognizes the content of work and counts each operating hour.
(1) Human detection function
The human detection function automatically detects workers using input data from a camera installed near the factory ceiling that provides a bird’s-eye view of the work site, and automatically counts and aggregates the number of workers at each workbench and each worker’s operating hours. This mechanism combines the worker detection process, which employs deep learning, and the rule-based process for the detected coordinates in order to automatically aggregate the operating hours for each of the workbenches at each work site.
Based on the analysis results of MIMAMORI AI, it is possible to grasp imbalance of workloads at each work site and to help equalize workloads for the entire factory.
In addition, if MIMAMORI AI is installed at multiple work sites, the total operating hours can be calculated for each work site based on the number of workers and operating hours at each such site. This data enables us to visualize which workplaces have high workloads and to take measures such as increasing the number of workers. As the number of workers and work sites increases, manual measurement of operating hours becomes even more time-consuming, so the human detection function of MIMAMORI AI becomes more effective.
(2) Work classification function
The work classification function automatically recognizes work content from the images captured by a camera installed near a workbench and matches it with one of several preregistered work classifications, and automatically measures and aggregates the operating hours. This mechanism automatically recognizes work content from image data by sequentially performing the human detection process, which employs deep learning, and estimation process of the work content based on the joint point information of the detected person.
Previously, the breakdown of operating hours for each work task had to be determined visually by other workers or by the worker’s own memory. Visual judgment is costly because it requires the presence of other workers, while relying on a worker’s memory may not clarify which work should be improved.
By using the work classification function, it becomes possible to continue measuring and aggregating operating hours against each work content over a long period without using manpower. This enables us to identify bottlenecks more quickly than before and to work on improvements. For example, this function enables us to grasp the difference in operating hours when a novice and a skilled worker work on the task that is considered to be a bottleneck, thereby enabling us to perform a quantitative study of improvement effects by reviewing personnel allocations.
Using these two functions of MIMAMORI AI simultaneously, changes in work content can be grasped along with each worker’s overall movements. For example, if we use MIMAMORI AI to aggregate and analyze the time required for checking information and discover that the time spent for searching electronic manuals is redundant, we can enhance the data retrieval performance to help reduce the time spent for reading manuals. Similarly, if we use MIMAMORI AI to aggregate and analyze the time spent for making phone calls and discover that this time is redundant, we can change the means of communication from phone to email to help qualitative improvement of work content. Thus, by introducing MIMAMORI AI at each work site, it is possible to visualize the operating status of the factory and improve the work content quantitatively and qualitatively throughout the factory.
In addition, factory managers and supervisors can grasp the work efficiency of workers based on the visualized operating hours and work content. They can identify number of workers from the human detection results (overall grasping). They can also identify which task workers are engaged in, or whether workers are in a conversation, from the work classification results (detailed grasping). From the results of these activities, it will be possible to identify work content with low work efficiency to appropriately formulate and enhance improvement activities. In this way, MIMAMORI AI help factory managers and supervisors to facilitate their decision-making.
Efforts to Improve Functions
In addition to being used as a fixed device in a factory, MIMAMORI AI is also intended to be easily installed and used at work sites where visualization is desired. Factory managers, supervisors, and workers themselves can easily change the installation and usage locations of MIMAMORI AI according to their own purposes.
Until now, MIMAMORI AI has consisted of a workstation-like server and a camera connected to the factory network. Such a configuration is time-consuming and costly because it requires hardware installation and coordination with the factory’s security department to connect the cameras and hardware to the factory’s network. We have therefore adopted smaller hardware, such as an edge computer, and connected the cameras directly to the hardware to complete the configuration on a local area network, thereby improving portability and reducing the installation time and costs.
For advancing this initiative, MIMAMORI AI is being implemented on the edge computer developed by the IHI Group. The edge computer has a built-in interface and communication module for data collection, which facilitates data transfer when data is aggregated into data collection infrastructure such as the IoT platform operated by the IHI Group. In the future, implementing MIMAMORI AI on the IoT platforms will improve compatibility with developed programs (for example, virtualization technology) and data analysis capabilities in the cloud as well as promote value creation using in-house data.
For operability, MIMAMORI AI is equipped with a graphical user interface (GUI) that is easy to operate visually and intuitively without the need of inputting text in a command line such as a command prompt. This enables the user to easily specify areas on-screen by drag-and-drop operation, instead of opening a configuration file and manually inputting the coordinate values from the camera image. In addition, when displaying the results of detection by the system on a graph, instead of manually graphing the numerical values in the table data format, the graph can easily be displayed with the click of an on-screen button. The AI system is easy for everyone to use, as it can be operated with the mouse to specify the work area, record images as well as to display analysis results in graph form.
Conclusion
The IHI Group has begun trial operation of MIMAMORI AI with the aim of automatically visualizing operating hours and work content of factory workers. Next, we will implement MIMAMORI AI on edge computers and verify its operation, and promote service deployment so that MIMAMORI AI can be used wherever needed for as long as necessary. We will actively promote the development of MIMAMORI AI so that we can complete the trial operation and put it into full-scale operation.
Furthermore, when MIMAMORI AI is fully introduced into the factories after actual operation, it is expected that the image recognition technologies, such as human detection, can be applied not only to improve work efficiency but also to raise the level of worker safety and disaster prevention in the future. We will continue our development based on these prospects, and would like to contribute to the digitalization of factories using MIMAMORI AI for various locations and situations.