COMMiT: System for detecting fatigue and anomalous behavior in heavy machinery operators using computer vision

In those years, the growth of computer vision-based products was making this technology a way to solve problems in people's everyday lives in areas such as safety, education and health. The industrial sector was one of those that could benefit most from computer vision solutions for capturing, analyzing and visualizing information from increasingly complex processes, although their use was still at an early stage of experimentation and deployment, especially in critical applications or tasks of the production process, such as occupational risk prevention.

One of the jobs with the highest risks was that of heavy machinery operator, found in processes for manufacturing and distributing equipment and materials in very diverse contexts (construction, transport, warehouse management). Handling this type of machinery required operators to perform precise, repetitive tasks that, for safety reasons, were critical and demanded their full attention for long periods of time, so monitoring and early detection of fatigue or signs of stress were crucial for preventing workplace accidents and for the productivity and efficiency of the process.

Despite the progress made in the industrial sector, supervision of heavy machinery handling still depended on a person watching over and assisting the operator during loading and movement of material, communicating with them through microphones or intercoms that were difficult to use in very noisy scenarios generated by the machinery itself, which made it harder to detect or react in time to an incident.

The COMMiT project addressed this problem by developing computer vision algorithms with machine learning techniques for the facial analysis of heavy machinery operators, with the aim of detecting signs of fatigue and stress to enable the prevention of, or early reaction to, accidents, and thereby reduce costs and improve productivity. To this end, a camera-based system designed to be integrated into cabs was created, capable of continuously capturing and analyzing the operator's facial expressions to raise an alert in the event of any anomalous behavior. It also used machine learning techniques to train the algorithms and adjust them to each particular scenario and case, and kept a history of data for each operator that made it possible to analyze their evolution over time, identify signs of future problems and feed back into the system to improve the monitoring process.