Real-Time Hazard Detection via Machine Vision for Retail Robots

The Challenge

Today’s fast-paced retail environment demands that companies invest in solutions that speed up their services and respond quickly to issues that impact their customers. One such issue is quickly detecting spills, trash, and hazards in retail stores and notifying staff to promptly take care of it. Our partner company was a provider of retail store maintenance robots. In the testing stage, they learned that their robots were experiencing issues in recognizing smaller objects (under 2 cm in height) and transparent fluids. While the robots were equipped with monocular vision to collect and sort hazard information, they weren’t able to identify and bypass these particularly challenging objects, which often resulted in their malfunction. I.e., robots would often get stuck in sticky fluids like honey, oil, etc.

The Solution

We had a three-pronged approach to finding a solution to the problem. Considering the pressing deadlines of our partner, we formed three separate teams that would work on three possible solutions simultaneously.

Machine Learning

We used machine learning algorithms to improve the trash recognition capacity of the robots. While those algorithms were successful in improving hazard detection, they required a higher processing power, which was draining battery life and decreasing the operational time of the robot.

Image Processing

Given that image processing does not achieve 100% accuracy, we came up with algorithms that would increase the robot’s performance for specific scenarios. E.g., if the pattern and texture of the floor in a particular retail store were constant, our solution provided 90-95% accuracy. The algorithm configurations could be easily adjusted and uploaded into the robot controller application and calibrated for different environments.

Our algorithm also worked without preconfiguring the pattern/texture of the floor, however with less accurate performance.

Infrared Data Processing

We developed and tested several image processing algorithms that would use the data received from the infrared camera of the robot. The plan was to combine and process data from both the infrared and the regular cameras of the robot for more accurate results. However, yet again we were hit with the robot’s hardware limitations, which didn’t allow us to make the necessary changes for complete infrared support.