By combining reflections, scientists at MIT reconstruct images with high precision

Using the reflections of electromagnetic waves and artificial intelligence (AI), researchers at the Massachusetts Institute of Technology (MIT) are enabling robots to locate objects that are completely or partially hidden. The waves penetrate whatever is concealing the objects being sought. These reflect the signals, or parts thereof. From this, the receiver generates an image that shows the object being sought in full, though usually only partially. The system compares this with images it has stored during training and selects the one that most closely matches the object being sought.

Packaged goods are identified

These innovations enable warehouse robots to check packaged items before dispatch, thereby avoiding returns of goods packed in error. They also enable smart-home robots to detect a person’s location within a room, thereby improving the safety and efficiency of human-robot interaction.

“This opens up many interesting new applications, but technically speaking it also represents a qualitative leap, from the ability to fill gaps we couldn’t see before to the ability to interpret reflections and reconstruct entire scenes,” says MIT computer scientist Fadel Adib.

Some of the reflections are lost

The developers use millimetre-wave signals, such as those used for Wi-Fi. These penetrate common obstacles such as plasterboard walls, plastic and cardboard, and are reflected by hidden solid objects. However, these reflected signals are scattered in various directions by surfaces that are not perpendicular to the incoming waves, so that the sensor intended to capture them can only detect a fraction of them. This results in a partial image, which is completed using insights from AI training.

The system, called “Wave-Former”, suggests a range of potential object surfaces based on millimetre-wave reflections, feeds these into the AI model to complete the shape, and then refines the surfaces until a complete reconstruction is achieved. Wave-Former can thus generate true-to-life reconstructions of around 70 everyday objects such as tins, boxes, utensils and fruit. In tests, the objects were hidden behind or beneath cardboard, wood, plasterboard, plastic and fabric.