Experts at Columbia University synchronize motor control with spoken words
Researchers at Columbia University’s School of Engineering (https://www.columbia.edu/ ) have taught a robot to move its lips in sync with spoken and sung words. The machine can also mimic human facial expressions and, in the best case, adapt them to what is being said. The aim is to create a basis of trust between machine and human.
Unnatural and eerie
Teaching robots realistic lip movements is challenging for two reasons: it requires special hardware with flexible facial skin that must be moved quickly and silently by numerous tiny motors, and software that controls the lips and facial skin to match the spoken word and its meaning. Rule-based movements appear artificial, unnatural, and are sometimes even eerie.
The engineers avoided this by using highly flexible artificial facial skin; they had the robot learn how to change its facial expressions by observing humans. First, they placed the robot’s face, equipped with 26 motors, in front of a mirror so that it could learn how it moves in response to its artificial muscles. Like a child making faces in front of a mirror, the robot contorted its face randomly thousands of times. Over time, the machine learned how to control its motors to achieve certain facial expressions.
Pursing the lips doesn’t work
The researchers then showed the robot videos of people talking and singing to teach the artificial intelligence that controls its lips how people’s mouths move when making different sounds. The researchers admit that the lip movements are still far from perfect. The scientists show their work in a video (https://www.science.org/doi/10.1126/scirobotics.adx3017).
“We had particular difficulty with hard sounds and words that require pursed lips. But that will improve with time and more practice. The more it interacts with people, the better it will become,” emphasizes Hod Lipson (https://www.me.columbia.edu/faculty/hod-lipson), professor of innovation in the Department of Mechanical Engineering and director of the Creative Machines Lab at the School of Engineering.
