A system that allows home robots to cook in collaboration with humans

Home robots could assist humans with the completion of various chores and manual tasks, ranging from washing dishes or doing the laundry to cooking, cleaning and tidying up. While many roboticists and computer scientists have tried to improve the skills of home robots in recent years, many of the robots developed so far are still unable to tackle more complex and creative tasks, such as cooking in collaboration with human users.

Researchers at Cornell University have recently developed MOSAIC, a modular architecture that allows robots to perform complex household tasks that involve closely interacting with humans, including interactive cooking. This system, introduced in a paper published on the arXiv preprint server, was tested in a series of real-world experiments, showing that it could support humans with cooking various recipes.

"We present MOSAIC, a modular architecture for home robots to perform complex collaborative tasks, such as cooking with everyday users," Huaxiaoyue Wang, Kushal Kedia and their colleagues wrote in their paper. "MOSAIC tightly collaborates with humans, interacts with users using natural language, coordinates multiple robots, and manages an open vocabulary of everyday objects."

The researchers' proposed system is modular, which means that it is comprised of different parts or modules that tackle different aspects of the task at hand. The system's components can be broadly divided into an interactive task planner, an architecture for identifying objects and planning the movements of robots and a model designed to predict the movements of humans.

"At its core, MOSAIC employs modularity: It leverages multiple large-scale pre-trained models for general tasks like language and image recognition, while using streamlined modules designed for task-specific control," the researchers explained in their paper.

Wang, Kedia and their colleagues have so far evaluated their system's performance in 60 experimental trials, using two different robotic systems, namely the mobile manipulator Stretch Robot RE1 and the tabletop manipulator Franka Emilka Research 3. During these trials, the two robotic manipulators closely collaborated with a human user to prepare six relatively simple recipes, including two different types of salad, three different types of soup, and a tuna sandwich.

"We also extensively test individual modules with 180 episodes of visuomotor picking, 60 episodes of human motion forecasting and 46 online user evaluations of the task planner," the researchers wrote. "We show that MOSAIC is able to efficiently collaborate with humans by running the overall system end-to-end with a real human user, completing 68.3%(41/60) collaborative cooking trials of 6 different recipes with a subtask completion rate of 91.6%."

In the interactive cooking experiments carried out by the researchers, MOSAIC performed considerably well, successfully completing approximately two-thirds of the recipes it prepared with humans. MOSAIC could soon serve as an inspiration for other research studies, thus contributing to the advancement of assistive robotic systems designed to be deployed in household environments.

In the future, Wang, Kedia and their colleagues could further improve some of their system's underlying components, to further boost its performance in both collaborative cooking and other interactive tasks. For instance, the system has currently been found to perform well on simple manual subtasks, such as picking up and moving objects, but it has not yet been applied to more advanced subtasks. This limitation could be tackled and improved in their next studies.

"The extension to more intricate tasks such as cutting, rolling, and spreading is less trivial," the researchers wrote in their paper. "While our system is tested extensively on different recipes, our experiments are restricted to one kitchen environment. Future work will attempt to measure the limits of our system's generalization capability in the face of a wider range of kitchen environments."