Controlling Robots using a Large Language Model
In the past, a human controlling a robot has depended entirely on what commands the human can send the robot, either through structured messages or through user interfaces. That's changing with Large Language Models (LLMs), because we can provide a few tools for controlling robots to an LLM and then give commands in natural language, and the LLM works out how to send the right commands to the robot. This gives us a lot more flexibility in how we can interact with robots.
In this post, I show an example project that runs a Gazebo simulation of a robot in a small house. The project uses Claude, an LLM from Anthropic, to interact with the robot, which allows the user to give the robot natural language commands and see those changes happen in simulation.
After the demo, I show how to set it up, then talk about how the code interfaces with Claude.
The demonstration project in this post is from a university-based robotics course. One of the groups on the course kindly released their submission on Github so that I could show it in this post. Therefore, I'd like to thank Senior Lecturer Giovanni Toffetti at Zurich University of Applied Sciences and his students Alexander Kolenaty, Jan Affeltranger, and Ilimea Gall.
This post is also available in video form. If you'd prefer to watch, click the link below: