I Gave My OpenClaw Agent a Physical Body
I recently gave my OpenClaw a real robot arm to play with. The results almost blew up my own neural network.
The AI agent was able to configure the arm, use it to slowly see and grasp objects, and even train another AI model to pick up and place specific objects. And they say AGI is still a few years away! (Just kidding, it probably is).
The results convinced me that we may be on the verge of a breakthrough in robotics. Previously, training and controlling robots required considerable skill. Today’s AI models can make things almost easy.
“AI-based coding is extremely exciting because it has the potential to bridge the gap between conventional engineering methods, which are reliable but not becoming widespread, and contemporary vision-language-action models, which are becoming widespread but are not yet reliable,” says Ken Goldberg, a roboticist at UC Berkeley who is exploring this approach.
I bought a pre-made arm called LeRobot 101. It’s part of an open source project by HuggingFace that makes it relatively cheap to build and experiment with robotics.
The LeRobot comes with two arms: a controller arm that a person operates using a handle and trigger, and a follower arm with a camera that replicates these movements. You can train an AI model by teleoperating the controller’s arm and having it learn to move the follower in response to what it sees on the camera.
Building with OpenClaw
Before using OpenClaw I spent several hours trying to connect and calibrate the robot, at one point I almost broke the motors by applying the wrong settings, causing them to overheat.
Then, with the help of OpenClaw and the Codex, I was able to code a simple program that closed the claw’s claw when it spotted a red ball. In the terminal, Codex did the delicate work of setting up the connections to the robot. Then, with my help, he calibrated the positions of his joints. He also wrote a Python script using several libraries to identify and grab the ball in question. Vibrational coding is of course not perfect and hallucinations can introduce bugs, especially when working with different hardware, but the results have been impressive.
