robocodebench Explore the data →
Research paper · 11 November 2026 · 30 pages

Benchmark Models on Their Ability to Navigate AIM Robot

Can an AI model write code that drives a real robot, with real physics, from nothing but a photo of the field and the robot's code library? We gave three frontier models four games of rising difficulty on a VEX AIM field. Each model got up to three attempts per game, with its camera frames, console logs and telemetry fed back after every try.

4games
3models
27attempts
29robot runs
780overhead frames
Fig. A · Game 3 field at the start Overhead USB camera · cropped to field
ItemPartQtyNote
1VEX AIM robot1Kiwi drive, AI vision camera, kicker
2Sports ball1Magnetic; the kicker holds and launches it
3AprilTag target2Knock both down to pass
4Barrel2Hitting one directly is a fail
The photo Claude Opus 5.5 received with its Game 3 prompt. Field: four 9 × 9 in tiles. Watch this run →
§6 · §8

Findings

Opus passed 3 of 4 games, each on its first attempt. Muse passed 2, Astra 1. No model passed Game 4. Performance fell as the games got harder. Select any result to watch the run behind it.

Model Game 1One target Game 2Two targets Game 3+ two barrels Game 4Sort barrels PassedGames
Claude Opus 5.5 Pass · 1st try Pass · 1st try Pass · 1st try Fail · 3 tries 3 of 4
Muse Spark 1.3 Pass · 2nd try Pass · 2nd try Fail · 3 tries Fail · 3 tries 2 of 4
GPT-6 Astra Pass · 1st try Fail · 3 tries Fail · 3 tries Fail · 3 tries 1 of 4
Passed on this attempt Failed attempt One square per attempt. A model stops after a pass or three attempts.
Hypotheses (§1.4, checked in §8)
  1. H1Models will perform significantly differently.Matched
  2. H2Performance drops as games get harder.Matched
  3. H3More objects on the field, more errors.Matched
  4. H4Fewer images given, worse performance.Not tested

Each model played each game once, with up to three attempts. A real robot adds noise from battery level, wheel slip and how the ball rolls, so the paper reports what happened in these trials and doesn't claim the differences are significant (§7).

§5

Four games

Each game is played on the same 18 × 18 in field with 60 seconds on the clock. The photos are the overhead images the models received with their prompts.

§3.2 · §4

How a run works

A harness connects the model, an overhead camera and the robot, so every attempt runs the same way and is recorded the same way. The models never see the robot directly. They see a photo, the API reference and what happened last time.

01 · PROMPT

Rules, API, photo

The VEX AIM Python API reference, the game rules and an overhead photo of the field with a ruler for scale.

02 · MODEL

Writes the program

The model returns one complete Python program, at high reasoning effort, with logging built in.

03 · ROBOT

Runs for 60 s

The harness runs the code against the robot over Wi-Fi while a USB camera watches from above.

04 · RECORD

Frames, logs, telemetry

A frame every 2 s, the program's console log, and the robot's heading, position and vision readings.

Not passed yet? Code, logs, telemetry and frames go back to the model for the next attempt, up to 3.

The paper

Benchmark-Models-AIM-Robot.pdf · 30 pages · 2.7 MB
Page 1 / 30
Download PDF Open in new tab ↗
Loading the paper…
Data availability

Data & code

Every run in the paper is published as the harness recorded it. The dataset explorer replays each run with its console log and telemetry on one clock. It also shows the exact prompt each model received next to the code it wrote.

input/prompt.txtThe full prompt: API reference, game rules and, for repair attempts, the previous code, logs, telemetry and frames
output.txtThe Python program the model returned
reasoning.txtReasoning text returned with the code
runs/*/frames/Overhead camera frames, one about every 2 s
runs/*/console.logEverything the program printed while it ran
runs/*/telemetry_series.jsonHeading, position, IMU and vision readings, 4–5 per second
harness.logTimestamps for generation and each run
Cite this paper
Meher and Abhay. “Benchmark Models on Their Ability to Navigate AIM Robot.” November 11, 2026.
Paper and data: https://robocodebench.org · Harness: https://github.com/Meher-ilovepandas/robo-bench