Data companion · robo-bench sessions

Benchmark Models on Their Ability to Navigate AIM Robot

The paper's results come from these session files. This page reads them as they are, from the sessions/ folder beside it: the prompt each model received, the code it wrote, its reasoning, the overhead camera frames of every run, the robot's console log and its telemetry.

← Paper and overview Paper PDF ↗ Harness source ↗ Reading sessions/games …
Dataset
Sourcesessions/games/
Games—

Results by game

Outcome of each model on each game, as reported in §8 of the paper. One square per attempt. The photo is the last overhead frame of the deciding run. Select a cell to open it below.

Show under each attempt
Loading…
Passed on this attempt Failed Not scored Models stop after a pass or after three attempts (§3.2).
Every run, one row each
OVERHEAD CAMERA
t 0.0 s
0.0 / 0.0 s
Console log raw ↗
Odometry path · mmwheel odometry from start pose; +y = initial facing
Inertial
Robot camera · detections320 × 240 px frame
Telemetry over the runtelemetry_series.json · hover to read values, click to jump

Reading the data

  1. Outcomes come from the paper. The harness does not record pass or fail. The marks above apply §8 (“In Game 1 all three models passed, in Game 2 two models passed, in Game 3 only Opus passed, and in Game 4 none of the models passed”) and §5.3, which names Muse as the second Game 2 pass. Under the stop rule in §3.2 a passing model's last attempt is its pass and every earlier attempt failed.
  2. Game 3 has two Opus experiments. exp_001 ran one attempt and was replaced eight minutes later by exp_002, which the harness names “Actual Opus”. The paper reports one Opus entry for Game 3, so exp_001 is shown but not scored.
  3. Some attempts have two runs. The same code was run twice. §3.1.1 says code was run once “unless there was an outside problem”. The paper doesn't say which run was scored, so both are shown.
  4. Frames. The overhead camera saves a frame about every 2 seconds. The player holds each frame until the next one, using the times in frames.json. Field crops each 1920 × 1080 frame to the middle 60% of its width, where the field sits. Full frame shows the file as recorded.
  5. Console log timing. console.log has no timestamps of its own. Line times come from the times the model's program printed ([ 12.3s], t=12.3, elapsed=12.3). A printed time that runs ahead of the lines after it, such as a deadline, is ignored. Lines in between are spaced evenly and marked ≈. The program starts its timer after it connects, so expect up to about a second of offset from the frame clock.
  6. Telemetry. Each run's telemetry_series.json holds about 4–5 samples per second. Position is wheel odometry relative to the start pose. It drifts and cannot see the robot being pushed. Speed is computed from consecutive positions. The camera panel plots only the detection the telemetry reports (ai_x, ai_y, ai_score), in the robot camera's 320 × 240 frame.
  7. What the model saw. The Prompt tab splits input/prompt.txt into its tagged sections and shows the images from input/ that were attached. The checks at the top of that tab compare files byte for byte in your browser: trace.txt against the system prompt plus prompt.txt, <previous_code> against the previous attempt's output.txt, and the game description against the game's prompt.txt.
  8. Feedback and usage. Messages sent with the Feedback button go privately to the authors and aren't shown on this site. The page also counts which games and attempts are opened, replays started and time spent on the page. It uses a random ID per browser tab, sets no cookies, and stores no IP addresses.

Send feedback to Meher and Abhay

They read every message. Nothing you send here is published on this site.

What kind of feedback?