A robot demo winner can lose on your workbench
RoboColosseum’s distributed real-robot evaluations show why performance can change across hardware, tasks and fine-tuning. Build a pilot around the work you need.
Two robotic arms look equally convincing in videos. One slips on an unfamiliar container. The other gets the grip right and places it in the wrong bin. An operations team choosing between them needs to know which failures occur on its own line and what it takes to recover.
RoboColosseum is a research effort to compare robot policies on physical hardware across distributed evaluation sites. In the November 2 arXiv version, the authors report 5,699 pairwise comparisons across 124 sites. They find weak agreement in policy rankings across robot embodiments and report that relative performance can change after fine-tuning. A buyer or researcher can use that evidence to design a better evaluation before making decisions from a demo reel.
A Unitree G1 (left), one of the robot platforms evaluated, at Japan Mobility Show 2025. Photo by RuinDig/Yuki Uchida, resized, CC BY 4.0. Image source · License.
A ranking has a robot, a task and a setting
On-site robot evaluation. Original illustration by Telli.sh.
A policy can perform well in one combination of robot, sensor position and task. Change the gripper or alter where objects enter the workspace, and a slightly different skill may determine success. A fine-tuned policy is also a different proposition from one used out of the box. Its performance depends on the demonstrations and adaptation process it received.
The study compares policies in matched local conditions. In its open track, evaluators compare attempts on the same task without knowing which model produced which attempt. The fine-tuning track uses a more controlled set of tasks and demonstrations. That separates two questions: what happens on a fresh instruction, and how well the policy adapts given examples.
| A result describes | A deployment team still needs to know |
|---|---|
| One robot embodiment | Whether its hardware and sensors resemble yours |
| One task distribution | Which objects, layouts and exceptions the tasks contain |
| A fine-tuned policy | Which demonstrations, time and compute produced it |
| A preference or progress judgment | How often a complete cycle succeeds without intervention |
| An evaluation period | Whether the configuration matches the version you will deploy |
The paper reports low agreement in rankings across four embodiments in its study. The project site can list a broader and newer set of active evaluation hardware. Use the study’s scope when explaining its numbers. A public platform’s current inventory describes what it offers now, while a paper describes the experiment at a particular point.
Look at how a failure unfolds
Read partial progress. Original illustration by Telli.sh.
The study reports that no tested policy completed the five tasks evaluated on a particular Unitree G1 setup. Progress scores still differed. That distinction is useful for anyone trying to understand whether a model misses the first step or fails near the end. It describes those tasks and policies under the study’s conditions.
Picture a sorting task with five milestones: identify the object, approach it, grip it, move it and release it in the right place. A model that reliably gets to the release step may be worth improving if the last step is simple. If the failure drops a heavy object, the same high progress score has little operational comfort. Partial progress tells you where to investigate. The physical consequence tells you whether the behavior belongs in the workplace.
For a short demonstration, a retry can be easy to overlook. On a busy line, a retry takes time and may need a person to reset the scene. A ten-second success followed by a one-minute recovery on some runs can produce worse throughput than a slower policy with fewer interventions. Measure both success and recovery time over enough cycles to observe the interruptions that matter.
Visual review is useful here. Keep the clips for a sample of full successes, partial attempts and failures with different causes. A video that ends immediately after a grasp can leave out the release. An evaluator who sees the entire attempt can distinguish a completed job from promising motion.
A field pilot can learn from a research comparison
A pilot should start with work that the site actually needs. A warehouse might care about a mixed tote with reflective packages and occasional crushed boxes. A lab might care about a sequence of handoffs involving small tools. The exact task, starting state and expected outcome make a test repeatable and help vendors understand what to improve.
- Pick a task whose successful completion you can observe. Write down the starting arrangement and the finish condition. Include representative objects and a few challenging but legitimate cases.
- Record the robot, sensors, end effector, site conditions and policy version. Where possible, run competing policies against the same local setup and instructions.
- Separate an out-of-the-box run from a run after adaptation. Record how many demonstrations and how much work the adaptation required, along with which examples were held out.
- Observe complete success, partial progress, completion time and interventions. Keep attempts that fail at setup or need an operator’s reset in the record.
- Repeat the test on a shift or in a location that changes a relevant condition. For example, use a new incoming object mix or measure behavior when the task starts slightly off center.
- Evaluate deployment controls and physical safety with qualified personnel and local procedures. The research benchmark measures policy behavior. A workplace evaluation also needs to account for the people and equipment near the robot.
If teams judge qualitative differences, try reviewing anonymized clips in pairs. Give reviewers a written definition of acceptable behavior and a way to flag an unsafe or unusable attempt. Retain enough metadata to understand disagreements afterward. Blinding can reduce brand preference, while a shared rubric helps a reviewer at one site evaluate the same outcomes as a reviewer somewhere else.
Compare the tracks before comparing policies
The Open Track and Fine-tune Track ask related questions with different experimental designs. The Open Track uses 5,699 pairwise comparisons across 124 sites, as reported in the preprint. A pairwise trial can show how two policies perform when the task and site are comparable. It does not imply that every robot was tested on every task at every site.
The Fine-tune Track uses 100 demonstrations per task across 20 tasks in the paper’s reported evaluation. The authors report zero full completions by the Unitree G1 across five evaluated tasks in that track, with mean progress scores ranging from 13.3% to 23.2%. The partial scores give readers more information about the gap than a single zero on the final outcome. They show that a policy can make observable progress under the evaluation rubric while falling short of the completed task.
The authors report a Kendall coefficient of concordance of 0.24 for progress rankings across four hardware-specific task suites. That is a measure of agreement between rankings, not a percentage of completed tasks. The suites differ in both the embodiment and the tasks and settings used with it. A reader can take the weak agreement as a reason to validate the chosen combination locally. It does not isolate robot shape as the cause of ranking differences.
Match the two shifts in a field trial
Suppose a distribution center wants a robot to move the same packaging insert from a tray to a carton during two shifts. Operators notice that the evening shift receives a different tray and has less natural light. Instead of moving the trial to a cleaner lab, run both candidate policies in both settings. Alternate the order within each shift so a late battery, a shifted fixture or operator practice does not consistently benefit one candidate.
Write down the initial object pose, which sensor or lighting settings are allowed, how the worker should intervene, and what counts as a completed placement. Keep unsuccessful video and intervention notes. If one policy improves only after the tray is repositioned, the intervention may be more relevant to deployment than the final average. That observation can lead to a fixture change or a better selection of cases for the next policy test.
A useful scoring sheet gives partial credit to observable phases. A gripper that finds the correct insert but cannot lift it needs different work from a gripper that lifts it and misses the carton. Use a simple definition for each stage so two reviewers can interpret the same trial consistently. Add a separate category for a safe stop. A run that stops because a worker enters the area may have behaved correctly even though the task did not finish.
Decide what evidence justifies the next step
A successful laboratory demo can justify an on-site trial, and a successful on-site trial can justify a longer pilot. Each decision has its own acceptance threshold. A purchasing team can ask for repeatability across two common shifts before discussing expansion to a less predictable area. A researcher can ask for failure labels and repeat conditions before claiming a policy transferred.
When results disagree, record what changed between settings and which action went wrong. A rank reversal can reveal a task the original benchmark did not represent. That helps the team decide whether it needs another robot, a different policy, a modified workcell or simply a more representative test.
Read the rank reversals in the paper
The Fine-tune Track provides concrete examples of rankings moving across the four task suites. π0.5 ranks first in the progress rankings on Franka and Bimanual-YAM, and fourth on G1. GR00T N1.7 leads the SO-101 suite while ranking fourth on Franka and Bimanual-YAM. Those results refer to the reported task suites and their progress metric. A procurement team should take the order as an example of how a ranking can move when the combination of tasks, settings and hardware changes.
The paper also studies active selection of which comparisons to run. In its analysis, roughly 400 to 500 adaptively selected evaluations reached around 90% agreement with the ranking using the full data, compared with roughly 1,000 under uniform selection. That is a result for this evaluation design. It offers a practical idea for a local pilot: devote more repetitions to cases where the leading candidates are difficult to distinguish, while still keeping a fixed minimum of representative cases.
A facility can separate an initial round and an adaptive round. In the initial round, give both candidates the same jobs under the same intervention policy. In the second, repeat settings where the initial evidence is mixed. Keep the original acceptance rule in place. If a setting was excluded for safety or practicality, record the reason. A final report should be able to explain how the additional trials changed the decision.
Buy information about the jobs that matter most
Distributed evaluations can collect a range of settings that a single laboratory cannot easily reproduce. A local pilot gives depth on the jobs your team owns. These approaches work well together. Use broad evidence to find promising candidates and design your own evaluation to discover the exceptions that determine whether a system will be useful.
The strongest comparison is one your team can rerun after a new policy or a hardware change. Keep the task, configuration and failure examples with the score. When the ranking shifts, you will have enough information to decide whether the change improves your work.
Sources
For a searchable record of the conversations and decisions behind your work, Create a Telli.sh account.