Frozen environment · live consequences · multi-step tasks
Dynamic role routing,
made inspectable.
RoleRouteBench evaluates whether a decision model can coordinate specialized executors across evolving task states—not merely whether the final answer happens to be correct.
Final task success cannot tell you whether a router coordinated well.
Outcome quality also depends on the frozen executor. RoleRouteBench therefore retains every routing decision, failed action, state transition, budget event, and independent evaluator result.
Competing routers see the same public state, role pool, adapters, executors, budgets, and evaluator. The environment stays fixed; only the routing policy changes.
JEV 1.13 results
Reference signals for environment validation—not an oracle and not a composite leaderboard score.
| Domain | Packets | Trajectory pass | Trusted task pass | Primary decisions | Budget-suppressed |
|---|---|---|---|---|---|
| Loading reference results… | |||||
Follow each routed consequence
Filter public compact traces. Raw prompts and executor outputs are not included.
Public/private isolation by construction
“Private” means evaluator-only under the benchmark protocol; the artifacts remain downloadable for transparent reproduction.
Public state
- Task and public artifacts
- Full role pool plus STOP
- Prior observable consequences
- Remaining call budget
Reference state
- Gold patches and proofs
- Answer workbooks and oracles
- Trusted evaluator results
- Track A labels
The evaluator runs only after termination. It does not feed private artifacts back into the trajectory.
Bring your own router.
JEV is the reference system. Submit reproducible results from another decision model under the frozen environment—without sending us your model.