Baseten Grid asks what a completed coding task costs
Baseten Grid routes coding requests across approved models. Compare accepted changes, retry costs and review time before choosing a routing policy.
A coding agent produces a neat change for a failing test. The change fails in CI, so the developer asks the agent to try again. The second attempt touches a larger part of the repository, and a reviewer spends half an hour checking a patch that was supposed to take five minutes. Which model was cheaper?
Baseten introduced Grid in public preview on November 4. It combines a gateway for approved models, a router and central controls for usage and guardrails. The service can sit between supported coding tools and their models so that a team can apply a common policy across requests. Its early-access page directs teams to a waitlist.
Put the denominator in the right place
Cost per accepted task. Original illustration by Telli.sh.
A token price measures a unit of model usage. A team has to ship something that passes its acceptance criteria. A request can produce an answer, a run can produce a diff and a task can still be incomplete. Counting those events separately makes a routing experiment useful.
Suppose two routes each receive 100 similar tasks. The inexpensive route costs $50 across its attempts and completes 50 acceptable tasks. The other costs $90 and completes 90. Both cost $1 in model usage per acceptable result in this illustrative calculation. Add human review time, tool charges and the cost of waiting for retries, and the routes may differ. Those costs can swing in either direction.
Keep failures in the numerator. A route that gives up on difficult requests can look efficient if its abandoned attempts disappear from the accounting. Also keep the definition of success stable. For a documentation change, an approved diff might be sufficient. A migration needs tests for existing data and a rollback path.
| Measure | What it captures | Where to look |
|---|---|---|
| First-attempt acceptance | Work a model finishes without a rerun | Test results and reviewer decision |
| Completed-task cost | All attempts and relevant tool charges per accepted task | Inference bills, tool logs and ticket results |
| Human correction time | Effort to make the output usable | Reviewer edits and follow-up commits |
| Tail latency | How long slow jobs keep people waiting | Queue, model and tool timing |
| Change safety | Defects that pass initial tests but surface later | Reverts, incidents and regression reports |
The job category matters. A repository-wide API migration can be hard because it requires context, coordination and many small changes. A one-line patch can be harder because a subtle concurrency bug is involved. File count and prompt length are useful signals only when they predict outcomes in your environment.
Read the router experiment as a trade-off
Routing experiment. Original illustration by Telli.sh.
Baseten’s technical report compares always using Opus 5.5 with routing tasks between that model and GLM 5.3 Flash. Across its mixture of DeepSWE, SWE Atlas QnA and Terminal-Bench tasks, the company reports a point with 72% lower cost and a pass rate within 0.8 percentage points of always using Opus. The report compares the router with random routing at comparable cost. That is important because sending more work to a cheaper model would reduce average model cost even with a coin flip.
The threshold encodes a choice about how much expected success is worth paying for. Raising the priority of cost can send more tasks to the cheaper route. A different setting can favor the more capable model where the outcome matters more. Baseten’s results describe its selected models, tasks and price assumptions. Your team will bring a different distribution of work and a different cost of failure.
The product announcement uses a separate rounded claim of up to 70% lower cost in early benchmarks. Keep that language separate from the experiment’s 72% result and its associated pass-rate comparison. The useful takeaway from the technical report is the shape of the trade-off. A team can choose a point according to what it values and then check whether that point survives real work.
A gateway can make policy visible
Grid’s announcement describes approved models, team-level controls and usage visibility. A platform team can give developers access to models their organization has already evaluated while seeing how usage is distributed. This can help a team learn where it is spending money even before it delegates route selection.
In practice there are several decisions. Which provider can see proprietary code? Which model is approved for a specific repository? Which team owns the budget for retries? Who can override a route after an incident? A gateway can put the decision at a shared boundary, provided administrators understand what the controls govern.
The model gateway and the agent’s runtime are separate control points. A gateway can decide which model receives a request. The tool or execution environment decides whether an agent can read credentials, write a file or make a network call. Route restrictions are valuable when they work alongside those runtime permissions. An inexpensive route still needs the same constraints around a privileged tool.
There is also a question of evidence. Routing logs help explain why costs moved, but traces can contain code, paths and details of a customer incident. Decide what the platform may record, who may see it and how long it can be retained. A policy a developer cannot inspect is difficult to troubleshoot when a request is denied or unexpectedly sent to a different route.
Test routing before changing your default
- Pick two or three common task families. Include a straightforward group, such as routine test updates, and a harder one, such as migrations with backward-compatibility requirements.
- Define the acceptance evidence before starting. For a coding task this can include a passing regression test, a clean diff, a reviewer’s approval and a check for unexpected file changes.
- Record a baseline using the current default model. Include failed attempts, retries and work a developer finishes manually. Preserve any security or data constraints used in production.
- Compare routing on equivalent tasks. If access is available, check the gateway’s supported tools and permissions in the official documentation before connecting an account.
- Decide when to escalate. A first-pass failure can trigger a second model, but an unbounded retry loop can exceed the cost of using a stronger model first. Set a limit and record why the task was escalated.
- Keep an override for tasks where an incorrect result has large consequences. Review exceptions alongside aggregate cost and acceptance figures after the trial.
For an internal pilot, compare a period of developer-chosen models with a period of routing, using similar work. The first group may benefit from developers’ knowledge of the repository. The second may benefit from consistent policy and better visibility. Document task differences so the comparison shows where automation helps and where judgment still earns its keep.
Read a matched task log
A useful place to begin is a month of completed pull requests. Tag work by its initial request, such as a dependency update, a bug with a reproducible test, a change that crosses repository boundaries, or an open-ended request to investigate a failing deployment. Save the repository revision and an acceptance check for each task. Replay a sample against a stable baseline and against the router. Use the same tools, time budget and test environment in both runs.
A dependency update is a useful low-ambiguity case. The work is successful when a named dependency changes, compatibility checks pass and the pull request contains no unrelated edits. A task that touches a billing contract needs stricter review. It may also require a human to interpret incomplete requirements. Keeping these task families visible can show where lower average spend comes from. If the cheap route wins on dependency updates while needing repeated attempts on billing changes, a team can route the two differently.
Record the time a human spends deciding whether to accept the result. Suppose a baseline produces 19 accepted results out of 20 at a cost of $19, while the router produces 18 accepted results for $9. Model cost per accepted result would be $1.00 and $0.50 in this illustrative comparison. If reviewers spend an additional 20 minutes fixing one failed routing result, the cost comparison changes. The right unit for an organization can include that rework without pretending the dollars and minutes are the same measurement.
Know what the router learned from
Baseten’s technical post names DeepSWE 1.1, SWE Atlas QnA and Terminal-Bench 2.1. The study compares Claude Opus 5.5 and GLM 5.3 Flash across tasks and evaluates the router on held-out data. Those details matter when deciding how far to generalize. Repository navigation questions have a different cost shape from tasks that run a shell, use external tools or change several files. The customer’s evaluation should represent the tasks they actually expect to send.
The technical post describes Cost, Balanced and Intelligence settings as request-time choices that move the routing threshold. A product team can use that distinction to run a shadow comparison. Compare settings on a fixed set of authorized replays and record the selected model if your integration exposes it. Inspect the disagreements. If high-cost choices mostly occur on cases the team considers sensitive or hard, the policy may be doing useful work. If the disagreements are concentrated in a common simple task, it may be worth adjusting the task classification or policy.
Watch the slow and expensive tail
Mean latency can hide the user waiting for a difficult patch that went through several models. Record the wall time from request to accepted result and tag any retry, escalation, timeout or handoff. A dashboard that only shows the first model's response time can report a win on a task the operator had to restart. Include spend and time after handoffs.
Cap the trial before enabling it broadly. Use a budget for the entire task and a rule for when to stop or move up to a more capable model. A failed attempt should leave enough evidence for the next attempt to continue. Avoid re-running tests or reading the same files merely because the router selected another model. The useful measure is how far the work progressed.
For a rollout, start with a narrow family where the acceptance rule is automated and a human can inspect exceptions. Next, add a family where acceptance requires judgment. Compare cost per accepted task, the share of escalations, and time to review separately. If the gain survives that change, the organization has better evidence for setting a default.
Keep the baseline useful after a model changes
A result collected in one week can become stale when either model, a tool, or a repository changes. Save the model identifier, the test revision and the acceptance rules alongside the routing decision. Re-run a small set of representative tasks when changing the policy. Keep a few expensive failures in that set, because their costs tend to disappear into averages.
Ask a reviewer to examine a route disagreement without seeing which model was chosen. If the person cannot tell which output is acceptable, refine the acceptance check before collecting more runs. A routing policy is easier to compare when the team agrees on how to recognize success.
The policy should improve as the work changes
Routing is valuable when it frees a team from making the same small choice repeatedly. It is especially useful when performance and price can be measured in the same unit of work. Over time, models, prices and your repository will change. Keep the benchmark tasks and update the test set so the routing policy can move with them.
The number worth taking to a planning meeting is what the team paid to deliver accepted work, with a clear account of the trade-offs. It gives the engineering lead and the budget owner something they can act on together.
Sources
For a searchable record of the conversations and decisions behind your work, Create a Telli.sh account.