Compare
One prompt, several models, side by side on your own credentials -- an ephemeral playground for feeling a candidate out before spending trial money.
Compare
Compare fans one prompt out to several models and returns each one's output, latency, cost and score, side by side.
It exists to feel a candidate out before spending trial money. It is not a small trial and it never becomes evidence.
Nothing on this page is stored. That is why it has no list, no history and no detail route -- there is nothing to come back to. The way a comparison outlives the tab is by becoming a trial.
Limits
| Limit | Value |
|---|---|
| Models per run | 6 |
| Runs per user | 20 per 5 minutes |
| Default output cap | 1024 tokens |
| Maximum output cap | 8192 tokens |
| Output caps offered | 256, 512, 1024, 2048, 4096, 8192 |
| Default temperature | 0 |
The rate limit is a playground guard against a runaway client, not a billing control.
Who pays
Compare splits the money between two parties on purpose:
| Half | Paid by |
|---|---|
| The answers | You. Each column is dispatched on your team's own credential. |
| The judging | DevZero, on its own judge credential. |
That split is the reason a compare run costs two parties money, and it is also the reason a gateway-side trial cannot make the same split -- DevZero's judge credential does not exist inside your deployment, so on a trial you pay for both arms and the judge.
Each column resolves your credential through the same ladder the gateway forwards on. A model with no resolvable credential is a first-class result naming its remedy, not an omission and not an error -- the column tells you which credential to connect.
Streaming
The interface uses the streamed form, which emits results as they become true: the resolved slate first, then each column's answer as it lands, then its score once the judge returns, then the run totals.
The split exists because the facts of a run become true at different moments. A column's credential is known before dispatch, its latency and cost the instant it answers, and only its score waits for the judge. A unary request would withhold all of them until the slowest column finished.
One run costs one rate-limit token whichever form you use, and neither form can describe a column differently from the other.
Reading a column
| Field | Notes |
|---|---|
| Output | The model's answer. |
| Latency | Measured per column. |
| Cost | Where the upstream reports its own charge, that reported cost wins over a contracted rate. |
| Score | From the absolute judge -- one answer on its own merits. |
| Status | ok, no_credential, or error. |
| Stop reason | end_turn, max_tokens, stop_sequence, or unset. |
max_tokens is the stop reason you must see. An answer cut off at the cap
ends mid-sentence, and the absolute judge scores what it was given -- so it
marks the column down for being incomplete. That penalty belongs to your
output cap, not to the model. Raise the cap and run it again before
concluding anything.
Why its scores cannot be pooled
Compare carries its own judge version, distinct from the trial rubric, because its rubric is absolute where a trial's is comparative. Compare results are never written to the pair-score store at all, so no playground run can move a trial's verdict.
An absolute score is not a delta. Two columns' absolute scores are both opinions about single answers; the difference between them is not the same measurement a paired judge produces, and the product will not let one stand in for the other.
What to do with a promising result
Start a trial. A compare run tells you a model is worth measuring; only a trial on your own traffic tells you whether it can replace what you are running, and only a trial's verdict can promote.