Evals Overview
The family of judged model comparisons -- trials, experiments, compare and watches -- and the one rule that separates traffic evidence from corpus evidence.
Evals Overview
An eval is an input, one or two dispatched arms, and a scored result. Every score carries the stamp of the rubric that produced it, so scores from different rubrics can never be pooled.
The family has four members, and they differ in what they run over and what their verdict entitles you to do.
| Runs over | Arms dispatched | Verdict may promote | |
|---|---|---|---|
| Trial | Your own live traffic, as it is served | The candidate only -- your users already got the incumbent's answer | Yes |
| Replay trial | Your own retained traffic, from a past window | The candidate only | Yes |
| Experiment | A dataset you supplied | Both -- nobody ever ran these inputs | No |
| Compare | One prompt you type | Every model in the slate | No -- nothing is stored |
A watch is the fifth thing, and it is a sentinel rather than a comparison you start: after a promotion it keeps the model you replaced running at a low sample rate and tells you if it starts winning again.
A trial never exposes users to a candidate. It runs beside serving, never inside it, so observing one cannot delay or change what a caller receives. Your users keep getting the incumbent's answer throughout.
Traffic evidence and corpus evidence are not interchangeable
This is the load-bearing product rule, and the interface enforces it rather than suggesting it.
Measured on your real traffic. The incumbent's answer already reached a user, so only the candidate is dispatched. The verdict is a shadow of production and carries full weight. Promotion is available from it.
Measured on a corpus you supplied. Not a shadow of your traffic -- both arms are dispatched, so it costs 2x per item. It can cover cases your traffic does not. Promotion is refused from it, and that refusal lives in the API, not merely in the interface.
The Results screen shows both, on one page with one filter bar, in two tables that never interleave -- because pairs and items are not the same measure and a shared table would imply the verdicts are comparable evidence.
The verdict wording differs too, deliberately:
| Outcome | On traffic | On a corpus |
|---|---|---|
| Candidate scored better | Adoptable, or Adopted once promoted | Candidate ahead |
| Candidate scored the same | Adoptable | Parity |
| Candidate scored worse | Rejected | Candidate behind |
A corpus row must never say "adoptable".
The screens
| Screen | What it is for |
|---|---|
| Live | What needs attention, and what is in flight -- across trials and standing watches. The alert strip sits above the tiles, because a firing drift alert is the only reason to open this page in a hurry. Tiles: running trials, active watches, firing alerts, evals spend. |
| Results | Every eval and its verdict -- measured on your traffic, or on a corpus you supplied. |
| Trials | Shadow a candidate model against your incumbent on real traffic: no dataset, no code change, no user exposure. |
| Watches | Keep the model you replaced running as a low-rate sentinel, and hear about it if it starts winning again. |
| Compare | One prompt, several models, side by side on your own credentials -- to feel a candidate out before spending trial money. |
| Datasets | Corpora an experiment runs over: imported from a file, or seeded from your own traces. |
The Watches screen has no "new watch" action by design -- a watch is created only from a promotion, so the affordance lives where the adoption decision is made. Compare has no list, no history and no detail route, because nothing on it is stored. The way a comparison outlives the tab is by becoming a trial.
The judge
Two rubric families, and their scores can never be pooled:
| Rubric | Used by | Produces |
|---|---|---|
| Paired judge | Trials, replay trials, experiments, watches | One comparative call scoring both answers together, at a randomized presentation order. |
| Absolute judge | Compare | One answer scored on its own merits, with no counterpart. Its scores can never be read as a delta. |
Every score carries a judge version, reported beside every verdict. A rubric change is a new judge version, never an edit in place.
See Judging and verdicts.
What an eval costs
Evaluation spend is money spent to evaluate models rather than to serve your own traffic -- a trial's arms and its judge, and the same for an experiment. It is your own money on your own credential, so it reaches your provider bill like any other usage. The running figure is list-priced rather than contracted -- see Trials.
It is bounded by the per-trial spend cap approved when the trial starts, and deliberately not by your guardrail's budget. Otherwise the two would govern one pot while only one of them can refuse a real user's request -- so an evaluation nobody asked about would end up denying traffic somebody did.
Evaluation spend is reported beside your metered spend lines and is never summed into your total. See AI spend.
Where a comparison gets judged
Where a trial runs is decided by your content uplink posture together with a staged per-team rollout:
- Content uplink on for every gateway your team runs, prompt retention on, and your team cut over by your deployment: the trial runs in DevZero's control plane. Control-plane execution is staged per team rather than general, so check with DevZero whether yours is cut over.
- Any of those missing: the trial runs in your own gateway's engine.
- Never both. The exclusion is structural, not conventional.
One condition the uplink does not imply is prompt retention: uplink decides whether a trial may run control-plane-side, retention decides whether it can, because a control-plane trial reads bodies out of stored telemetry.
Turning prompt retention off keeps trials working. What becomes unavailable is replay and seeding a dataset from your traffic, both of which read stored bodies.
Experiments are outside this predicate entirely. A corpus is content you deliberately uploaded and no production traffic is involved, so an experiment always runs control-plane-side -- including on a content-free deployment.