Experiments
Running a candidate over a corpus you supplied -- quoted before it runs, both arms dispatched, and never a route to promotion.
Experiments
An experiment is a comparison over a dataset rather than over traffic. Everything below the corpus is the same object as a trial: the same rubric, the same judge version, the same spend cap, the same lifecycle.
What differs is upstream of all of that. Nobody ever ran a corpus item, so it carries no incumbent answer and both arms are dispatched.
| Trial | Experiment | |
|---|---|---|
| Runs over | Your traffic | A corpus you supplied |
| Incumbent arm | Already answered a user; read, not charged again | Dispatched |
| Cost per item | One dispatch plus a judge call | Two dispatches plus a judge call -- roughly 2x |
| Can be priced before it runs | No. Traffic is not finite. | Yes. A corpus is finite. |
| Verdict may promote | Yes | No |
Promotion is refused from an experiment, in the API and not merely disabled in the interface. Whether evidence may change production routing is a property of the evidence, and a corpus comparison never touched production.
The two objects also refuse each other's fields: a trial may not name a dataset, and an experiment may not carry a replay window or a live pair source.
Why you would run one
A corpus covers cases your traffic does not. If the behaviour you care about happens rarely -- a class of prompt you see once a week, a regression you fixed and never want back -- a trial will wait a long time to see 20 sessions of it. A dataset puts those cases in front of every future candidate on demand.
Setting one up
The door is the dataset detail page: filter the corpus down to what you want to run, then start an experiment carrying that filter.
The form has three parts.
What to compare. The incumbent and the candidate slate. The incumbent is dispatched too -- nobody ever ran these prompts, so there is no production answer to read.
Scope. A routing profile, required. It names what the comparison is about, and promotion into it is still refused.
Hard spend cap. Consent, not advice: reaching it pauses the run and marks the verdict provisional. Left absent, the run is uncapped.
The quote
A finite corpus can be priced before it runs, and the quote is recomputed as you add arms or change the metadata filter.
| Figure | Meaning |
|---|---|
| Items | Items after the metadata filter -- the number you are really approving. |
| Skipped rows | Rows excluded at import. Stated so that "497 of 500" is visible at the point of approval. |
| Arms, dispatches | Dispatches are items multiplied by arms. This is the count the 2x cost is explained by. |
| Inference spend, judge spend, total | Judging is one call per item per candidate -- the judge scores a pair, so it scales with candidates rather than with arms. |
| Charged credentials | Which provider credential each arm will actually be charged to, and under which custody. Never the requested custody, which is always auto. |
| Input and output tokens per item | What the estimate is built from. Input is averaged over a sample of the corpus and converted at roughly four characters per token; output is a fixed assumption of 600 tokens -- nobody has run these inputs, so there is nothing to average. |
| Unpriced models | Named. A quote carrying any of these is a floor rather than a price. |
The quote's "does not exceed your spend cap" answer is not a guarantee when any model in the slate is unpriced. The response tells you which models those are.
The quote is priced through the same rate table the accrual path uses, over the same item count, with skipped rows excluded -- so the quantities approved are the quantities that run. The dollar figure is still a projection: it prices an assumed answer length, not a measured one. It also resolves your credential per provider through the ladder the run will dispatch on, which makes it a preflight as well as a statement -- a missing credential shows up here rather than mid-run.
A quote is billing-grade where your contracted rates resolved, and it says so per arm: contracted, list price, or nothing resolved at all.
Running
Every corpus item is its own session in the score store, which is what makes the 20-session floor mean what it says over a corpus.
Over a slate the incumbent arm is dispatched once per item and its answer is reused across candidates -- both because the answer does not depend on the challenger, and because a slate must judge every candidate against the same reference reply.
Progress is a row-number cursor on the experiment itself, so a resumed pass never re-dispatches both arms over items you already paid for.
Exhausting the corpus completes the run. A live window never ends and a replay's end is reached by its own drain, but no gateway ever sees an experiment -- so the runner that read past the last item is the only thing that can end it.
Where an experiment runs
Always in DevZero's control plane, including on a content-free deployment. A corpus is content you deliberately uploaded and no production traffic is involved, so the residency predicate that decides where a trial runs does not apply.
Reading the result
The Results screen shows experiments in their own table, headed measured on a corpus you supplied, with the rule stated in place: not a shadow of your traffic -- both arms are dispatched, so cost is 2x per item.
Verdict wording differs from a traffic row on purpose:
| Outcome | Wording |
|---|---|
| Candidate scored better | Candidate ahead |
| Inside the tie band | Parity |
| Candidate scored worse | Candidate behind |
A corpus row never says "adoptable". See Judging and verdicts for the thresholds, which are identical to a trial's.
Judging and Verdicts
The composite score, its five dimensions, the session floor, the tie band, and exactly what each verdict entitles you to do.
Datasets
Building a corpus an experiment runs over -- importing a file, the limits, why expected outputs are stored and never scored, and seeding from your own traces.