Your first model trial: shadow, promote, then arm a watch
Shadow a cheaper model against the one you serve today, read a verdict measured on your own traffic, promote the winner by hand, and arm a watch that flags drift.
16 min readLast reviewed October 19, 2026Verified against gateway chart 0.10.0
What you’ll have at the end
By the end you will have run a cheaper candidate model in the shadow of the model you serve today, read a verdict that was measured on your own traffic rather than a leaderboard, promoted the candidate as the routing default, and armed a watch that keeps the old model running as a sentinel. Nothing in this flow changes what your users receive until you click Promote.
Before you begin
- A team on the Nautili dashboard with at least one routing profile. A trial observes one profile's traffic and promotion writes back to the same profile, so the wizard cannot start without one.
- Live traffic through the gateway on that profile. A verdict needs 20 judged sessions per candidate, so a profile that serves a handful of requests a day will take a while to conclude.
- Both the incumbent and every candidate in
author/slugform, for exampleanthropic/claude-haiku-4.5. Bare names the catalog does not resolve are refused at the gateway with 400invalid_model_id, and the wizard refuses vendor-native names too. - Candidates from an author the trial engine can dispatch to:
anthropic,openaiorgoogle(geminiis accepted as a spelling ofgoogle). The wizard refuses any other author; a trial created through the API with one starts, drops that candidate on every pair as non-dispatchable, and ends as not enough data. - Session ids on your requests. Sampling is per session, so send
X-Dz-Session-Idfrom your client if you want the sample to follow conversations rather than single calls.
What a trial is
A trial is a shadow-arm comparison. For each sampled request, the incumbent's answer is what your user already received. The candidate answers the same input afterwards, off the request path, and a paired judge scores both answers together at a randomized presentation order. The candidate's answer is judged and discarded. Users never see it, and observing the trial cannot delay or change what a caller receives.
That is also why a trial is traffic evidence and an experiment is corpus evidence. On traffic, only the candidate is dispatched and the verdict is a shadow of production, so promotion is available from it. On a corpus you supplied, both arms are dispatched, nobody ever sent those inputs, and promotion is refused from the verdict. The two are never shown in one table, because pairs and items are not the same measure.
Step 1: Answer "On which traffic?"
Open New trial from Trials under Evals. The page is one scroll of four numbered questions with a sticky price receipt on the right. Above the questions, leave What are you trialling? on A single call type. Under 1. On which traffic? pick the Scope (routing profile), leave the source on Traffic as it happens, add any targeting conditions, then set Session sample.
The conditions are composed with AND in order, over eight fields (workload, namespace, endpoint, requested_model, issued_key, invoked_tool, conversation_depth, billing_regime) with two operators, is and is_not. An empty list means every request in the scope. sample_pct runs from 1 to 100 after the conditions and is deterministic per session: two trials at the same rate see the same sessions, and a session is either in or out for its whole life. Sending 0 means not set and resolves to 100.
Step 2: Answer "Which model are you running today?"
Under 2. Which model are you running today? choose the incumbent from the models recently seen on this team, or type one under Or enter a model id. Give the trial a Trial name if you want one; blank defaults to the model pair.
The incumbent keeps serving every request for the whole trial. It is the reference the candidates are judged against, and nothing you do on this page changes it.
Step 3: Answer "Which models do you want to try?"
Under 3. Which models do you want to try? pick candidates from the catalog cards, or add one under Models not in the catalogue in author/slug form. A slate holds up to 8 candidates.
Every candidate is judged against the same served answer, so a slate costs linearly more and dilutes nothing. The 20-session floor is per candidate and is not divided by the slate. The wizard refuses a slate that repeats a model, includes the incumbent, exceeds 8, or names an author the engine cannot dispatch to, and it says which rule it hit.
Step 4: Answer "When should it stop?"
Under 4. When should it stop? drag any of the three sliders. Whichever bound binds first ends the trial. A slider left at zero is not configured.
| Slider | Field | What reaching it does |
|---|---|---|
| SliderStop after sessions | Fieldmax_sessions | What reaching it doesCompletes the trial, counted on the same judged-session denominator the verdict uses. |
| SliderStop after days | Fieldmax_duration_seconds | What reaching it doesCompletes the trial at the next arm poll. |
| SliderHard spend cap | Fieldspend_cap | What reaching it doesPauses the trial with pause_reason: spend_cap and marks the verdict provisional. |
Spend accrues at list price, not your contracted rate. For a team with a negotiated discount the running figure is systematically high, and it is the figure the cap is enforced against. Treat accrued spend as evidence, never as a charge. Evaluation spend is reported beside your metered spend on Spend and is never summed into the total.
The panel Will this conclude? below the sliders says whether these bounds reach the 20-session floor from the observed traffic rate. An affordable trial that ends as not enough data has spent the money anyway, so read it before starting.
Step 5: Read the receipt and start
The sticky receipt on the right reprices as you move any control. It shows Shadow inference / day, Judging / day, Served traffic (always unchanged, stated on purpose), and Estimated total once a bound exists to project over. Models with no resolvable rate are listed by name rather than priced at $0.

The primary button carries the projected total, so the price and the commitment are one gesture. Save draft keeps the configuration in draft without dispatching anything, and reopening it edits the same draft. Start trial saves and moves it to running.
Verify
Open the trial from Trials. While it runs, the progress tiles show Paired turns, Judged pairs, Shadow spend and Judge spend, with observed sessions against your session bound. If you want to confirm that the trial is invisible to callers, send a request on the scoped profile and read the response headers.
The serving path names the upstream, provider and custody that served the incumbent's request; the candidate never appears in a response header or on a serving path, because it never serves. X-Dz-Serving-Path is only returned when the scope profile's Provider choice is the Nautili router; on a delegated profile, the default, read X-Dz-Request-Id and open the request under Traces, where the served model is the incumbent. Once a candidate reaches 20 judged sessions the verdict card on the trial page fills in.
Reading the verdict
Each candidate gets one of four verdict values, trimmed here to the fields that matter.
| Verdict | Shown as | Condition | Adoptable |
|---|---|---|---|
insufficient_data | Shown asNot enough sessions | ConditionFewer than 20 judged sessions for this candidate | AdoptableNo |
degraded | Shown asDegraded | ConditionComposite delta at or below the parity floor, -0.10 | AdoptableNo |
parity | Shown asParity | ConditionComposite delta inside the tie band of 0.10 | AdoptableYes |
improved | Shown asImproved | ConditionComposite delta above the tie band | AdoptableYes |
The composite is the mean of five dimensions, each scored one to five: quality, faithful, concise, instruct and clarity. The card shows the composite delta and the per-dimension deltas, so you can see which dimension moved. The unit is the session, not the pair: a session's pairs count as one observation however many turns it has, which is why the card reports sessions beside pairs.
The tie band is the measured noise floor of this composite, not a comfort margin. Two populations judged on identical treatment differ by about 0.10 from composition alone, so a delta inside the band is parity, not signal. A candidate at or below -0.10 is degraded and never adoptable, whatever its cost delta. Cost alone is never a verdict.
Every verdict carries a judge_version stamp, shown beside the population on the card. A rubric change is a new judge version, never an edit in place, and scores from different judge versions or rubrics are never pooled. Compare uses the absolute judge and its scores can never be read as a delta, so a Compare run can never feed a trial verdict.
Promote the candidate
Promotion is a separate human action. No verdict, no stop condition and no recommendation promotes anything. The Promote button appears only on an adoptable candidate, and the control plane refuses a promotion below the parity floor even if a client sends one.
Click Promote on the candidate. The dialog Promote this candidate states the change as incumbent → candidate and the scope it applies to. Confirm with Promote.
What changes: the candidate becomes the scope profile's Default model, written through the ordinary routing-profile machinery, so field-by-field merging down the ladder and session pinning apply unchanged. Requests that name no model on that profile now resolve to the candidate. The trial records a receipt: which candidate, who, when.
What does not change: requests that name a model explicitly keep that model. Fallbacks, provider ordering and guardrails on the profile stay as they were. The trial itself stays completed with its evidence. The candidate's answers are not replayed to anyone; the next request is the first one it serves.
Arm a watch
Tick Watch this adoption in the same dialog before confirming. A watch keeps the model you replaced running as a sentinel: the promoted model now serves, the old incumbent is the sole candidate, sampled at 2% of sessions, and judged by the same paired judge the trial used. A Monthly budget (USD) is required and must be positive; a zero budget is a refusal, not unlimited. The button reads Promote and watch.
Drift shows on Watches under Evals. Each watch reads collecting until its window reaches 20 judged sessions, then healthy or drifted. The sign is inverted relative to a trial: a watch's delta is adopted minus sentinel, so a negative delta means the model you replaced is scoring better again. A drifted watch names its failing dimensions and offers Review evidence scoped to the window that fired. A firing watch also appears on the Evals Live alert strip and as a marker on the routing profile.
A watch alerts only. It never changes routing. Rolling back is the same explicit promotion on the watch's own candidate. The budget is a monthly window: reaching it pauses the sentinel, and the next window re-arms it. Only one watch stands per scope, and a newer promotion supersedes it.
Options
Replay a past window
When you cannot wait for traffic, run the same trial over traffic you already retained. Under 1. On which traffic? switch the source to Replay a past window and set From and To. Optionally name sessions to replay the conversations that went wrong; the list is always combined with the window, never used instead of it.
Both bounds are required and replay_from must precede replay_to; a half-open window is refused rather than read as everything retained. The window is stored as you chose it and narrowed at read time to your prompt-retention bound, which moves daily. The estimate under the window says how much of it is reachable and why any part is zero. A replay is still a trial: the incumbent already answered, so only the candidate is dispatched, and promotion is available from its verdict. It needs prompt retention on; a live trial works either way. A replay and a live trial on the same team's traffic are mutually exclusive.
Stop early
Any running trial can be stopped or paused from its page. A deliberate pause records pause_reason: user and does not mark the verdict provisional. Stopping completes the trial and keeps everything collected, including a verdict if any candidate passed the floor.
Troubleshooting
| Symptom or message | Cause | What to do |
|---|---|---|
Symptom or messageVerdict chip reads Not enough sessions (insufficient_data) | CauseFewer than 20 judged sessions for that candidate | What to doKeep running, raise Session sample, widen targeting, or raise Stop after sessions. Adding candidates does not help; the floor is per candidate. |
Symptom or message"Paused at its spend cap" and provisional: true | CauseAccrued spend at list price reached spend_cap | What to doEnter New spend cap (USD) above the current cap and click Raise cap & resume. |
| Symptom or message"The trial engine can shadow anthropic, openai, google models" | CauseA candidate from an author the engine cannot dispatch to | What to doRemove that candidate. It would start and collect nothing. |
| Symptom or messageModels not in the catalogue entry rejected | CauseThe id is not in author/slug shape | What to doType it as author/slug, for example openai/gpt-5.6-luna. |
| Symptom or message"A slate caps at 8 candidates." | CauseMore than 8 candidates | What to doRemove candidates or run a second trial. |
| Symptom or message"A model cannot trial against itself" | CauseThe incumbent is in the slate | What to doRemove it from the candidates. |
| Symptom or message"Your team has no routing profile yet" | CauseNo scope to observe | What to doCreate one under Routing, then return to New trial. |
| Symptom or message"No stop condition set" on the receipt | CauseAll three sliders at zero | What to doSet at least one bound; without one there is no horizon to project a total over. |
| Symptom or message"Prompt retention is off for this team, so nothing was kept to replay." | CauseReplay needs stored bodies | What to doA team admin turns retention on under Traces. It keeps traffic from that moment; it cannot recover the past window. |
| Symptom or message"This whole window predates your retention bound." | CauseThe window ends before retention started | What to doMove From and To inside the reachable bound the estimate reports. |
| Symptom or message"Promotion refused" | CauseCandidate is not adoptable, or the trial is an experiment | What to doOnly parity or improved with 20 or more sessions promotes, and only from a trial. |
| Symptom or message"A watch needs a positive budget." | CauseMonthly budget (USD) empty or zero | What to doEnter a positive amount; zero is not unlimited. |
| Symptom or messagePromoted — but the watch did not arm toast | CauseRouting changed, watch creation failed | What to doThe promotion is live. Check watch_error and re-arm from the trial. |
Symptom or messageWatch reads collecting for a long time | Cause2% of sessions has not reached 20 judged sessions | What to doExpected on low-traffic scopes. The watch will not guess below the floor. |
Next steps
- Experiments, datasets and Compare, the guide for the cases your traffic does not cover, with the understanding that a corpus verdict never promotes.
- Routing profiles and auto mode, the guide for where the promoted default model sits in the merge ladder and how session pinning applies.
- Experiments, the reference for corpus runs.
- Routing profiles, the reference for the merge ladder.
- Trials reference, for the full lifecycle, progress counters and workflow trials.
- Judging and verdicts, for the frontier, the recommendation and the readable evidence behind each pair.
- Watches, for the monthly window, supersession and the inverted sign convention.