Skip to content

Your first model trial: shadow, promote, then arm a watch

Shadow a cheaper model against the one you serve today, read a verdict measured on your own traffic, promote the winner by hand, and arm a watch that flags drift.

16 min readLast reviewed October 19, 2026Verified against gateway chart 0.10.0

What you’ll have at the end

By the end you will have run a cheaper candidate model in the shadow of the model you serve today, read a verdict that was measured on your own traffic rather than a leaderboard, promoted the candidate as the routing default, and armed a watch that keeps the old model running as a sentinel. Nothing in this flow changes what your users receive until you click Promote.

Before you begin

  • A team on the Nautili dashboard with at least one routing profile. A trial observes one profile's traffic and promotion writes back to the same profile, so the wizard cannot start without one.
  • Live traffic through the gateway on that profile. A verdict needs 20 judged sessions per candidate, so a profile that serves a handful of requests a day will take a while to conclude.
  • Both the incumbent and every candidate in author/slug form, for example anthropic/claude-haiku-4.5. Bare names the catalog does not resolve are refused at the gateway with 400 invalid_model_id, and the wizard refuses vendor-native names too.
  • Candidates from an author the trial engine can dispatch to: anthropic, openai or google (gemini is accepted as a spelling of google). The wizard refuses any other author; a trial created through the API with one starts, drops that candidate on every pair as non-dispatchable, and ends as not enough data.
  • Session ids on your requests. Sampling is per session, so send X-Dz-Session-Id from your client if you want the sample to follow conversations rather than single calls.

What a trial is

A trial is a shadow-arm comparison. For each sampled request, the incumbent's answer is what your user already received. The candidate answers the same input afterwards, off the request path, and a paired judge scores both answers together at a randomized presentation order. The candidate's answer is judged and discarded. Users never see it, and observing the trial cannot delay or change what a caller receives.

That is also why a trial is traffic evidence and an experiment is corpus evidence. On traffic, only the candidate is dispatched and the verdict is a shadow of production, so promotion is available from it. On a corpus you supplied, both arms are dispatched, nobody ever sent those inputs, and promotion is refused from the verdict. The two are never shown in one table, because pairs and items are not the same measure.

Step 1: Answer "On which traffic?"

Open New trial from Trials under Evals. The page is one scroll of four numbered questions with a sticky price receipt on the right. Above the questions, leave What are you trialling? on A single call type. Under 1. On which traffic? pick the Scope (routing profile), leave the source on Traffic as it happens, add any targeting conditions, then set Session sample.

{
"scope_profile_id": "<profile id>",
"pair_source": "live",
"targeting_conditions": [
{ "field": "workload", "op": "is", "values": ["support-bot"] }
],
"sample_pct": 25
}

The conditions are composed with AND in order, over eight fields (workload, namespace, endpoint, requested_model, issued_key, invoked_tool, conversation_depth, billing_regime) with two operators, is and is_not. An empty list means every request in the scope. sample_pct runs from 1 to 100 after the conditions and is deterministic per session: two trials at the same rate see the same sessions, and a session is either in or out for its whole life. Sending 0 means not set and resolves to 100.

Step 2: Answer "Which model are you running today?"

Under 2. Which model are you running today? choose the incumbent from the models recently seen on this team, or type one under Or enter a model id. Give the trial a Trial name if you want one; blank defaults to the model pair.

Incumbent model anthropic/claude-opus-4
Trial name support-bot: opus vs haiku

The incumbent keeps serving every request for the whole trial. It is the reference the candidates are judged against, and nothing you do on this page changes it.

Step 3: Answer "Which models do you want to try?"

Under 3. Which models do you want to try? pick candidates from the catalog cards, or add one under Models not in the catalogue in author/slug form. A slate holds up to 8 candidates.

Which models do you want to try? (2 selected)
anthropic/claude-haiku-4.5
openai/gpt-5.6-luna

Every candidate is judged against the same served answer, so a slate costs linearly more and dilutes nothing. The 20-session floor is per candidate and is not divided by the slate. The wizard refuses a slate that repeats a model, includes the incumbent, exceeds 8, or names an author the engine cannot dispatch to, and it says which rule it hit.

Step 4: Answer "When should it stop?"

Under 4. When should it stop? drag any of the three sliders. Whichever bound binds first ends the trial. A slider left at zero is not configured.

{
"max_sessions": 200,
"max_duration_seconds": 604800,
"spend_cap": { "currency_code": "USD", "units": "40" }
}
SliderStop after sessionsFieldmax_sessionsWhat reaching it doesCompletes the trial, counted on the same judged-session denominator the verdict uses.
SliderStop after daysFieldmax_duration_secondsWhat reaching it doesCompletes the trial at the next arm poll.
SliderHard spend capFieldspend_capWhat reaching it doesPauses the trial with pause_reason: spend_cap and marks the verdict provisional.

Spend accrues at list price, not your contracted rate. For a team with a negotiated discount the running figure is systematically high, and it is the figure the cap is enforced against. Treat accrued spend as evidence, never as a charge. Evaluation spend is reported beside your metered spend on Spend and is never summed into the total.

The panel Will this conclude? below the sliders says whether these bounds reach the 20-session floor from the observed traffic rate. An affordable trial that ends as not enough data has spent the money anyway, so read it before starting.

Step 5: Read the receipt and start

The sticky receipt on the right reprices as you move any control. It shows Shadow inference / day, Judging / day, Served traffic (always unchanged, stated on purpose), and Estimated total once a bound exists to project over. Models with no resolvable rate are listed by name rather than priced at $0.

The New trial wizard with the scope, traffic source and session sample set, and the sticky receipt showing shadow inference, judging, served traffic and the estimated total

Shadow inference / day $3.10 Candidate responses, judged then discarded
Judging / day $1.45 One paired verdict per sampled request per candidate
Served traffic unchanged The incumbent keeps serving exactly as before
Estimated total $31.85 the session count binds first, ~7.0 days
[ Start trial · $31.85 ] [ Save draft ]
Stop any time — you keep the evidence collected so far. Auto-pauses at $40.00.

The primary button carries the projected total, so the price and the commitment are one gesture. Save draft keeps the configuration in draft without dispatching anything, and reopening it edits the same draft. Start trial saves and moves it to running.

Verify

Open the trial from Trials. While it runs, the progress tiles show Paired turns, Judged pairs, Shadow spend and Judge spend, with observed sessions against your session bound. If you want to confirm that the trial is invisible to callers, send a request on the scoped profile and read the response headers.

curl -s -D - -o /dev/null https://<your-gateway-host>/v1/chat/completions \
-H "Authorization: Bearer $NAUTILI_KEY" \
-H "Content-Type: application/json" \
-H "X-Dz-Session-Id: sess-7f3a" \
-H "X-Dz-Workload-Type: support-bot" \
-d '{"model":"anthropic/claude-opus-4","messages":[{"role":"user","content":"Where is my order?"}]}'
X-Dz-Request-Id: req_01HZ...
X-Dz-Serving-Path: anthropic-direct:anthropic@managed

The serving path names the upstream, provider and custody that served the incumbent's request; the candidate never appears in a response header or on a serving path, because it never serves. X-Dz-Serving-Path is only returned when the scope profile's Provider choice is the Nautili router; on a delegated profile, the default, read X-Dz-Request-Id and open the request under Traces, where the served model is the incumbent. Once a candidate reaches 20 judged sessions the verdict card on the trial page fills in.

Reading the verdict

Each candidate gets one of four verdict values, trimmed here to the fields that matter.

{
"judge_version": "v1-sonnet-4-6-trial-paired",
"provisional": false,
"parity_floor": -0.1,
"min_sessions_for_verdict": 20,
"candidates": [
{
"candidate_model": "anthropic/claude-haiku-4.5",
"sessions": 212,
"judged_pairs": 611,
"quality_delta": -0.04,
"cost_delta_pct": -71.2,
"verdict": "parity",
"adoptable": true
}
]
}
VerdictShown asConditionAdoptable
insufficient_dataShown asNot enough sessionsConditionFewer than 20 judged sessions for this candidateAdoptableNo
degradedShown asDegradedConditionComposite delta at or below the parity floor, -0.10AdoptableNo
parityShown asParityConditionComposite delta inside the tie band of 0.10AdoptableYes
improvedShown asImprovedConditionComposite delta above the tie bandAdoptableYes

The composite is the mean of five dimensions, each scored one to five: quality, faithful, concise, instruct and clarity. The card shows the composite delta and the per-dimension deltas, so you can see which dimension moved. The unit is the session, not the pair: a session's pairs count as one observation however many turns it has, which is why the card reports sessions beside pairs.

The tie band is the measured noise floor of this composite, not a comfort margin. Two populations judged on identical treatment differ by about 0.10 from composition alone, so a delta inside the band is parity, not signal. A candidate at or below -0.10 is degraded and never adoptable, whatever its cost delta. Cost alone is never a verdict.

Every verdict carries a judge_version stamp, shown beside the population on the card. A rubric change is a new judge version, never an edit in place, and scores from different judge versions or rubrics are never pooled. Compare uses the absolute judge and its scores can never be read as a delta, so a Compare run can never feed a trial verdict.

Promote the candidate

Promotion is a separate human action. No verdict, no stop condition and no recommendation promotes anything. The Promote button appears only on an adoptable candidate, and the control plane refuses a promotion below the parity floor even if a client sends one.

Click Promote on the candidate. The dialog Promote this candidate states the change as incumbent → candidate and the scope it applies to. Confirm with Promote.

{
"candidate_model": "anthropic/claude-haiku-4.5",
"arm_watch": false
}

What changes: the candidate becomes the scope profile's Default model, written through the ordinary routing-profile machinery, so field-by-field merging down the ladder and session pinning apply unchanged. Requests that name no model on that profile now resolve to the candidate. The trial records a receipt: which candidate, who, when.

What does not change: requests that name a model explicitly keep that model. Fallbacks, provider ordering and guardrails on the profile stay as they were. The trial itself stays completed with its evidence. The candidate's answers are not replayed to anyone; the next request is the first one it serves.

Arm a watch

Tick Watch this adoption in the same dialog before confirming. A watch keeps the model you replaced running as a sentinel: the promoted model now serves, the old incumbent is the sole candidate, sampled at 2% of sessions, and judged by the same paired judge the trial used. A Monthly budget (USD) is required and must be positive; a zero budget is a refusal, not unlimited. The button reads Promote and watch.

{
"candidate_model": "anthropic/claude-haiku-4.5",
"arm_watch": true,
"watch_monthly_budget": { "currency_code": "USD", "units": "25" }
}

Drift shows on Watches under Evals. Each watch reads collecting until its window reaches 20 judged sessions, then healthy or drifted. The sign is inverted relative to a trial: a watch's delta is adopted minus sentinel, so a negative delta means the model you replaced is scoring better again. A drifted watch names its failing dimensions and offers Review evidence scoped to the window that fired. A firing watch also appears on the Evals Live alert strip and as a marker on the routing profile.

A watch alerts only. It never changes routing. Rolling back is the same explicit promotion on the watch's own candidate. The budget is a monthly window: reaching it pauses the sentinel, and the next window re-arms it. Only one watch stands per scope, and a newer promotion supersedes it.

Options

Replay a past window

When you cannot wait for traffic, run the same trial over traffic you already retained. Under 1. On which traffic? switch the source to Replay a past window and set From and To. Optionally name sessions to replay the conversations that went wrong; the list is always combined with the window, never used instead of it.

{
"pair_source": "replay",
"replay_from": "2026-09-22T00:00:00Z",
"replay_to": "2026-10-06T00:00:00Z",
"replay_session_ids": ["sess-7f3a", "sess-91c0"]
}

Both bounds are required and replay_from must precede replay_to; a half-open window is refused rather than read as everything retained. The window is stored as you chose it and narrowed at read time to your prompt-retention bound, which moves daily. The estimate under the window says how much of it is reachable and why any part is zero. A replay is still a trial: the incumbent already answered, so only the candidate is dispatched, and promotion is available from its verdict. It needs prompt retention on; a live trial works either way. A replay and a live trial on the same team's traffic are mutually exclusive.

Stop early

Any running trial can be stopped or paused from its page. A deliberate pause records pause_reason: user and does not mark the verdict provisional. Stopping completes the trial and keeps everything collected, including a verdict if any candidate passed the floor.

Troubleshooting

Symptom or messageVerdict chip reads Not enough sessions (insufficient_data)CauseFewer than 20 judged sessions for that candidateWhat to doKeep running, raise Session sample, widen targeting, or raise Stop after sessions. Adding candidates does not help; the floor is per candidate.
Symptom or message"Paused at its spend cap" and provisional: trueCauseAccrued spend at list price reached spend_capWhat to doEnter New spend cap (USD) above the current cap and click Raise cap & resume.
Symptom or message"The trial engine can shadow anthropic, openai, google models"CauseA candidate from an author the engine cannot dispatch toWhat to doRemove that candidate. It would start and collect nothing.
Symptom or messageModels not in the catalogue entry rejectedCauseThe id is not in author/slug shapeWhat to doType it as author/slug, for example openai/gpt-5.6-luna.
Symptom or message"A slate caps at 8 candidates."CauseMore than 8 candidatesWhat to doRemove candidates or run a second trial.
Symptom or message"A model cannot trial against itself"CauseThe incumbent is in the slateWhat to doRemove it from the candidates.
Symptom or message"Your team has no routing profile yet"CauseNo scope to observeWhat to doCreate one under Routing, then return to New trial.
Symptom or message"No stop condition set" on the receiptCauseAll three sliders at zeroWhat to doSet at least one bound; without one there is no horizon to project a total over.
Symptom or message"Prompt retention is off for this team, so nothing was kept to replay."CauseReplay needs stored bodiesWhat to doA team admin turns retention on under Traces. It keeps traffic from that moment; it cannot recover the past window.
Symptom or message"This whole window predates your retention bound."CauseThe window ends before retention startedWhat to doMove From and To inside the reachable bound the estimate reports.
Symptom or message"Promotion refused"CauseCandidate is not adoptable, or the trial is an experimentWhat to doOnly parity or improved with 20 or more sessions promotes, and only from a trial.
Symptom or message"A watch needs a positive budget."CauseMonthly budget (USD) empty or zeroWhat to doEnter a positive amount; zero is not unlimited.
Symptom or messagePromoted — but the watch did not arm toastCauseRouting changed, watch creation failedWhat to doThe promotion is live. Check watch_error and re-arm from the trial.
Symptom or messageWatch reads collecting for a long timeCause2% of sessions has not reached 20 judged sessionsWhat to doExpected on low-traffic scopes. The watch will not guess below the floor.

Next steps

Ready to keep going?

Explore the next guide or start routing through Nautili.