Datasets

A dataset is a stored corpus an experiment runs over. It is a first-class, team-owned object -- not a file attached to a run.

The reason it is an object is the use case: the same corpus is meant to be evaluated repeatedly against successive candidate models, and a verdict months later has to be able to say what it was measured on.

What an item is

Metadata is what lets one corpus serve several evaluations instead of being forked into near-identical copies. Filtering is an exact match on keys and values, combined with AND, and the dataset detail page drives it from server-supplied facets -- so the choices you are offered are what the corpus actually contains, and the matching count is the number an experiment over that filter would run.

Importing a file

The import screen is a single page with no steps, deliberately: a stepped wizard puts the column mapping behind a screen you have already left by the time you doubt it. Here the file, what was read from it, and the correction path are all in view at once.

The flow is two calls, and both are stated on screen.

First, the file is analysed and nothing is stored. The screen states its reading as a sentence you confirm rather than four dropdowns you fill -- for example: we read prompt as the input, golden_sql as an expected output (stored, never scored), and dialect as filterable metadata.

Every column gets a role and a reason, and a confidence:

Then the import is approved, and the mapping is sent back explicitly whether or not you changed it: the import must honour what was on screen when you approved it, so a later change to detection cannot silently import a file differently than the page said it would.

What is accepted

CSV, TSV and JSONL, including Braintrust exports -- whose top-level metadata object is flattened one level -- and promptfoo's double-underscore control columns, of which only __expected carries data.

Accepted extensions: .csv, .tsv, .txt, .jsonl, .ndjson, .json. Anything else is sniffed: a leading { reads as JSONL, everything else as comma-delimited.

Header matching is done after normalising -- lowercased, punctuation and spaces removed -- so expected_output, expectedOutput and promptfoo's __expected all match the same role.

Limits

The import screen states its rules before you choose a file, so these are enforced server-side and held against the interface by a test that fails the build if the two drift.

Whole-file bounds fail the upload; per-row bounds skip the row.

A column becomes filterable metadata only while its values stay short and few -- at most 64 characters and 20 distinct values. Longer or more varied than that reads as free text and is dropped.

Rows with an empty input are skipped and counted rather than blocking the import. The count you see at approval is the count that imports, because detection and the build share one code path.

The list and the detail

The Datasets list shows, per corpus: name, origin, items, expected outputs, skipped, and when it was imported. Both the corpus list and the item list are keyset-paginated rather than offset-paginated, because the list is newest-first and an import inserts at the front -- an offset would show you a duplicate and hide the row it displaced.

The detail page filters the corpus by the dimensions it carries, and says so plainly when it carries none. Its item list reports the total matching the current filter, because paging must not appear to shrink the number an experiment would be quoted for.

Origin: where the items came from

Origin is recorded and displayed, because an imported set may or may not resemble production, where a seeded one does by construction. Appending traces to an imported corpus does not rewrite its origin -- an imported corpus with traffic added is still what it says it is.

Seeding from traces

This is the one route by which production traffic becomes a corpus: a regression you saw in the trace explorer becomes a permanent test case, re-run against every future candidate.

Three rules govern it, and none of them is expressed by this path itself -- each is the same rule enforced elsewhere, called through the same code:

Residency. Seeding is refused outright unless every gateway installation your team runs reports content uplink. Promoting retained traffic through DevZero's control plane would move production content past the very setting that keeps it in your network. The refusal is explicit and names its reason, because a write that silently promoted nothing would read as success.

Access. Only traces you could have read anyway may be promoted -- otherwise promotion would be a laundering route: copy a colleague's prompt into a corpus and read it back off the dataset page. The retention window is inside that rule already.

Redaction visibility. Per-candidate redaction outcomes are shown -- which detector categories fired, categories only and never the matched text.

Two more properties worth knowing before you promote anything:

• Promoted items carry no expected output at all. Production traffic supplies no reference answer, and storing the incumbent's reply as one would pin today as correct.
• A trace whose only prompt text is the stored 256-character excerpt is flagged, because the promoted case is truncated.

A preview states what a selection would produce and stores nothing, exactly as file analysis is to file import -- and it carries the refusal as a stated reason, so the action can be shown disabled with an explanation instead of vanishing unexplained. A selection past the server's bound is refused rather than silently truncated.