Datasets

Datasets are curated collections of test cases used as the input for evaluations. Each item in a dataset represents a single test scenario: an input, an expected output, and optional metadata. Evaluations run against datasets to produce objective quality scores for your agents.

Datasets are most commonly built from LLM traces — real conversations captured from agent runs. When an agent handles a conversation, that interaction generates a trace. Traces can be added to datasets directly from the LLM Traces tab in the Command Center (see LLM Traces). This workflow produces datasets grounded in real-world usage patterns rather than hypothetical test cases.

Datasets can also be populated manually or via CSV upload for cases where you want to build test cases from scratch, migrate from another system, or generate test data programmatically.

Scope

A dataset is not scoped to a single agent. The same dataset can be used by multiple evaluations across different agents, making it possible to benchmark multiple agents against the same test cases.

Access control

The canViewDatasets and canViewEvaluations permissions control dataset visibility. Dataset sharing follows the same resource-level sharing model as other Foundry resources.

In this section:

• Creating Datasets
• Managing Dataset Items