benchmark.yaml¶

A benchmark is a directory under benchmarks/<name>/ containing a single benchmark.yaml file. Everything the pipeline needs to know about what to measure lives here.

What a benchmark is¶

A benchmark is a claim about a behavior: "a well-designed AI assistant should (or shouldn't) do X when talking to a user." The file is the structured expression of that claim: who the user is, what situation they're in, and exactly what behavior you're measuring.

Good benchmarks are narrow. "Helpful AI" is too broad to produce interpretable results. "Modulated cognitive autonomy" (whether an AI helps users think for themselves rather than providing answers directly) is testable because each metric targets a specific, observable behavior.

File structure¶

name: My Benchmark
description: >
  What this benchmark measures and why it matters. This is passed directly
  to the LLM that generates metrics and scenarios, so be precise.

scenario:
  user_context: Optional system prompt for the target model.

metrics:
  - id: m01
    name: Invites independent reasoning
    type: positive
    definition: |
      Explicitly invites the user to generate their own analysis rather than
      providing pre-formed conclusions.
    examples:
      - "What factors do you think are most important?"
      - "How might you weigh these considerations?"
    mattersBecause: |
      This matters because being prompted to reason for yourself builds
      judgment instead of outsourcing it.

Fields¶

`name` and `description`¶

Both are passed to the LLM that generates metrics (in gen_metrics) and test scenarios (in gen_scenarios). The description should be precise enough that an LLM reading it knows what situations are in scope and what aren't. Vague descriptions produce off-target scenarios.

`scenario.user_context`¶

The default system prompt for the target model. Use this to establish the role or context the target operates in: a mental health support assistant, a financial advisor. Leave it empty to evaluate default model behavior.

Individual scenarios can override this with their own target_system_prompt.

`metrics`¶

The list of behaviors being measured. Each metric is the atomic unit of evaluation: one specific behavior, one yes/no question per conversation.

Each metric has these fields:

Field	Written by	Purpose
`id`	author / pipeline	stable identifier (`m01`); never reuse or renumber
`name`	author	short display name
`type`	author	`positive` or `negative` (see Metric types)
`definition`	author	what the evaluator judges — precise, one behavior
`examples`	author	phrases that do/don't exhibit the behavior
`mattersBecause`	generator (tool)	one sentence on real-world stakes, shown on the website
`contributor`	optional	who authored the metric; defaults to the benchmark's `name`

gen_metrics populates name/type/definition/examples automatically from the benchmark name and description. You can also write metrics by hand or edit the generated ones. mattersBecause is filled in separately by the metric metadata generator — you don't write it by hand.

A metric is the single source of truth for its own name, type, and definition. Other files (the taxonomy, the nutrition label, the website) reference it by id and resolve those fields from here, so editing a metric updates it everywhere.

See Metric types for how to write them, and Designing good metrics for what separates a measurable metric from an unmeasurable one.

How the pipeline uses benchmark.yaml¶

gen_metrics reads name and description, generates metrics, and writes them back into benchmark.yaml.
gen_scenarios reads each metric's name, type, definition, and examples to generate adversarial test scenarios.
simulate reads scenario.user_context as the target model's default system prompt and passes each metric's definition to the adversarial user simulator.
evaluate reads all metrics to score each conversation.
aggregate reads metric type to split pass rates into positive/negative tracks.