Editing a benchmark¶

The benchmark data shown on impactbench.media.mit.edu lives in the impactbench-data repository. Editing a metric, a scenario, or the way things are grouped is a change to that repo, submitted as a pull request. When it merges to main, a GitHub Action rebuilds the site's data and uploads it (see How publishing works).

Getting access¶

The data repo is private. Ask Pub for an invite to chayapatr/impactbench-data on GitHub before you start.

The repository¶

Two kinds of file live in the repo:

# benchmark data: one folder per benchmark
<benchmark>/
  benchmark.yaml        the benchmark: its metrics and their definitions
  scenarios.json        generated scenarios
  runs/<model>/...      each model's conversations and scores

# web metadata: top-level, hand-edited
taxonomy.json           how metrics are grouped for display (areas > subareas > groups)
nutrition-label.json    which metrics make up each nutrition-label category
models.yaml             model identity, display name, and which pages it appears on

Only benchmark.yaml and the three web-metadata files are meant to be edited by hand. Everything under runs/ is produced by the pipeline.

Which file to edit¶

I want to…	Edit
Change a metric's name, definition, or type	a metric in `benchmark.yaml`
Add or reword a metric	the `metrics` list in `benchmark.yaml`
Change the "why this matters" text	`mattersBecause` on the metric in `benchmark.yaml`
Move a metric to a different area / subarea / group	`taxonomy.json`
Rename an area, subarea, or group	`taxonomy.json`
Change which metrics are in a nutrition category	`nutrition-label.json`
Add a model or change its display name / pages	`models.yaml`

A metric's name, type, and definition are defined once, in its benchmark.yaml. The taxonomy and nutrition label reference the metric by id, so you don't repeat those fields there. Edit the metric and the change flows through to every page.

For what each metric field means, see benchmark.yaml; for how to choose positive vs negative, see Metric types; and for writing a definition that's actually measurable, see Designing good metrics.

New metrics need two edits

Adding a metric to benchmark.yaml is not enough for it to appear on the site. Its id also has to be placed in a group in taxonomy.json. A metric absent from the taxonomy is published but not shown (the publish step warns about it).

Submitting a change¶

Create a branch and make your edit.
Open a pull request against main.
Have it reviewed and merged.

On merge, the publish Action runs automatically and the change is live within a minute or two. The website reads the data at runtime, so there is no separate site deploy. See How publishing works for what happens after the merge.