Skip to content

How publishing works

You don't publish anything by hand. When your pull request merges to main, a GitHub Action builds the site's data and uploads it automatically, usually live within a minute or two. The site reads its data at runtime, so there's no separate deploy to wait on.

merge to main  →  GitHub Action builds + uploads  →  Cloudflare R2  →  live on the website

The Action (in the impactbench-data repo, under .github/workflows/) runs a script that reshapes the benchmark files into the compact data files the website reads, and uploads them to a Cloudflare R2 bucket. The website fetches those files from R2 at runtime; nothing is baked into the site build, which is why a data change goes live without redeploying the site. The Action runs on every merge to main, and can also be started by hand from the repo's Actions tab if you ever need to re-publish without a new commit.

Why the data is compiled

The benchmark files are laid out for authoring. A single metric's data is spread across its benchmark.yaml, scenarios.json, and every model's scores.json, and a full set of runs is thousands of files totalling tens of megabytes. A browser can't practically load all of that and join it on the fly.

So the build reshapes the source into files laid out for reading: one small file per view the site renders, with the joins already done. Each page then fetches a single file instead of stitching together raw data. It also strips out everything the site doesn't need (raw transcripts stay in their own files, loaded only when opened), which keeps the up-front download small.

data/taxonomy.json          areas > subareas > metrics
data/models.json            the model list and which pages each appears on
data/benchmark-data.json    every model's score on every metric
data/scenario-index.json    per metric: its scenarios and each model's verdict
data/metric-details.json    per metric: definition, contributor, "why this matters"
data/nutrition.json         the nutrition-label categories and their scores

scenarios/<benchmark>/<model>/<scenario-id>.json
                            one full conversation, loaded only when someone
                            opens that specific transcript

The six data/ files load up front; the per-conversation files (there are a lot of them) are fetched on demand.

Checking that your change went live

If an edit doesn't show up, look at the Actions tab in the impactbench-data repo for the run triggered by your merge:

  • Green run. Your change published. If it still looks wrong on the site, it's usually a browser cache; hard-refresh the page.
  • Green run with warnings. The change published, but something needs attention. The most common one: a new metric was added to a benchmark.yaml but not placed in taxonomy.json, so it's live in the data but not shown on the site. Add it to a group in taxonomy.json.
  • Red run. The data was rejected and nothing was published. This happens when the taxonomy or nutrition label points at a metric that no benchmark.yaml defines, or models.yaml is broken. The run log names the problem; fix it and push again.

Previewing before you open a PR

If you want to see what your edit produces without merging, you can run the build locally (nothing is uploaded):

# from the impactbench-data repo root, with boto3 and pyyaml installed
python tools/publish.py --dry-run

It writes the built files to tools/_out/ so you can inspect them.