extractingbench

How the benchmark works

This is a summary of the benchmark design, version 0.3 of 20 Oct 2026. The design is not final until it is approved, and no model has been run on a test item yet. When the first run is published, this page will link the signed preregistration that fixed every rule below before any result existed.

What is tested

Each test item is one document (text, page images, or both), a list of fields to fill, and a checked answer for every field. About two thirds of the fields are printed in the document, about one in five is missing (the right answer is null), and the rest have to be computed, in one step or several.

There are six document types, each its own dataset:

  • Receipts, as photos and text
  • Invoices
  • US government forms and notices
  • Emails and plain text
  • Tables and financial figures
  • Documents where most of the requested fields are missing

Public and private test sets

Every document type has a public half, which anyone can download and rerun, and a private half, which we keep back so no model can be trained on it. The private half is built from documents we generate and from government documents published after the test set is frozen. Before each run we publish a signed, timestamped commitment for every private item. When a private set is retired, about every six months, we publish its items so anyone can check the commitments, and a new private set takes its place.

The two halves are built to the same blueprint and checked for balance on length, number of fields and other measurable features. Each model's score on both halves is shown, so a model that does much better on the public half than the private half stands out.

How the answers are checked

For documents we generate, the answers come from the data the document was generated from. For real documents, a paid human labeller fills every field, a second labeller independently labels a 30% sample and every missing or computed field in it, and a third person settles every disagreement without knowing who said what. We publish the agreement rate and the estimated error rate for each document type. Machine checks catch arithmetic and formatting slips. No model ever sets an answer.

Scoring

  • Money must match to the cent. Counts must match exactly. There is no percentage tolerance.
  • Dates must be the full correct date. Every model gets the same reference date, so a date without a year resolves the same way.
  • Numbers are read with the document's own locale, so 1.234,56 in a German invoice is 1234.56.
  • Text is compared after case and whitespace are normalized. Identifiers keep their hyphens and leading zeros.
  • Lists are compared as sets of items; order is reported separately.
  • A value returned for a missing field is an invented value. A null returned for a field that is there is a false null.

The headline index

Extracting Index = 100 × (0.75 × accuracy + 0.25 × null honesty), on the private test set, with text input. Accuracy is the share of printed fields read right. Null honesty is the share of missing fields returned as null. Every document type counts equally. The index is small on purpose: every other measurement has its own page.

Intervals and winners

Every number has a 95% interval from 10,000 bootstrap resamples of whole documents. A model is called better than another only when the interval on their difference excludes zero after correcting for the number of comparisons. Otherwise the page says the two can't be told apart.

How the results are checked

  • Every request and response is stored. Responses for the public set are published.
  • Two scorers, written independently in two languages, must agree on every verdict before anything is published.
  • A person checks 50 random verdicts per model against the raw response, and we publish the scorer error rate.
  • A sample of documents is run three times to check answers don't change between identical requests.
  • A fixed block of anchor items runs in every round, so a silent change to a hosted model shows up.

Providers, speed and cost

Each model is tested on every provider that serves it. Latency is timed from one server, network included. Cost per 1,000 documents is the list price applied to the usage each API reports. Models we run ourselves are reported separately and never ranked against hosted providers on speed or cost.

Rerun it yourself

The public test set, the harness, both scorers and our raw responses for the public set will be in the public repository when results are published. One command runs a model on the public set; another rescores our responses offline with no keys.