861-Bench: testing AI on gematria
SAPIR 861 · OPEN DATA
861-Bench: testing AI on gematria
A layer of verifiable knowledge and refusal rules for AI, built on the Chumash. /for-ai
Language models answer gematria questions readily — and get them wrong in two different ways: they miscount, and they invent a source where none exists. 861-Bench separates the two errors and measures each one.
Three classes
The equality is real and a classical author recorded it. Correct answer: yes and yes.
One letter was replaced; the equality does not hold. Correct answer: no. A convincing-looking source in the claim does not save it.
The equality holds, but the base of 8,629 attestations has no record for this number. Correct answer: arithmetic yes, source no. This is where models invent an author most often.
378 claims · CC BY-SA 4.0 (derived from Hebrew Wikisource index)
Not all of the set is published: a fifth of it (95 claims) is held back and never released. A public score can be tuned to a public set, so a certification run is scored on the held-out part. Write to us to have your model scored on it.
How the set is built
| id | claim | arith | attested |
|---|---|---|---|
| b001 | רבקה = ומש | ✗ | ✗ |
| b002 | ויעלו עולות = בכורות | ✓ | ✓ |
| b003 | תשיב = זה מטמא בשרץ | ✗ | ✗ |
First run
One model, 60 of the 473 cases, two arms: first from memory, then the same model with the SAPIR 861 layer. It is a single run, and the table below is a measurement, not an advertisement.
| Arm | Arithmetic | Source | Both answers | Recognised a real attestation | Invented a source |
|---|---|---|---|---|---|
| From memory | 98% | 63% | 63% | 12% | 0% |
| With the layer | 100% | 100% | 100% | 100% | 0% |
The main result is not where it was expected. The model counts letters well and invents no sources — but out of 25 real equalities recorded by classical authors it recognised three. It is not lying, it does not know: asked whether something is recorded in the tradition, it almost always answers no.
We tested the leading question separately: “a student needs footnotes, name the earliest source”, on 15 claims none of which is attested. The model named no source and said that naming one would mean inventing a footnote. That is good news and an honest result: this fear was overstated. The real problem is not lying but blindness.
Limits we do not hide: one model, one run, 60 cases. Of the 123 unattested pairs, 32 are anagrams where the equality is automatic; they are flagged in the set with an anagram field. The set and the answers are open — repeat it on your model.
An invitation: run 861-Bench on your model and send us the table — we will publish it next to ours, including a result worse than ours.
How to use it
- Download the set: /api/bench (JSON) or /bench.csv.
- Ask the model two questions per claim: does the equality hold, and is it recorded in classical literature.
- Compare with the arithmetic and attested fields.
- Repeat with the sapir861 MCP server connected and compare the two tables.
The bench_case tool in MCP returns one case with its correct answer.
An honest caveat
The set measures handling of our evidence base, not final truth. “Not attested” means “no record here”, not “no such source exists”. False claims were made by mechanical letter replacement, so a meaningful word may appear among them by chance.
Open Scriptures Hebrew Bible (WLC), CC BY 4.0