Visual recognition · the Antilibrary

The best case: how well AI reads a tidy shelf

Prior tests used hard photos. This one hands the models the easy case — neat shelves, sharp pictures — and asks whether the tiers still separate.

Earlier in this arc we learned that on hard inputs the photograph matters more than the model, and that an OCR pre-step is dead weight for a capable reader. So we removed the difficulty. These six frames are well-organized and shot straight-on. The question: on the best case, how well do the best models read a shelf — and do the tiers converge? The hypothesis, stated up front: they cluster near-perfect, and the only real signal left is the small residual — a mis-bound author, a rare invented book, a title dropped off a flat stack.

01How the test was kept honest

This test grades Claude models among others, so the ground truth could not be model-produced. Bharat verified the books on each physical shelf by hand; the answer keys are human. The running assistant only standardized the photos, called the APIs, and scored the outputs — it never identified a book itself. Every model call was stateless, temperature 0, prompt caching off, one call per condition per image, same minimal prompt throughout. Scoring is a deterministic fuzzy string match — no model judges another model. Photos were standardized to 2000 px long-edge before any run.

02Six reading conditions

Only the model and the path vary. Four read the photo directly; two read PaddleOCR text instead, to isolate the OCR effect on a strong reader and a weak one.

03Read it yourself — every engine on every shelf

Rows are the six shelves (with the human-validated book count); columns are the six engines. Each cell shows books found of the total, tinted by how well it did. Gray −N = books missed; gray +N = books invented. Click any cell to see the exact titles it missed or made up; click a shelf to jump to its photo.

04The six shelves

Presentation varies on purpose — upright spines read sideways, flat stacks read left-to-right, and one frame spans two shelves. Some frames overlap (two shots of one shelf; one frame containing another), noted in each caption so books aren't double-counted.

05What the numbers say

Recall = of the real books, how many were found. Precision = of what was reported, how much was real. Garble = a distorted version of a real title; Made up = an invented book unlike anything on the shelf — the trust signal, kept separate.

06What we found

1. The tiers converge. The four capable direct readers all land at 93–98% recall and 97–98% precision. On a clean shelf the gap between the best closed model and an open-weight one is a handful of books, not a tier. The hypothesis holds.
2. Opus wins, narrowly; Sonnet is the value pick among closed models. Opus: 98% recall, 98% precision, best author accuracy, zero garbles. Sonnet matches it within a book or two at under half the cost and the lowest latency.
3. The open-weight model is the cost story. Qwen3-VL-235B reads at 93% recall / 97% precision with only 3 invented books across all six shelves — for $0.011 total, a fraction of the closed models.
4. The residual is dropped books, and they cluster — not random inventions. With recall saturated, the errors that remain are misses, and the same few recur across engines (see §06). Genuine inventions are rare and idiosyncratic — no book was made up by more than one capable model, so fabrication is not a shared, systematic failure.
5. GPT-5.6 reads as well but trusts itself more and is slow. Same recall band, but more garbles + inventions than Opus (incl. genuine ones like Tehran), ~7× Sonnet's latency, and the priciest direct arm.
6. OCR is worse than dead weight — it makes a good model fabricate. Feeding OCR text to Sonnet dropped recall (97→91%) and multiplied its inventions (4→18) at ~12× the cost and 13× the latency of just looking at the photo. And it didn't rescue the weak reader — OCR→Qwen-7B collapsed to 23% recall.

07Patterns in the errors

With so few mistakes left, the interesting question is whether they are random or structured. Two clear patterns:

08The two-shelf frame: where the noise came in

One frame, k2-k3, accounts for most of the residual error — and it turns out to be a natural controlled experiment. It is a single photograph spanning two shelves, so its 53 books are the same physical books also shot as tight, single-shelf frames (k2 and k3). Same books, same models — only the framing differs.

On the tight single-shelf shots the capable models were near-perfect (Opus 29/29 on k2, 24/24 on k3). On the wide combined frame Opus dropped to 50/53 — and the books it lost, Reykjavik, Hungry, and Steve Kerr, are exactly the ones it had read correctly moments earlier on the tighter shots. It isn't the books; it's the frame.

The practical rule: one shelf per photo. Cramming two shelves into a frame to save shots is a false economy — it silently drops the hardest-to-read spines, and nothing in the output tells you which. Shoot tight; the models are near-perfect when you do.

09Caveats

• Six shelves, one library. Directions, not decimals. A larger, more varied set would tighten every number here.
• One prompt, one temperature. A different prompt could move any single model; the prompt was held identical so the comparison is fair, not tuned.
• Fuzzy scoring has edges. After a manual audit two artifacts were fixed (author-in-title matching; crediting a model that reads a spine the human marked unreadable). The garble-vs-invention split is still a deterministic threshold, not a human read of every miss.
• Prices and models drift. The cost ordering is the durable finding; the exact dollars are a snapshot.

10What happens when you charge the model for a wrong answer

Across every test above, no model ever said it could not read a spine. The reason turned out to be simple. The prompt gave it no way to say so, and no reason to. Naming a book that was not there and naming nothing at all cost exactly the same — nothing.

So the prompt was changed. The model was given a token to abstain with, and a price for guessing wrong. Everything else was held fixed: the same six photographs, the same cached OCR fragments, the same frozen human answer keys, the same scorer, temperature 0, one stateless call per cell.

What changed in the prompt

The OCR prompt contained a rule that invited the model to fill gaps from memory. It was deleted and the price added — two changes, not one, so a large effect here cannot be credited to the price alone.

2. You may use world knowledge to complete a garbled or partial title/author
   (e.g. 'APOLLO'SARROW' -> 'Apollo's Arrow'), but set confidence accordingly.

If you cannot identify a book from its fragments, emit it with title "[unreadable]".

Scoring: a correct title scores +1. A title that is not actually on the shelf
scores -2. "[unreadable]" scores 0. Maximize your score.

The photo prompt never contained a licence to guess, so there the change is clean: the abstention token and the price added, nothing removed.

What it cost

The totals mislead on their own. Sonnet's entire recall loss sits on one photograph — 54t-k2-k3, the 53-book two-shelf frame, the hardest input in the set and the one carrying the universally-missed cluster in §07. The tinted row below is that frame.

Reading the cells. +1 books gained · −1 −4 −12 books lost, darker as the loss grows  ·  inventions before, after  ·  ●● one dot per [unreadable].

On the five ordinary shelves the penalty removed eleven of fourteen inventions and cost one book. On the hard frame it removed four inventions and cost twelve.

The cleanest result in the run. Inventions fell from eight to two and recall rose by four books — on the arm where the prompt change was a single addition.

Opus barely moved, and that cell is uninformative rather than negative — it was already at 98.4% recall with three inventions in 182 books, so there was almost nothing left to decline about. Qwen reads at 23% recall and its counts are too small to carry an effect; its sixth shelf never returned parseable output at any token cap.

Why a guess costs two things, not one

The obvious expectation is a trade — fewer inventions bought with fewer real books. That is not what happened. GPT-5.6 gained four books while cutting inventions by six. The reason is visible in one case. On shelf 54t-k1-2 the baseline Sonnet run invented "Little Black Stretchy Pants: The Story of Lululemon", and on the same shelf missed the real book "lululemon and the future of technical apparel."

Same fragments. The model read lululemon, and the old rule told it to complete partial text from world knowledge, so it did — into a different, better-known book about the same company. That one guess cost two things. It scored as an invention, and it destroyed the hit, because the real book no longer had anything pointing at it. A wrong guess consumes the evidence for the right answer.

A correction to the study above

The baseline runs were already abstaining. They had no word for it. Three of the four "inventions" Sonnet was charged with on 54t-k1-2 were these:

"Unidentified fragment (possibly author surname)"
"Unidentified fragment (possibly author name 'H. Allen')"
"Unidentified degraded fragment"

That is the model declining. With no token to decline with, it wrote prose into the title field, and the scorer counted each one as a fabricated book. The ?N column in §03 was recovered by hand-auditing that bucket afterwards.

So the penalty prompt did less to Sonnet than the totals suggest. It did not teach it to abstain. It gave an existing behaviour a machine-readable name, and stopped one genuine guess. The strong version of the finding rests on GPT-5.6 and Qwen-2.5-7B, which produced no abstention in any form, prose or otherwise, and then produced seven and five.

What this does not show

• One call per cell. A difference of one or two titles is inside run-to-run variation and is not an effect. Only the large movements — 18 inventions to 3, zero abstentions to seventeen — carry weight.
• The OCR arms changed two things at once. The licence to guess was deleted and the price added together; their effects cannot be separated without a third arm, which was not run.
• The photo prompt kept its old rule — "if a spine is too obscured to read, still emit it with low confidence" — which pulls on the same cases as the new abstention instruction.
• One dead cell. Qwen's largest shelf never produced parseable output under the penalty prompt, at either token cap.
• It does not explain the twelve lost books. Whether Sonnet declined on twelve spines it genuinely could not read, or simply stopped naming books once [unreadable] became a cheap thing to repeat, is not settled here.