Work in progress · how this page was made
This page is shared to explain the method; the library keeps growing and the runs are repeated, so the exact
numbers will keep evolving — treat them as indicative, not final. Written by Claude (Opus 4.8) from two sources:
the statistics captured during the runs, and the working session between Bharat and the model in which the
experiment was designed and its results interpreted. Not yet reviewed by a human.
How seven AI models cluster one library
What we did: we ran seven language models over the same 1,232 books, clustering each book into the
same taxonomy with the same prompt, and changed only the model.
Why we did it: to see how much of a classification comes from the method and how much
from the model, and to learn how to run the comparison fairly. It is one task on one small library, not a verdict on
any model. Manthan Cluster engine · Antilibrary · 2026-07-11.
Nothing was tuned per model. If the numbers move, the model moved them — not the setup.
The debiasing lives in the taxonomy, not the model. So every model's output can be read on the
same axes, which is what let us compare all seven directly.
The seven engines we tried
Three families · we picked them to vary one thing at a time — family, tier, scale
Anthropic · cloud, pay-per-token
Mid tier · baseline
Claude Sonnet-5
The reference every column is measured against.
Flagship
Claude Opus 4.8
Same family, top tier — isolates tier.
OpenAI · cloud, an independent frontier family
Flagship
GPT-5.5
A different lab, same task — isolates family. Does an outsider see the books the way Claude does?
Set aside after the exam
gpt-5.6-sol
Didn't clear the entrance exam — the story is below.
Open / sovereign · self-hostable, rented or local
Open · 671B · rented
DeepSeek-V3
A second open family — separates "big" from "Qwen-specific".
Open · 235B · rented
Qwen3-235B instruct
8× the local model — isolates scale.
Open · 235B · reasoning
Qwen3-235B thinking
Same size, reasoning mode — isolates thinking vs instruct.
Open · 30B · local
Qwen3-30B
On-device. The sovereign floor: $0 cash, 81 minutes.
The result
Precision (cleanliness) × recall (coverage) · bubble = cost per run
Each engine is plotted on two axes: how clean its tags are (precision, →) and how much of the
truth it covers (recall, ↑). Bubble size is the run cost. The engines did not line up on a single axis; they
scattered.
AnthropicOpenAIOpen · rentedOpen · localbubble area ∝ cost per run · P/R on the blind-judged pooled gold
Precision and recall are different things. GPT-5.5 covered the most (recall 72) but was less
precise; Sonnet was the most precise (precision 66) but covered less. So the useful question is "best for what" — a
cleaner list for a public window, or wider coverage for discovery. That is a choice we make, not the model.
All seven engines, every axis we measured. Extremes are bolded only to help the eye — read it as a landscape, not a ranking.
Engine
Leaves/bk
Precision
Recall
F1
Reject %
Pile-on K≥4
Wall
$/run
$/1k correct
GPT-5.5
5.71
53.9
71.7
61.6
0.78
4.3%
11m 0s
$17.47
$4.61
Opus 4.8
4.00
62.9
56.1
59.3
1.06
1.1%
9m 16s
$9.14
$2.95
Sonnet-5 · base
3.33
65.5
48.7
55.9
1.40
0
6m 34s
$5.06
$1.88
Qwen-235B thinking
4.20
55.7
50.9
53.2
1.69
2.0%
~65m
$4.07
$1.41
Qwen-235B instruct
3.11
53.2
36.5
43.3
11.07
0
6m 30s
$0.13
$0.06
DeepSeek-V3
2.53
55.0
30.9
39.6
3.05
0
10m 25s
$0.32
$0.19
Qwen-30B · local
5.33
35.7
37.4
36.5
21.65
9.7%
80m 38s
$0
$0
Precision / recall / F1 are on one blind-judged pooled gold standard (50 books, 619 pooled tags judged once by
Opus). "$/1k correct" = run cost ÷ estimated correct tags — the money-for-quality axis. Cost spans 135×,
from $0.13 to $17.47 for the identical job.
Three things we found
What moved when only the model changed
Q1 · Is Claude correct, or just Claude-flavored?
Correct — at the core
An independent frontier family (OpenAI) and two open families were run blind. Wherever any of them
agrees with Sonnet, the blind judge endorses the shared tags 83–95% of the time. The agreed
core is real, not an Anthropic artifact. Claude isn't uniquely right — if anything Sonnet slightly
under-tags — but its calls are sound.
Q2 · Does scale rescue the open / self-hostable path?
Yes
From the 30B local model to the 235B: the "pile-on" failure vanishes (9.7% → 0% of books),
the reject rate halves (22% → 11%), precision jumps (41% → 68%), F1 climbs 36 → 43 → 53.
DeepSeek, a different open family, independently reaches the same tier — so it's scale, not a Qwen quirk.
The sovereign path now buys ~frontier precision for ~1% of the cost.
Q3 · Quality versus money?
A 135× spread — and cost reorders quality
The precision leader is the most expensive; the open 235B delivers 68% precision for
6 cents per 1,000 correct tags versus GPT-5.5's $4.61. Anthropic's 83% prompt-cache discount keeps
Sonnet and Opus efficient despite premium rates. There is no free lunch — but there is now a cheap one.
What the entrance exam caught
A check we ran before any full run
Before running the full library, each model had to pass an entrance exam — a few bias probes
and a zero-hallucination check. Two of our four first-choice cloud engines failed it, in ways that are easy to miss
by eye.
gpt-5.6-sol — the intended OpenAI pick, same price tier as Opus — attached
Place: Japan to Taiichi Ohno's operations book, tagging it by the author's nationality. That is exactly
the identity leak the taxonomy exists to stop. Dropped to GPT-5.5, which passed 3/3 clean. Two same-price
flagships, identical prompt, split on the bias test.
Qwen3-235B (base, hybrid-thinking) — under the constant JSON prompt its reasoning
degenerated into a repeating "9000…" string, emitting zero usable tags. Replaced
with the instruct refresh, which passes clean — noted as a real confound (scale and instruct-tuning entangle).
What we are not claiming
The limits of this test
The judge is Claude-family (Opus). Absolute quality levels carry a family lean. Trust the
ordering and the recall/precision split — and the token/cost numbers, which are measured, not judged.
Caching isn't held constant across providers. Anthropic cached 83%, GPT-5.5 ~2%. The cost column is
real-world spend, not a clean model-efficiency comparison.
The thinking-235B stands on 98% of the library. Its reasoning blows the output budget on batches; ~24
books it simply refused to emit. Reasoning models are fragile at structured batch output — an operational tax
the raw quality numbers hide.
Recall is pool-relative. A correct tag that no engine found can't be credited to anyone. The
numbers rank the seven against each other, not against an absolute ceiling.
A non-LLM baseline we set aside
Embeddings + cosine · local · $0 — no language model at all
We also tried the cheapest possible path: file each book by pure similarity — embed its ideas and
each shelf's definition as vectors, and attach a shelf only when they sit close enough, with the similarity cut-off
doubling as a built-in "stay silent when unsure" dial. Free, instant, fully repeatable. On the psychology shelves it
did not hold up: its tags agreed with the three frontier models only about half as often as those models
agree with each other, and it kept pulling unrelated books — investing, distance-running — onto psychology shelves,
because similarity carries no judgment and cannot abstain. We set it aside as a production path; the models earn their keep.
▸ See the detail
The tool. qwen3-embedding-8B, run locally — deterministic, ~$0 per book, one pass.
Psychology only (26 sub-shelves), on the same fixed 400-book set as the pile-on test.
The finding. On books all four engines tagged, its shelves overlapped the three
models at ~0.36–0.39 (Jaccard), versus 0.53–0.70 among the models themselves. Only 53% of its tags were
backed by any model; 47% were its alone.
Why it fails. No abstention — it files every book to its nearest-sounding
shelf even when the honest answer is “not psychology.” Thinking, Fast and Slow → only
“Behavioral economics” (missing the cognitive / decision shelves every model caught); Born to Run
→ “evolutionary psychology,” matched on the word “evolution” in one sentence about running anatomy.
Scope. A psychology-only, agreement-based read — not the blind-gold precision / recall the seven engines
above were graded on. Its one virtue: it beat the raw local 30B on shape (no pile-on) at zero cost — useful as a
pre-filter, never as a judge.