2026-09-23-topic-evidence-drift.md
Topic-Evidence Drift Audit — 2026-09-23
Bead: craig-topic-evidence-drift-audit-trc
Scope: Byproduct of S21 topic_evidence sidecar — 54 over-tagged posts, 775 under-tagged (topic, post) pairs in blog/metadata.json.
Question: How much is real classifier drift vs. intentional MANUAL_TAG_ADD / MANUAL_TAG_REMOVE overrides in scripts/blog-categorise.py, and where in the classifier is the leak?
Summary
- Over-tagged (54): 18 intentional (
MANUAL_TAG_ADDforinformation-theory, matches S7 audit list) / 36 classifier "bugs" — all one shape: thefallback_mapincategorise_post()(lines 658-673) applies a category → topic map to stub posts that score 0.0 on every topic. Every one of the 36 lands onphilosophyviaacademics → philosophy(35 cases) orart → philosophy(1 case). - Under-tagged sample (20, seed=
2026-09-23): 1 REAL / 18 NOISE / 1 BORDERLINE. The 775-pair figure is dominated by URL-noise: 557/775 (72%) areidentity-historywhere the only structural hit is\bcraig\swright\bmatching the stringcraigwrightinside the source/wayback URL in every post's meta block, plus 65 moreidentity-historycases with\bdr\.?\swright\bfrom the "by Dr Wright" byline. Only ~40 of the 775 are candidates worth reviewing.
(a) Over-tagged bugs vs. intentional
| Bucket | N | Where |
|---|---|---|
Intentional (MANUAL_TAG_ADD[information-theory]) |
18 | 18/19 of the S7-audit IT list; the missing 4 (defining-scaling, how-bitcoin-won-the-race, the-myth-of-forks, simplified-payment-verification) qualified organically without needing the override, so they don't appear as "over-tagged" (they have real evidence). |
Classifier fallback → philosophy on stubs |
36 | All 36 are stub duplicates (word_count 2-88) with category ∈ {academics, art}. The full-length version of the same paper lives under a different category. E.g. geographical-aspects-of-cybercrime-a-literature-review has a wc=48 stub in academics (over-tagged philosophy) AND a wc=11,397 real copy in bitcoin-blockchain-tech (correctly tagged security-forensics, law-governance). |
Verdict: zero "the classifier scored philosophy on real content it shouldn't have" bugs. The 36 are (i) stub-duplication in the archive + (ii) an over-broad fallback_map.
(b) Under-tagged sample (20)
Seed: 2026-09-23. topic_evidence[T].structural_hits non-empty but T ∉ post.topics.
| # | Slug | Missed | struct_hits (compact) | chc | Verdict | One-line reason |
|---|---|---|---|---|---|---|
| 1 | the-scene-and-the-unseen | identity-history | craigwright | 1 | NOISE | craigwright matches URL craigwright.net in meta block. |
| 2 | papers-...-in-law-part-xvi | identity-history | craigwright | 1 | NOISE | URL-only. |
| 3 | genesis | identity-history | craigwright | 1 | NOISE | URL-only. Post is on Genesis block + Adam Smith. |
| 4 | papers-...-in-law-part-ii | identity-history | craigwright | 2 | NOISE | URL + wayback URL. |
| 5 | stable-by-design | identity-history | craigwright | 2 | NOISE | URL-only. (Already tagged IT via manual override.) |
| 6 | lightning-is-malleable-steel-is-not | identity-history | craigwright | 2 | NOISE | URL-only. |
| 7 | the-labour-fallacy-of-mining | identity-history | craigwright | 1 | NOISE | URL-only. |
| 8 | cryptocurrency-and-the-law-of-the-horse | identity-history | craigwright | 1 | NOISE | URL-only. |
| 9 | verifiable-ownership-framework-vof-... | identity-history | craigwright | 1 | NOISE | URL-only. |
| 10 | what-is-bitcoin | identity-history | craigwright | 1 | NOISE | URL-only. |
| 11 | satoshi-never-posted-on-bitcointalk | identity-history | craigwright | 1 | REAL | Body is Wright's own account of the bitcointalk migration and his Satoshi history. Also had title_hits=['\\bsatoshi\\b'] — was scored but outranked below the 40% relative-score gate by bitcoin-protocol. See recommendation 2. |
| 12 | feign-madness-but-keep-your-balance | identity-history | craigwright | 1 | NOISE | URL-only. |
| 13 | the-fallacy-of-selfish-mining-... | philosophy | \bfallac\w+\b |
1 | NOISE | Match is the word "fallacy" in the title/URL echoed in body; post is a mathematical critique, not philosophy. |
| 14 | mythology | philosophy | \bphilosoph\w+\b |
2 | NOISE | Actual title is "Mythology Archives" — a WordPress category archive page, not a post. Hits come from category-nav sidebar links. |
| 15 | book-review-after-virtue | identity-history | craigwright | 3 | NOISE | Byline / URL noise. Post is already tagged philosophy correctly; adding IH would be wrong. |
| 16 | bitcoins-privacy-model | identity-history | craigwright | 1 | NOISE | URL-only. |
| 17 | electronic-contracting-new-wine-... | identity-history | craigwright | 1 | NOISE | URL-only. |
| 18 | bitcoin-as-a-notary | philosophy | \bfallac\w+\b |
2 | BORDERLINE | Rhetorical "fallacy" ("the fallacy of decentralisation"). Post is a philosophical rebuttal of Vitalik on notarisation, but its core is protocol/law and it's already tagged bitcoin-protocol + economics + law-governance. Adding philosophy would be defensible but not required. |
| 19 | spam-away | identity-history | craigwright | 1 | NOISE | URL-only. |
| 20 | smart-card-based-mobile-wallets | identity-history | craigwright | 1 | NOISE | URL-only. |
Extrapolation: 557/775 (72%) of all under-tagged pairs are the \bcraig\swright\b → craigwright.net false positive; another ~65 are \bdr\.?\swright\b byline noise. Real signal missed ≈ ~1/20 in the sample; scaled up ≈ 30-50 posts in the whole population, mostly identity-history outranked by the 40% threshold (11) and philosophy on posts that do discuss fallacies/epistemology substantively but score below Bitcoin's dominant weight.
(c) Recommended actions (top 3)
1. Strip the meta block before scoring (root cause of 622/775). extract_meta_and_content() already removes the comment, but content still includes the human-visible source: https://craigwright.net/blog/... line and any embedded craigwright.net image URLs. That is what \bcraig\swright\b catches — the regex has no boundary between craig and wright (\s matches zero whitespace). Fix by either (a) stripping the header block up to and including the first --- before scoring, or (b) tightening the pattern to \bcraig\s+wright\b (require actual whitespace). Option (b) is a one-line fix and removes 557 spurious under-tagged pairs; it also drops the book-review-after-virtue-style byline noise (all "by Dr Craig Wright" occurrences use a space). Verified impact.
2. Loosen the 40%-of-top-score gate for identity-history when there is a title hit (line 684 in categorise_post). satoshi-never-posted-on-bitcointalk is the sample's one clear miss — title said "Satoshi", body was autobiographical, but bitcoin-protocol's score was too high. Suggested rule: any topic with a title_hits non-empty AND score >= 5.0 (i.e. at least one title match's raw weight) always qualifies, bypassing the 40% relative gate. Estimated recovery: 5-15 real posts, per-topic dominated by identity-history and philosophy.
3. Downgrade the academics/art → philosophy fallback OR pre-filter stubs. All 36 over-tagged bugs are stub duplicates with wc < 100 falling through fallback_map. Two options: (a) drop the fallback_map for posts with word_count < 100 and leave them untagged (they're stubs anyway — the real post lives elsewhere), or (b) change academics → philosophy to academics → NONE and let the primary scorer decide (the real academic posts already qualify on their own content). Either kills all 36 spurious philosophy tags without touching any legitimate assignment.
Overall verdict: within tolerance only after action 1 lands. The 775 under-tagged number is a mirage produced by one bad regex; the real drift after de-noising is ≈40 pairs, split between 3 recoverable buckets. Actions 1+3 together are ~5 lines of code and eliminate the two systemic leaks; action 2 is a small rule change with a demonstrated single-sample win. No new MANUAL_TAG_ADD entries recommended — the classifier's decisions after these fixes will be honest.
Read-only audit. blog/metadata.json was not modified. All findings verifiable by re-running the walker in scratchpad/audit_state.pkl against the classifier at scripts/blog-categorise.py.