2026-09-18-pillar-convergence-plan.md

audits/2026-09-18-pillar-convergence-plan.md · 9,606 bytes

Pillar Convergence Plan — `craig-converge-pillar-scorers-kw8`

Authored by Plan-mode subagent, 2026-09-18, from read-only inspection of scripts/, blog/metadata.json, substack/metadata.json, wisdom/insights-cleaned.json.

Findings

There are FIVE pillar-scoring implementations, not three

#	File	Pillars	Style	Multi-tag	S15 gate	Overrides	Callers/output
1	`scripts/blog-categorise.py`	10	regex, word boundaries	1–3 (40% thresh, min 3.0)	yes	`MANUAL_TAG_ADD/REMOVE`	`blog/metadata.json` (795 posts), `blog/by-topic/*.md`
2	`scripts/substack-categorise.py`	10	regex, word boundaries	1–3 (60% thresh, min 3.0)	no	none	`substack/metadata.json` (324 posts), `substack/by-topic/*.md`
3	`scripts/wisdom-extract-v2.py::determine_pillar`	8 (missing `mathematics`, `enterprise`)	plain `in lower` substring	1 (argmax)	no	none	`wisdom/insights.json` (paragraph-level pillar)
4	`scripts/wisdom-extract.py::determine_pillar` (v1)	8	substring	1	no	none	stale — v1 kept beside v2
5	`scripts/substack-wisdom-extract.py::{determine_pillar, determine_pillar_for_post}`	8	substring	1 (paragraph, with post-level fallback if <2 kw hits)	no	none	`wisdom/substack-insights.json`

Plus scripts/wisdom-cleanup.py::reassign_pillar (uses TOPIC_TO_PILLAR and reads blog.topics[0]), which produced wisdom/insights-cleaned.json — this is why that file carries enterprise (60) and no mathematics even though v2's PILLARS omits both.

Divergence details vs. blog-categorise.py (post-S15)

substack-categorise.py

Same 10 pillars, same keyword regex lists, but no structural_keywords / S15 gate — so single mentions of blockchain, history, information qualify a topic.
No category_hints / category_weight (substack has no author category), no MANUAL_TAG_ADD/REMOVE.
Different secondary threshold (60% vs 40%).
Different fallback: no-match → ["philosophy"] (57 posts today are ["philosophy"]-only, likely wrong).
Different title input: concatenates title + subtitle.
Content sample: first 1000 words of raw file; density is normalised by sample word count.
Its own regex cache (_compiled_cache by id()); blog compiles per call.

wisdom-extract-v2.py::determine_pillar

8 pillars only. mathematics and enterprise are literally unreachable.
Uses substring in lower — no word boundaries. hash matches hashtag; block matches unblock; history matches historical; honest matches dishonestly.
Single argmax (never multi-tag). Any tie chooses arbitrary key.
Fallback: no match → "bitcoin-protocol" (not identity-history).
Called per paragraph (~10 k insights). Impact: 2,835 / 5,848 = 48% of bitcoin-protocol insights on disk have no strong bitcoin term (bitcoin / blockchain / merkle / utxo / spv / nakamoto / satoshi / proof-of-work / coinbase / miner). 3,444 / 5,848 = 59% do not contain the word bitcoin at all. The blockchain-drift is live and enormous.
Downstream: insights-cleaned.json re-labels some of these via blog.topics[0], but that only covers blog posts, and the drift is baked in for anything that couldn't match a blog slug.

substack-wisdom-extract.py

8 pillars + slightly expanded keyword lists (e.g. philosophy adds beauty, aesthetics, art, culture, entropy, virtue, hubris, genius; law adds bailment, custody, lien; economics adds financial, monetary, trade, tariff, payment, settlement).
Has an interesting post-level fallback (determine_pillar_for_post on title + first 500 words) — genuinely useful signal blog doesn't have.

The canonical pillar list

scripts/audit-sample.py (lines 34–36) is the ground truth: bitcoin-protocol, identity-history, law-governance, economics, security-forensics, philosophy, mathematics, enterprise, computation, information-theory. Wisdom scripts silently dropped two.

Proposed shared module

File: scripts/pillars.py.

Public surface


PILLAR_IDS = [                       # canonical order — matches audit-sample.PILLARS
    "bitcoin-protocol", "identity-history", "law-governance", "economics",
    "security-forensics", "philosophy", "mathematics", "enterprise",
    "computation", "information-theory",
]

PILLARS = { ... }                    # merged catalogue: blog's regex + S15 gate
                                     #   + useful additions absorbed from
                                     #   substack-wisdom (philosophy: beauty/
                                     #   aesthetics/virtue/hubris; law:
                                     #   bailment/custody/lien; economics:
                                     #   trade/tariff/settlement)

MANUAL_TAG_ADD = { ... }             # from blog-categorise.py
MANUAL_TAG_REMOVE = { ... }          # from blog-categorise.py

def score_text(text, *, title="", category=None, word_count=None,
               apply_structural_gate=True, density_sample_words=None) -> dict:
    """dict[pillar_id] -> (score: float, matches: list[str]). Only positive."""

def top_pillar(text, *, title="", **kw) -> str:
    """Single-argmax convenience for wisdom paragraph case. Fallback: 'bitcoin-protocol'."""

def tag_post(text, *, title, category=None, word_count=0, subtitle=None,
             secondary_threshold=0.40, min_absolute=3.0,
             max_tags=3, min_tags=1,
             fallback_pillar="identity-history",
             category_fallback_map=None,
             short_post_word_limit=100) -> list[str]:
    """Multi-tag convenience for blog and substack."""

def apply_manual_overrides(slug, tags) -> list[str]:
    """Idempotent MANUAL_TAG_ADD/REMOVE from the merged tables."""

Key design choices:

One catalogue with word-boundary regex. substring in lower is retired.
S15 gate is default-on, applied by score_text when the pillar has structural_keywords.
Multi-tag vs single-tag stays a caller choice.
Thresholds and fallbacks are parameters — historical behaviour preserved on flip day, changes become reviewable one-liners.
MANUAL_TAG_ADD/REMOVE moves into pillars.py; wisdom can start honouring slug-level overrides.

Migration deltas per caller (5 lines each, mechanical)

See agent output for exact deltas — omitted here for brevity. Every caller reduces to from pillars import ... + a thin shim that preserves its distinctive input/output shape.

Frozen 60-post hand-labelled eval set

Path: evals/pillar-gold-60.json (new evals/ directory).

20 blog posts (2 per pillar), 20 substack posts (2 per pillar), 20 wisdom insights (2 per pillar). Seeded 20260918. Pinned to source content, not re-sampled.
JSON entries: {id, source_type, slug, title, subtitle, category, word_count, text_ref, gold_tags, gold_primary, excludes, note}.
Runner: scripts/eval-pillars.py — imports pillars, iterates the JSON, prints (id, expected, actual, PASS/FAIL, reason); exits non-zero on any failure. Wire into scripts/pre-commit-hook.sh.
Freezing: any edit requires a co-signed audit entry in audits/YYYY-MM-DD-eval-changes.md.

Risks (with numbers)

Caller	Corpus	Reclassification pressure	Mitigation
`blog-categorise.py`	795 posts	Near-zero — shared module IS blog's current logic. ≤2 shift expected.	Diff old vs new metadata; block ≥5 shifts.
`substack-categorise.py`	324 posts	High. S15 gate + regex changes will move dozens. ~30–50 substack posts change tags. 57 currently `["philosophy"]`-only reroute.	Diff old vs new; hand-review; 20 substack cases in eval set.
`wisdom-extract-v2.py`	9,808 insights (10,770 in bead)	Very high. 48% of `bitcoin-protocol` insights lack a strong bitcoin term. Adding `mathematics`/`enterprise` back re-homes many.	Shadow file `insights.v3.json`; diff by id and by source slug; hand-review top-50 diffs.
`wisdom-cleanup.py`	Same	Moderate. Optional relaxation of `TOPIC_TO_PILLAR["mathematics"] = "computation"` collapse.	Explicit policy decision.
`substack-wisdom-extract.py`	`wisdom/substack-insights.json` (17 MB)	High (same reason).	Shadow-file diff pattern.

Secondary risks: downstream consumers (topics/.md, blog/by-topic/.md, wisdom/tier-*.md, study-guide.md) all regenerate — verify no external URLs depend on churn-free labels.

Recommended order

1. Extract scripts/pillars.py + scripts/tests/test_pillars.py (golden loader, S15 regression cases, MANUAL_TAG_* round-trip). No caller changes yet.

2. Freeze eval: produce evals/pillar-gold-60.json (sampling + hand-review by Adam). Gate — nothing else lands until agreed.

3. Migrate blog (lowest risk): swap; require ≤5 shifts.

4. Migrate wisdom-extract-v2 in shadow mode: shadow file, hand-review top-50, promote.

5. Migrate substack-categorise: diff, require eval pass, publish audit note.

6. Migrate substack-wisdom-extract, wisdom-extract v1, fix-info-theory-tags, wisdom-cleanup, audit-sample: mechanical single PR.

7. Retire dead code: v1 → sys.exit("use v2"). Pre-commit greps for PILLARS = { outside pillars.py.

8. Regenerate all derived views in ONE commit.

Report path: /home/a/Projects/craig/audits/2026-09-18-pillar-convergence-plan.md (this file). Original agent transcript at /tmp/claude-1001/.../tasks/aec7ad0996f0dc915.output (session-only).