README.md
Audits — Content-Honesty Track
Derived-view audits that verify descriptions, tags, and cross-references match what source posts actually say. Catches classifier drift, rank inflation, and surface-keyword matching (L29 trap).
Why this exists
- Session 7 finding (manual one-off): study guide Tier 1 had 3/4 dishonest descriptions; wisdom engine was 15% junk (category page scraps);
information-theorypillar was 6 mistagged stubs while 22 real IT posts sat mis-tagged. - Lesson L29: fingerprint-by-identifier classifiers fail on unseen data. Any classifier matching surface strings (keywords, tags, title patterns) rather than content structure will silently drift.
- Without a recurring audit, derived views rot between sessions and nobody notices.
Targets (what we audit)
| Target | Population | Sample size | Schedule |
|---|---|---|---|
tier1 |
10 Tier 1 study-guide entries | 9 (all blog posts; entry #1 is the PDF white paper, skipped) | monthly |
wisdom-rank9plus |
58 insights at impact_rank ≥9 | 20 | monthly |
blog-tags |
794 tagged blog posts | 15 random | monthly |
topic-crossrefs |
10 topic files × ~30 posts each | 5/topic × 2 topics | quarterly, rotating topics |
How to run
# Generate sample inputs (deterministic by date seed)
python3 scripts/audit-sample.py --target tier1
python3 scripts/audit-sample.py --target wisdom-rank9plus
python3 scripts/audit-sample.py --target blog-tags
python3 scripts/audit-sample.py --target topic-crossrefs --topics bitcoin-protocol,law-governance
# Then spawn 4 Explore agents (parallel), one per target.
# Each gets the sample JSON + the prompt criteria from audits/PROMPTS.md.
# Each reads source post content and writes a JSON + MD audit result.
# Finally, write a YYYY-MM-DD-summary.md roll-up.
Why the evaluation isn't automated
Per L29: if you auto-evaluate "does this description match this post?" with keyword overlap or embedding similarity, you rebuild the exact bug the audit is designed to catch. Reading the post and judging whether the description is honest requires understanding structure and argument — that's the whole point. Agents or humans do this step.
The sampling is automated (deterministic by date seed). The judgment is not.
What to do with findings
- Text-level failures (quote missing from source): cleanup needed on
wisdom/insights-cleaned.json— S7 handled this and it held. - Classifier drift (surface-match tags): fix the classifier rules (
scripts/blog-categorise.pyetc), not the output. Rerun classifier after rule fix; update BASELINE.json. - Rank inflation: tighten rank criteria; re-extract wisdom.
- Empty-stub tagging: fix the pipeline (skip if word_count < 200).
History
2026-04-24— first recurring run (was Session 7 manual one-off). 4 targets audited in parallel. Headline: frozen-track archive honest; derived-track classifiers drift. Details in2026-04-24-summary.md.