Question: where should LLM wiki verification start?

[kiluazen]

Hi, I'm Kushal.

I'm looking at verification layers around Karpathy-style LLM wikis.

Your README says the skill ingests sources, answers with citations, and lints for index integrity, links, stale cross-references, and wiki health.

After repeated ingests, where does trust break first in practice?

1. ingest creates too few or wrong pages, but lint still passes
2. citations exist, but are too weak to audit a specific claim later
3. links still resolve, but the semantic relationship drifted
4. pages become stale after source updates or manual edits

I'm trying to separate what belongs in deterministic lint from what needs an LLM verifier. A one-line answer is useful.

[hiSandog]

My instinct is that verification should start at claim-to-source auditability, before broader semantic drift checks. If each generated page can tie important claims back to exact source spans, then later lint passes can detect stale links, missing pages, and weak citations with something concrete to inspect. A useful first metric might be “percentage of non-trivial claims with source-span citations that still resolve after re-ingest,” because that catches both ingestion loss and citation weakness without requiring full semantic evaluation upfront.

[Astro-Han]

2, and it's not close. I've been ingesting sources into a wiki daily since April, about 90 articles now, and the thing that actually breaks is claims drifting from the source during compilation. Numbers get inflated, quotes come out slightly altered, dates end up wrong. In one recent batch of 14 freshly compiled articles, lint turned up 6 quote/number drifts and 3 date errors, with the index, links, and citations all passing. Worst case so far: a page reporting 3,020 upvotes where the source said 103.

4 is the slow second. A conclusion gets superseded by a newer source but the old page still states it with certainty, and I only catch it by re-reading. 3 has never bitten me. Links either resolve or they don't.

On the split: lint gets everything you can check without re-reading the source. Index, links, raw references, orphans, formatting. Claim-vs-source fidelity and staleness go to the LLM pass. When a heuristic finding shows up twice I turn it into a deterministic check, so the checklist is mostly scar tissue at this point.

What are you building? Claim-level checks on every ingest are still too expensive for me, so if your verifier has a better answer I'm interested.