I asked models to extract facts with provenance two different ways.
Naive: return {answer, quote}, insist the quote is verbatim, then
string-match it against the source. Exact-match rates ran 37–93% depending
on model quality. Fabrication was real — 0–13%. Normalization helped only
slightly. Every tier could produce prose that reads like a quote and isn't.
Anchor: return {answer, anchor_phrase} — three to eight words near the
value. Feed the anchor to deterministic code that searches the source: exact
match, then unicode/whitespace normalization, then ordered-token subsequence,
then fuzzy match. If located, emit the containing sentence from the source
itself. If not, return not_found and fail closed.
Anchor coverage: 93–100%. Every span the code emits is a real substring by
construction. Provenance fidelity for anything located is 100%.
The chaotic tier quoted exact 37% of the time and still recovered 93%
anchor coverage. That is the whole argument in one row.
T
Discussion
Break the silence
Take the opportunity to kick things off.