feat(eval): speaker attribution scored with four truth-free signals (#849) - #872
Draft
DmitriyG228 wants to merge 4 commits into
Draft
feat(eval): speaker attribution scored with four truth-free signals (#849)#872DmitriyG228 wants to merge 4 commits into
DmitriyG228 wants to merge 4 commits into
Conversation
added 4 commits
July 21, 2026 11:55
) The framework's attribution row was aspirational (gap G2): every signal was already emitted and none was read. `onHintOutcome` is the clearest case — #797 reports hints "detected, silently discarded", which is precisely this counter going unwatched since it was added. Four signals, none of which needs a labelled meeting: provisionalRate share of stored WORDS still under a seg_N cluster id — a turn no hint ever claimed hintMissRate the binder's own per-hint verdict, finally counted renames/churn turns whose name settled somewhere other than where it started (A→B→A is defective whichever name is right, so distinctness is the measure) cardinality published speakers vs distinct hinted names Scored against the store, not the wire: `rename` repaints rows in place, so the attribution a reader ends up with is the LAST name each row was given. First measurement, jitsi/2026-07-20-capture-gaps: 16.7% of words provisional, and 79 of 86 hints missed — a 91.9% miss rate. That second number is mostly NOT the binder. Of 83 in-window hints, 55 (66%) point at a wall instant where capture never delivered a frame, so no turn could exist to match them; 3 more fall outside the audio window entirely. Capture loss is upstream of attribution loss, which reorders #847's chain: is 21 hints that pointed at delivered audio and still missed — a real residue, and the honest size of the binder-side question. A fixture with no hint stream (a shared tab) reports attribution as n/a rather than 100% provisional: nobody to name is not the same as failing to name. Tolerances follow the lane block's measured jitter. What these exist to catch — a namer that stops naming, a hint stream that goes dark — moves in tenths. (cherry picked from commit 8842477)
Coverage matrix carries the first attribution numbers (jitsi: 16.7% provisional, 91.9% hint miss) and G2 closes for the mixed lane. What stays open is named rather than implied: the gmeet lane binds at capture from the glow and needs its own reading, and these four signals score WHETHER a name was assigned, never whether it was right — that needs the self-ID oracle. (cherry picked from commit b0bc0c7)
…aker (#849, #852) The signal compared distinct hint NAMES against distinct published speakers. A hint stream is a poll, so anyone briefly lit counts as a full participant — and on zoom/2026-07-20-live-zoom that produced "3 of 5 speakers never reach the transcript", which I reported as a defect and then had to withdraw: Joyful Jen 226.0s lit Anthony 46.0s Rick 6.0s Antee 4.0s Tony 2.0s Twelve seconds between the bottom three, of a 269s call where two people did 98% of the talking. A two-speaker transcript is the correct outcome. Consecutive polls on one name now merge into a lit window, and only names lit at least 8s count toward the expected cardinality; the rest are reported separately rather than silently. The same entry now reads "2 published vs 2 substantively hinted (+3 lit under 8s, not counted)". This is the third over-claim of the day and the only one whose cause was the instrument rather than the pairing of artifacts. The pattern behind all three: a red number was treated as a defect before asking whether the EXPECTED value was right. The two guards now in the tree — a transcript must match its session's meeting id, and cardinality must weight by lit time — both attack that. (cherry picked from commit 751d7d6)
, #851) I claimed gmeet-lane attribution was unscored. It was not — quality.test.ts has always scored "segments under the right speaker"; nobody had run it on a multi-speaker fixture and written the number down. Corrected by measuring rather than by asserting again. Same 3 speakers, same known text, same local STT: gmeet (per-channel, named at capture) attribution 1.000 (14/14) · recall .873 · precision .963 mixed (one stream + DOM hints, clean) attribution 0.964 · recall .907 · precision .917 mixed (hints wrong 34% of the time) attribution 0.478 This is the layer model's central claim, finally priced: naming at capture is free and cannot be got wrong, while the mixed lane's attribution is only ever as good as the hint stream feeding it. It also says where per-platform effort belongs — not in the binder, which holds 82-94% given a good hint, but in whether each platform's DOM produces one. Coverage matrix updated; the gmeet row's content and attribution columns are no longer blank. (cherry picked from commit c3efae0)
This was referenced Jul 21, 2026
Draft
DmitriyG228
added
the
goal: PDG1 dependable capture
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this delivers (#849)
Speaker attribution is now scored, not eyeballed — four truth-free signals computed offline from a corpus entry and carried in the mixed lane's metric block, with no ground truth and no human:
seg_Nprovisional cluster id (a turn no hint ever claimed)matched/missedverdict, finally counted (speaker hints (Teams+Zoom): name resolution misses on live DOM and the no-name guard discards every detection SILENTLY — signal detection itself works (witnessed v0.12.13) #797's "silently discarded" hints become a number)FRAMEWORK.md's G2 is struck: the attribution row is promoted from aspirational to live, and priced on both lanes on identical material.Stack
Cherry-picked in order from
quality/0.12.17-base, stacked on PR 2:feat/848-framework-corpusfix/850-capture-delivery← this PR's basefeat/849-attribution-signalsBase will be retargeted to
quality/0.12.17-base(ormain) as the parents merge.Commits (
-x):88424778,b0bc0c72,751d7d60,c3efae0e.Known conflict resolved: PR 2 had dropped the
...attributionspread fromquality-mixed.test.ts's METRICS_JSON block (out of scope there). Restored fully — the block now carries BOTH the #850 metrics (resendRatio,maxSubmitSec) and the #849 attribution spread, matchingquality/0.12.17-base's reference region.Acceptance — #849 rows mapped to evidence
seg_54defect becomes a number)jitsi/2026-07-20-capture-gaps: 16.7% of published words provisional, 79/86 hints missed — recorded in the FRAMEWORK jitsi-bot row (✅ 16.7% provisional · 91.9% hint miss)n/arather than a false 100% — "nobody to name" distinguished from "failed to name"0 and miss→0; dropping every hint drives them to 1.0 (moves both directions)hints N matched / N missed (X% miss) · N renames, N turns churned · speakers P published vs H hintedturbo build17/17;node scripts/gates.mjs allgreen (eval + eval-baseline gates included)Gates
turbo build: 17/17 successfulnode scripts/gates.mjs all: greenMilestone v0.12.17. Changelog fragments ride with the picks (
docs/changelog.d/849-attribution-signals.md,docs/changelog.d/849-lit-time-cardinality.md).