PoC: client-side USFM document model + live checks via @sillsdev/lynx - #327
Open
henrique221 wants to merge 12 commits into
Open
PoC: client-side USFM document model + live checks via @sillsdev/lynx#327henrique221 wants to merge 12 commits into
henrique221 wants to merge 12 commits into
Conversation
…v/lynx - /lynx-usfm authenticated route: parse USFM in the browser into a typed ScriptureDocument (no server round-trip), live diagnostics from the four StandardRuleSets.English checkers + a vendored verse-order provider, quick fixes, on-type smart quotes, app-side ignore/undo - browser-safe UsfmStylesheet built from vendored usfm.sty (?raw import) - usfm-assembly mirrors fluent-api generateUSFMText so a chapter fetched from the existing bible-texts endpoint round-trips through the same canonical USFM the export produces - checks panel deliberately echoes docs/proposals/repeated-word-check - 14 vitest specs cover stylesheet, assembly, workspace wiring, verse map
Three packaging hazards confirmed in the browser (all worked around locally, all upstream candidates for sillsdev/machine + sillsdev/lynx): 1. @sillsdev/machine corpora computes dirname(fileURLToPath(import.meta.url)) at module scope; its browser-field stubs leave fileURLToPath undefined and the module throws on import. Fix: alias url/path/fs/fs-promises to inert shims (exact-match regex; test mode keeps real Node modules for vitest). 2. Vite 8 (rolldown) treats string alias keys as prefixes: 'fs' captured 'fs/promises'. Regex aliases avoid it. 3. lynx-punctuation-checker loads locale JSON via template-literal dynamic import, unresolvable after dep optimization -> raw i18next keys in the UI. Fix: pre-register the five namespaces with vendored en.json resources (Localizer.addNamespace is first-write-wins). Verified live: sample + real Genesis 1 (Gujarati IRV) via the bible-texts endpoint, quick fix, ignore/undo, on-type smart quote, structure/preview.
Contributor
|
Warning Review limit reached
Next review available in: 57 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (55)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…at viewport height)
…n/fluent-web into poc/lynx-client-usfm
…— issue #375 (#376) * docs(rte-poc): design for the SharedEditor PoC (issue #375) + install platform-editor * feat(rte-poc): TDD lib layer — usfm-to-usj, usj-verses, pericope-slice, lynx-annotations * feat(rte-poc): /rte-poc page — Editorial editor, pericope slicing, derived-verses save path, Lynx highlights * fix(rte-poc): browser-verified integration — annotation timing, mark CSS, editor corruption mitigations Findings from live verification (platform-editor 0.8.14), mitigated here: - setAnnotation races Lexical's async content commit: re-apply on editor commits + bounded DOM verification (marks-exist check, not count — counts legitimately shrink when adjacent marks merge). - TypedMark renders class editor-typed-mark-external-<type>, not <type>. - Typing while marks are present corrupts text (spaces at mark boundaries): marks are stripped on the first editing keystroke, re-applied when idle. - A mark ending exactly at a text piece's end appends a space to the document text: mapper keeps marks one char short of piece ends. - Verse-level (marker-anchored) diagnostics map to whole-verse spans; charset floods (>10/line = wrong-language charset config) are dropped. Known upstream residue (documented for the #375 comparison): editing commits materialize trailing spaces on touched/previously-marked verses (harmless to the save path - usjToVerses trims) and, in complex scripts, boundary spaces inside untouched verses (real corruption; English is unaffected). * feat(rte-poc): in-app ProseMirror counterpart behind the same RteEditor interface Editor toggle on /rte-poc: SharedEditor (Editorial) vs PmEditor share the page, hook, lib, panels, and FormatBar — only the editor component swaps (design.md comparison harness). PmEditor: minimal scripture schema (para/book/chapter/verse atoms), TDD'd usjToDoc/docToUsj round-trip + jsonPath→position mapping (pm-doc.ts), Lynx annotations as inline decorations applied in one transaction (ephemeral: no document or undo-history impact). Browser-verified head-to-head vs Editorial 0.8.14: - Text integrity while typing with checks live: PM exact +N for N typed chars, single verse touched, English AND Gujarati — Editorial gains boundary spaces and corrupts untouched complex-script verses on the same test. - Decorations: single-transaction apply (~0.5 ms for the sample; no clear-and- reapply cycles, no undo pollution, no apply/commit race). - Undo: single-step revert works with decorations present. * feat(lynx): per-content-language rule sets — no charset checking for unknown languages (#385) The workspace hardcoded StandardRuleSets.English, so the English allowed-character set flagged nearly every character of non-Latin text (~2,600 warnings on one Gujarati chapter), drowning real signal and, in the RTE PoC, amplifying an editor bug at scale. createLynxWorkspace now takes { locale, contentLanguage }: a language with a configured set (today: 'en') gets its full rule set; unknown languages run the script-agnostic rules only (quotes, paired punctuation, punctuation context — via createSelectedDiagnosticProviders) plus verse-order. loadSource carries the content language and rebuilds the workspace transparently when it changes. Sample loads pass 'en'; API chapters are unknown until a real project-language lookup exists. Verified live: Gujarati GEN 2 drops from ~2,100 warnings to 4 real ones; the English sample keeps charset checking (© still flagged). dropCharsetFloods stays as a belt in the annotation mapper.
# Conflicts: # src/routeTree.gen.ts
henrique221
marked this pull request as ready for review
July 30, 2026 19:26
Contributor
Author
|
Brought up to date with The only conflict was Note this branch now also contains the SharedEditor RTE PoC from #376, which was merged into it rather than into main, so this PR covers two routes: Verified on Node 24.13 after the merge:
Reviewers: worth deciding up front whether this is meant to merge at all, or stay a reference branch with only the design docs landing. It is ~11k lines of PoC across two routes. |
They came in with #376 unformatted, which fails the pre-merge format:check gate now that this branch is up for review.
Contributor
|
This is great work toward getting the RTE in place. This branch should stay a reference branch with only the design docs landing. |
henrique221
added a commit
that referenced
this pull request
Aug 5, 2026
* docs: add Lynx client-USFM and RTE PoC design docs Lands the three design docs from the PoC work; the code stays on the reference branches (poc/lynx-client-usfm, PR #327, and the stacked poc/rte-shared-editor) rather than merging to main. Headers point at the reference branches so the docs read correctly on main, where the /lynx-usfm and /rte-poc routes don't exist. * docs: state the PoC limitations reviewers asked about Review follow-ups. Verified each claim against the PoC code on poc/lynx-client-usfm rather than describing intent: Lynx design doc - step 4 said getDiagnosticFixes returns TextEdit[]; it returns DiagnosticFix[] and the clicked fix's .edits are applied (the Published-API note two paragraphs later already said so) - state the dismissal key (source|code|anchor|occurrence) and that the positional fallback does not survive edits - verse resolution returns undefined outside verses; findings are listed per provider, so unscoped ones render without a chip rather than being dropped. Corrected the 'grouped by verse' wording accordingly - qualify the 28 ms number: one browser, no percentiles, main thread, and it overruns a 60 Hz frame - say the assembly test asserts structure, not byte equality with generateUSFMText - new section 8 for what the PoC leaves open (diagnostic lifecycle, unscoped diagnostics, multi-edit fixes, canonical USFM, rule sets) Assessment: mark the dismissal rows Lynx-HEAD-only and say 0.3.5 contracts dismissal app-side. RTE design doc: known-limitations section (converter drops unknown markers, pericope fallback cannot tell 'none' from 'not loaded', save contract is one-POST-per-row with no retry/duplicate guard, format bar needs a pointer/AT-reachable trigger) + fence language.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Proof of concept for "USFM, but on the client": an isolated authenticated route (
/lynx-usfm) where fluent-web parses USFM into a typedScriptureDocumentin the browser using sillsdev/lynx (@sillsdev/lynx0.3.5,-usfm0.3.1,-punctuation-checker0.2.0), runs five checkers live on every edit, and offers quick fixes, on-type smart quotes, and ignore/undo — no server round-trips.Today Fluent only touches USFM server-side (
usfm-grammarimport, string-assembled export). This PoC gives the front end a real scripture document model and demonstrates the standard checking interface discussed in the 2026-02-10 Lynx | Fluent discovery session — the general form of the Checks tab/panel proposed for the Repeated Word Check (#305):DiagnosticProvider↔ per-check hook,Diagnosticranges ↔ verse-grouped findings, fixes/dismissal ↔ Ignore Here/Everywhere.Docs on this branch:
docs/proposals/lynx-client-usfm-poc/design.md— architecture + the three browser packaging hazards (all predicted, all confirmed live, all worked around; each is an upstream contribution candidate for sillsdev/lynx + sillsdev/machine)docs/proposals/lynx-client-usfm-poc/lynx-fluent-assessment.md— what Lynx can do for Fluent, ranked integration opportunities, risks, next stepsNot production code: everything is additive and isolated under
src/features/lynx/+ one route file; no existing flows touched (onlyvite.config.tsgains browser aliases for@sillsdev/machine's Node built-ins, test mode unaffected).Screenshots
1) Overview — sample with seeded issues
The PoC page after auto-loading the sample: USFM editor with inline diagnostic highlights on the left, Checks panel with per-check accordions on the right, stats chips showing counts and the client-side parse+check time (~3 ms).
2) Inline diagnostics in the editor
Severity-coded highlights rendered from Lynx
Diagnosticranges: red wavy = error (out-of-order\v 3, unmatched quote), amber = warning (missing-verse anchors, disallowed©).3) Checks panel (echoes the #305 proposal)
One accordion per provider — Quotation marks, Allowed characters, Paired punctuation, Punctuation context, Verse order — with verse references resolved from the typed node tree, localized messages, fix buttons, Ignore, and a zero state.
4) Quick fix round-trip
After clicking Insert missing verse: the
\v 4marker is spliced into the USFM via a typedScriptureVerseedit fromUsfmEditFactory, and the missing-verse diagnostic resolves (verse-order count drops, warnings 5 → 4).5) Ignore / Show ignored / Undo
App-side dismissal mirroring the Repeated-Word suppression UX: ignored findings are hidden from counts, revealed dimmed with an "Ignored" label and Undo ignore under the "Show ignored" toggle. (Published Lynx 0.3.5 has no dismissal store yet — it's in progress upstream; this maps Fluent's cascade onto it.)
6) On-type smart quotes
A straight
"typed after "named Boaz." was autocorrected on type to the context-correct closing curly quote”by Lynx'sQuotationCorrector(caret preserved).7) The typed document model
Structure tab, both views rendered purely from the parsed node tree (no regex): a formatted scripture preview (chapter numerals, superscript verse numbers — note the out-of-order 1, 3, 2, 5 straight from the data) and the 25-node Book/Chapter/Paragraph/Verse/Text tree with source ranges.
8) Real data: Genesis 1 assembled and checked client-side
"Assemble chapter → USFM" fetches the chapter from the existing bible-texts endpoint and assembles it with a client-side mirror of the server export's
generateUSFMText— Gujarati IRV Genesis 1 parsed + checked in ~28 ms. The ~2,600 warnings are the English rule set flagging Gujarati characters: concrete evidence that rule sets must be configured per target language (they're builder-based configs).Test plan
tsc --noEmit, eslint, andvite buildclean (PoC ships as a lazy route chunk, ~75 KB gzip; main bundle untouched)Notes for reviewers
@sillsdev/*browser-packaging workarounds (lib/stylesheet.tscast,vite.config.tsaliases →lib/node-shims.ts, vendored locale pre-registration inlib/workspace.ts) are each commented with the matching upstream ask.lib/verse-order-provider.tsis vendored from Lynx's unpublished examples package (MIT, provenance header) and adapted to the published 0.3.5 API — it doubles as the template for a future Greek Room provider.