Skip to content

change(web): add tokenized prediction intermediate type for whitespace correction support 🚂 🔪 - #15907

Merged
jahorton merged 10 commits into
epic/boundary-correctionfrom
change/web/multi-token-prediction-intermediates
Jun 29, 2026
Merged

change(web): add tokenized prediction intermediate type for whitespace correction support 🚂 🔪#15907
jahorton merged 10 commits into
epic/boundary-correctionfrom
change/web/multi-token-prediction-intermediates

Conversation

@jahorton

@jahorton jahorton commented May 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Converts early uses of CompositedPredictionData to TokenizedPredictionData, facilitating important token-based aspects of whitespace correction support, such as case-handling. Predictions will be computed token-by-token without immediately compositing them into a single token. The compositing step will be done later.

Note that the .metadata component of both CompositedPredictionData and TokenizedPredictionData is identical.

Build-bot: skip build:web
Test-bot: skip

@github-project-automation github-project-automation Bot moved this to Todo in Keyman May 4, 2026
@keymanapp-test-bot

keymanapp-test-bot Bot commented May 4, 2026 •
edited
Loading

Copy link
Copy Markdown

User Test Results

Test specification and instructions

User tests are not required

Test Artifacts

  • Web
    • KeymanWeb Test Home - build : all tests passed (no artifacts on BuildLevel "build")

@keymanapp-test-bot keymanapp-test-bot Bot changed the title change(web): rework prediction intermediate type for whitespace correction support change(web): rework prediction intermediate type for whitespace correction support 🚂 May 4, 2026
@keymanapp-test-bot keymanapp-test-bot Bot added this to the A19S28 milestone May 4, 2026
@jahorton
jahorton force-pushed the change/web/multi-token-prediction-intermediates branch from e8f3cad to 1a3e571 Compare May 6, 2026 20:17
@jahorton
jahorton changed the base branch from feat/web/multi-token-predict-core to change/web/rework-traversalless-prediction May 6, 2026 20:17
@jahorton
jahorton force-pushed the change/web/multi-token-prediction-intermediates branch from 1a3e571 to b243fc8 Compare May 6, 2026 20:19
@jahorton
jahorton force-pushed the change/web/rework-traversalless-prediction branch from f8a1e49 to daea6e5 Compare May 7, 2026 18:22
@jahorton
jahorton force-pushed the change/web/multi-token-prediction-intermediates branch from eb8c98b to 96b044a Compare May 7, 2026 18:45
@keyman-server keyman-server modified the milestones: A19S28, A19S29 May 11, 2026
@jahorton
jahorton force-pushed the change/web/multi-token-prediction-intermediates branch from 5150f26 to e622a4b Compare May 12, 2026 17:09
@jahorton
jahorton changed the base branch from change/web/rework-traversalless-prediction to fix/web/create-default-keep May 12, 2026 18:28
@jahorton
jahorton force-pushed the change/web/multi-token-prediction-intermediates branch from e622a4b to d79d54a Compare May 12, 2026 19:43
@jahorton
jahorton changed the base branch from fix/web/create-default-keep to change/web/multi-token-suggestion-similarity May 12, 2026 19:43
@jahorton jahorton changed the title change(web): rework prediction intermediate type for whitespace correction support 🚂 change(web): add tokenized prediction intermediate type for whitespace correction support May 12, 2026
@keymanapp-test-bot keymanapp-test-bot Bot changed the title change(web): add tokenized prediction intermediate type for whitespace correction support change(web): add tokenized prediction intermediate type for whitespace correction support 🚂 May 12, 2026
@github-actions github-actions Bot added the change Minor change in functionality, but not new label May 13, 2026
@jahorton jahorton mentioned this pull request May 15, 2026
Merged
@keyman-server keyman-server modified the milestones: A19S29, A19S30 May 23, 2026
@jahorton jahorton mentioned this pull request May 28, 2026
Merged
@jahorton
jahorton force-pushed the change/web/multi-token-suggestion-similarity branch from 61fc801 to 96339dc Compare May 28, 2026 21:43
@jahorton
jahorton force-pushed the change/web/multi-token-prediction-intermediates branch from b763609 to 71a9162 Compare May 29, 2026 21:40
@keymanapp-test-bot keymanapp-test-bot Bot changed the title change(web): add tokenized prediction intermediate type for whitespace correction support 🚂 change(web): add tokenized prediction intermediate type for whitespace correction support 🚂 🔪 May 29, 2026
jahorton
jahorton commented May 29, 2026
View reviewed changes
Comment thread web/src/engine/predictive-text/worker-thread/src/main/predict-helpers.ts
tokens.push({
correction: entry,
casingRoot: orderedTokens ? orderedTokens[i].exampleInput : entry.sample.insert,
autoSelectable: correctionValidForAutoSelect(entry.sample.insert)

@jahorton jahorton May 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This, along with the changes to the PredictionParameters interface, should probably be migrated earlier as well - likely to the implementation of buildCorrectionSequence in #16028 / #15977 (or earlier, if it can be moved earlier to prevent reworking things like #15851.

@jahorton jahorton mentioned this pull request Jun 1, 2026
Merged
@jahorton
jahorton force-pushed the change/web/multi-token-suggestion-similarity branch 2 times, most recently from 4d2321f to fa050f9 Compare June 2, 2026 14:32
@jahorton
jahorton force-pushed the change/web/multi-token-prediction-intermediates branch from 71a9162 to e1bba09 Compare June 2, 2026 21:30
@keyman-server keyman-server modified the milestones: A19S30, A19S31 Jun 8, 2026
@jahorton
jahorton force-pushed the change/web/multi-token-suggestion-similarity branch 2 times, most recently from 2a9c6f2 to 618e6bd Compare June 12, 2026 14:10
jahorton added 5 commits June 12, 2026 09:11
@jahorton
change(web): add tokenized prediction intermediate type for whitespac…
b12c912 
…e correction support

Converts early uses of CompositedPredictionData to TokenizedPredictionData to facilitate important token-based aspects of whitespace correction support, such as case-handling.

Build-bot: skip build:web
Test-bot: skip
@jahorton
fix(web): adjusts existing unit tests to match new intermediate-predi…
6b27134 
…ction-data format
@jahorton
fix(web): apply original casing-application logic on a per-token basis
fb0883e
@jahorton
change(web): clean up application of transitionIDs and applyInPost funcs
dbd060a
@jahorton
fix(web): unit test patchup, dummy model updates for changed dummied-…
acec50b 
…suggestion pattern
@jahorton
jahorton force-pushed the change/web/multi-token-prediction-intermediates branch from e1bba09 to acec50b Compare June 12, 2026 14:25
@jahorton
jahorton force-pushed the change/web/multi-token-prediction-intermediates branch from c6182e0 to 15080a6 Compare June 12, 2026 15:32
@jahorton
jahorton requested a review from ermshiperete June 12, 2026 17:36
@jahorton
jahorton marked this pull request as ready for review June 12, 2026 17:36
@keyman-server keyman-server modified the milestones: A19S31, A19S32 Jun 22, 2026
mcdurdin
mcdurdin approved these changes Jun 22, 2026
View reviewed changes
Comment thread web/src/engine/predictive-text/worker-thread/src/main/predict-helpers.ts Outdated
Comment thread web/src/engine/predictive-text/worker-thread/src/main/predict-helpers.ts Outdated
Comment thread web/src/engine/predictive-text/worker-thread/src/main/predict-helpers.ts Outdated
Comment thread web/src/engine/predictive-text/worker-thread/src/main/predict-helpers.ts Outdated
Comment thread web/src/engine/predictive-text/worker-thread/src/main/predict-helpers.ts Outdated
Comment thread web/src/engine/predictive-text/worker-thread/src/main/predict-helpers.ts Outdated
jahorton added 3 commits June 22, 2026 10:26
@jahorton
change(web): apply suggestions from PR review
ed42281 
- A few let -> const
- Adds and fixes doc-comments
@jahorton
change(web): rename compositeIntermdiatePredictions to composeInterme…
bdc824a 
…diatePredictions
@jahorton
Merge branch 'change/web/multi-token-suggestion-similarity' into chan…
473ca85 
…ge/web/multi-token-prediction-intermediates

Also renames IntermediateTokenizedPrediction to TokenizedIntermediatePrediction.
Base automatically changed from change/web/multi-token-suggestion-similarity to epic/boundary-correction June 29, 2026 18:54
@jahorton
jahorton merged commit eb93ac7 into epic/boundary-correction Jun 29, 2026
7 of 8 checks passed
@jahorton
jahorton deleted the change/web/multi-token-prediction-intermediates branch June 29, 2026 20:27
@github-project-automation github-project-automation Bot moved this from Todo to Done in Keyman Jun 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mcdurdin mcdurdin mcdurdin approved these changes
@ermshiperete ermshiperete Awaiting requested review from ermshiperete

Assignees

No one assigned

Labels

change Minor change in functionality, but not new epic-autocorrect epic-boundary-correction web/predictive-text/ web/

Projects

Keyman
Archived in project

Milestone

A19S32

Development

Successfully merging this pull request may close these issues.

3 participants

@jahorton @mcdurdin @keyman-server