Skip to content

feat: Jan manages Context Allocation dynamically#7609

Merged
louis-jan merged 1 commit intomainfrom
fix/auto-increase-context-size
Mar 4, 2026
Merged

feat: Jan manages Context Allocation dynamically#7609
louis-jan merged 1 commit intomainfrom
fix/auto-increase-context-size

Conversation

@louis-jan
Copy link
Contributor

Describe Your Changes

When a response is cut off due to hitting the context size limit (finish_reason: "length") OR request payload exceeds the context size window (context shift disabled),
Jan now automatically increases the context size by 50% (minimum 32768), reloads the model,
and seamlessly continues generation — prepending the partial response as a prefill so the
model resumes exactly where it stopped rather than regenerating from scratch.

  • Transport exposes setContinueFromContent to inject a partial assistant prefill before the
    next request
  • Stream wrapper injects the partial text as the first delta so the UI shows it immediately
  • Partial message is retained as a placeholder while the model reloads
  • Message actions (copy, edit, delete, regenerate) are hidden during continuation
demo.mov

Self Checklist

  • Added relevant comments, esp in complex areas
  • Updated docs (for bug fixes / features)
  • Created issues for follow-up changes or refactoring needed

Copilot AI review requested due to automatic review settings March 4, 2026 08:42
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds automatic context window growth + seamless continuation when a generation is truncated (finish_reason "length") or fails due to context-size limits, so users see an uninterrupted assistant message while the model reloads and resumes.

Changes:

  • Detects truncated responses and triggers an automatic context-size increase + regeneration, carrying the partial assistant output forward as a “prefill”.
  • Extends the custom transport/hook API with setContinueFromContent and injects the partial content into the next UI stream as the first text delta.
  • Updates message rendering to support a continuation placeholder state (hide actions, adjust animation, show “Growing the Mind...” shimmer).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
web-app/src/routes/threads/$threadId.tsx Continuation orchestration: detect truncation/context errors, increase ctx_len, show placeholder + loading UI, hide actions during continuation
web-app/src/lib/custom-chat-transport.ts Implements continuation “prefill” and stream wrapper that prepends partial text as the first text-delta
web-app/src/hooks/use-chat.ts Exposes setContinueFromContent to callers via the useChat wrapper
web-app/src/containers/RenderMarkdown.tsx Adds isAnimating control for streaming markdown animation
web-app/src/containers/MessageItem.tsx Plumbs isAnimating/hideActions to message rendering and action visibility

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2026

Barecheck - Code coverage report

Total: 22.19%

Your code coverage diff: -0.07% ▾

Uncovered files and lines
FileLines
web-app/src/containers/MessageItem.tsx2, 4-7, 12, 19-27, 29-32, 34-38, 55-68, 71, 74-76, 78-83, 85-87, 90-98, 100, 103-104, 106-109, 111, 113-115, 118-126, 128-134, 136, 139-142, 144-150, 152-156, 158-163, 165-173, 175-177, 179-182, 184-185, 187-194, 196, 198, 200-201, 207-209, 211-215, 217-224, 226-229, 231, 233-241, 243-244, 246, 248-249, 251-256, 258-263, 265-268, 270-277, 279-282, 284, 286-289, 291-296, 298-309, 311, 313-318, 320-325, 327-328, 330, 332-333, 336-350, 353-355, 357-362, 365-366, 368, 372-378, 380, 382-386, 389-390, 393-398, 400-401, 403, 405-408, 410-411, 415-418, 420-426, 428, 430-431, 433-435, 437-443, 445-446, 448
web-app/src/containers/RenderMarkdown.tsx37-38, 75-77, 133-135, 137
web-app/src/hooks/use-chat.ts1, 5, 15-17, 30-43, 46-47, 49-51, 54-62, 64-68, 71, 74-78, 81-82, 84-89, 91, 93-97, 99-105, 107-111, 114-118, 121-122, 125-127, 129-131, 134-149, 151-156
web-app/src/lib/custom-chat-transport.ts2, 13-21, 57-81, 83-93, 95-98, 100, 102-104, 106-108, 116-123, 126-127, 135-139, 141, 144-148, 150-153, 155-156, 159-161, 163-165, 167-184, 186-188, 191-194, 196, 198-213, 216-218, 221, 223-237, 239-240, 245-247, 253-255, 257-258, 266, 268, 271-278, 281-282, 286-299, 302-304, 308-312, 315-318, 321, 323-330, 332, 334-335, 337-339, 341-345, 348-349, 354-356, 359-360, 363-369, 371-385, 387-389, 393-404, 406-407, 410-416, 421-424, 426, 428, 431, 434-435, 442-445, 450-452, 454-461, 463-476, 478-481
web-app/src/routes/threads/$threadId.tsx1-3, 5-9, 11-20, 25, 27-28, 32-33, 39-40, 44-55, 57-60, 63-65, 67-77, 79, 82-86, 89-90, 93, 96-107, 110, 113-117, 121-124, 126-128, 132-134, 137-140, 143-161, 166-172, 174-178, 180, 185-186, 188-189, 194-205, 208-211, 213-219, 222-223, 226-227, 230-231, 233-235, 237-238, 241-243, 245, 247-254, 256, 259-272, 274-277, 279-293, 295-305, 308-310, 312-323, 326-328, 331-334, 337-350, 352-355, 357-362, 364-371, 374, 377-382, 384-389, 391, 394, 396-403, 405-412, 414, 417-421, 423-428, 431, 434-438, 440, 442-443, 445-446, 448, 451-455, 457, 460-469, 472-475, 478-491, 494-500, 502-504, 506, 508-514, 517, 520-525, 527-535, 537-541, 544-557, 560-561, 563, 565, 567, 569, 571-572, 575-577, 582-588, 591-599, 604-605, 608, 610-612, 614-615, 618-619, 621-627, 630, 633-639, 643-644, 647-652, 654, 656, 659-668, 671-680, 683, 686-689, 692-702, 705-707, 710-716, 719-720, 722-724, 726-729, 731, 734-739, 741-753, 755-756, 758-760, 762, 764-773, 776-777, 780-792, 795-799, 801, 803-810, 812-815, 817-833, 835-849, 851-854, 856-857, 859-864, 866-870, 872-884, 886, 888-892, 894-897, 900-909, 911

@louis-jan louis-jan merged commit e2146cd into main Mar 4, 2026
22 checks passed
@louis-jan louis-jan deleted the fix/auto-increase-context-size branch March 4, 2026 09:31
@github-project-automation github-project-automation bot moved this to QA in Jan Mar 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: QA

Development

Successfully merging this pull request may close these issues.

3 participants