Skip to content

fix: improvements (layout element shuffling and allow more strategies) in experimental two-stages pipeline #2723

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ElHachem02 wants to merge 6 commits into
base: main
Choose a base branch
from
+16 −5
Open

fix: improvements (layout element shuffling and allow more strategies) in experimental two-stages pipeline #2723

Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0904dbb
feat: update inference code to shuffle layout elements and discard in…
ElHachem02 Dec 3, 2025
3df3cf8
fix: add page as argument to build_prompt
ElHachem02 Dec 4, 2025
5b80d55
feat: cleanup printing and remove shuffling elements
ElHachem02 Jan 5, 2026
18d99b8
feat: add new line for layout injection
ElHachem02 Jan 5, 2026
32c1b95
feat: remove random library
ElHachem02 Jan 5, 2026
a9bab90
feat: set layout model on CPU
ElHachem02 Jan 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions docling/experimental/pipeline/threaded_layout_vlm_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,10 @@ def build_prompt(
base_prompt = self.prompt
augmented_prompt = base_prompt

# Only augment convert to docling base prompts
if base_prompt != "Convert this page to docling.":
return base_prompt

# In this layout-aware pipeline, _internal_page is always provided
if _internal_page is None:
return base_prompt
Expand All @@ -111,6 +115,14 @@ def build_prompt(
label=cluster.label
)

# Replace TABLE by otsl for consistency with doctags
if tag_name == DocumentToken.TABLE:
tag_name = "otsl"

# Remove section level details
if tag_name == "section_header_level_1":
tag_name = "section_header"

# Convert bbox to tuple and get location tokens
bbox_tuple = cluster.bbox.as_tuple()
location_tokens = DocumentToken.get_location(
Expand All @@ -126,11 +138,9 @@ def build_prompt(
if layout_elements:
# Join elements with newlines and wrap in layout tags
layout_xml = (
"<layout>" + "\n".join(layout_elements) + "</layout>"
"<layout>\n" + "\n".join(layout_elements) + "</layout>"
)
layout_injection = f"{layout_xml}"

augmented_prompt = base_prompt + layout_injection
augmented_prompt += f"\n{layout_xml}"

_log.debug(
"Enhanced Prompt with Layout Info: %s\n", augmented_prompt
Expand Down