Skip to content

fix: Avoid frequent writes to graph.json in a loop #738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix: Avoid frequent writes to graph.json in a loop #738

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
026a366
fix: Avoid frequent writes to `graph.json` in a loop
MichaelYochpaz Sep 1, 2025
2670fb7
Add `.idea` and `.vscode` directories to `.gitignore`
MichaelYochpaz Sep 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,6 @@ __pycache__/

/.mypy_cache/
.skip-coverage

.vscode/
.idea/
1 change: 0 additions & 1 deletion src/fromager/bootstrapper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -951,7 +951,6 @@ def _add_to_graph(
download_url=download_url,
pre_built=pbi.pre_built,
)
self.ctx.write_to_graph_to_file()

def _sort_requirements(
self,
Expand Down
14 changes: 9 additions & 5 deletions src/fromager/commands/bootstrap.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,11 +179,15 @@ def bootstrap(
)
requirement_ctxvar.reset(token)

for req in to_build:
token = requirement_ctxvar.set(req)
bt.bootstrap(req, requirements_file.RequirementType.TOP_LEVEL)
progressbar.update()
requirement_ctxvar.reset(token)
try:
for req in to_build:
token = requirement_ctxvar.set(req)
bt.bootstrap(req, requirements_file.RequirementType.TOP_LEVEL)
progressbar.update()
requirement_ctxvar.reset(token)

finally:
wkctx.write_to_graph_to_file()
Comment on lines +182 to +190
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about you write the graph file every time Fromager processes a top level requirement? A typical product has between one and a handful of top level requirement.

        for req in to_build:
            token = requirement_ctxvar.set(req)
            try:
                bt.bootstrap(req, requirements_file.RequirementType.TOP_LEVEL)
            finally:
                wkctx.write_to_graph_to_file()
            progressbar.update()
            requirement_ctxvar.reset(token)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the code you made will still write the file for all dependencies, as to my understanding, we call bootstrap on every dependency recursively, so this loop runs for every dependency down the tree and not just the top level requirements. (this loop you quoted is part of the bootstrap function, and we call bootstrap on every requirement).

About the idea itself - it's possible to only write after each top level dependency is completed, and will likely still be a big performance boost (slightly lesser since we still do more writes, but it's marginal), but what benefit does this approach have over the approach of writing once at the end, inside a finally block (which means we'll write the file even if there's an error)? I don't see any benefits to it.


constraints_filename = wkctx.work_dir / "constraints.txt"
if skip_constraints:
Expand Down
Loading