feat: Add --test-mode for resilient bootstrap with failure handling #719
Conversation
LalatenduMohanty
force-pushed
the
bootstrap-testmode
branch
from
09630a6 to
919ec13
Compare
|
Have not added the tests for this. Planning to add the tests after initial round of reviews |
There was a problem hiding this comment.
This implementation is going to be very slow, since it will re-process all of the packages that build successfully until it gets to the one that failed and is now marked as prebuilt.
It would be more efficient to do the check in the bootstrapper class at the point where a wheel is being built. If that build fails then we can treat the package as though it was prebuilt by running the logic to handle a prebuilt wheel, even though that wheel is not marked as prebuilt.
You could extract the logic from https://github.com/python-wheel-build/fromager/blob/main/src/fromager/bootstrapper.py#L188-L295 into its own function to make some of that logic easier to deal with. A closure inside the existing function might be easier, since that code uses a lot of the variables from elsewhere in the existing function.
LalatenduMohanty
force-pushed
the
bootstrap-testmode
branch
4 times, most recently
from
714b3f4 to
4a343fa
Compare
There was a problem hiding this comment.
I'm sorry for the blunt feedback, but I really don't like this feature. It's making one of the most complex parts of Fromager even more complicated. I spent half an hour with the new code and I still don't understand all its nooks and crannies. That worries me. I found at least one fundamental design flaw with bootstrap-parallel and failing wheel builds.
What use case do you want to solve? Make onboarding of new components easier for a developer? Can we implement simpler features to improve the UX of onboarding components?
- add a global command line flag
--prebuilt-packagesto mark packages are pre-built without creating a config file. - add a
--keep-goingflag tobuild,build-parallel, andbootstrap-parallelcommands. When the flag is set, then an exception fromwheels.build_wheelno longer stops the build and Fromager continues with building packages. At the end, print which packages have failed to build. - a new command that takes a constraints and requirements file, resolves all install dependencies, and tells the user which packages do neither have an sdist on PyPI nor settings/hooks to fetch sdist from somewhere else. Thanks to PEP 714 core metadata, this check should be fast.
src/fromager/bootstrapper.py
Outdated
| def _mark_package_as_pre_built_runtime(self, req: Requirement) -> None: | ||
| """Mark a package as pre-built at runtime for test mode.""" | ||
| from packaging.utils import canonicalize_name | ||
|
|
||
| from fromager.packagesettings import ( | ||
| Package, | ||
| PackageSettings, | ||
| Variant, | ||
| VariantInfo, | ||
| ) | ||
|
|
||
| canonical_name = Package(canonicalize_name(req.name, validate=True)) | ||
| package_settings = self.ctx.settings.package_setting(canonical_name) | ||
| variant_info = VariantInfo(pre_built=True) | ||
| new_variants = dict(package_settings.variants) | ||
| new_variants[Variant(self.ctx.variant)] = variant_info | ||
| new_package_settings = PackageSettings( | ||
| name=package_settings.name, | ||
| has_config=package_settings.has_config, | ||
| build_dir=package_settings.build_dir, | ||
| changelog=package_settings.changelog, | ||
| config_settings=package_settings.config_settings, | ||
| env=package_settings.env, | ||
| download_source=package_settings.download_source, | ||
| resolver_dist=package_settings.resolver_dist, | ||
| build_options=package_settings.build_options, | ||
| git_options=package_settings.git_options, | ||
| project_override=package_settings.project_override, | ||
| variants=new_variants, | ||
| ) | ||
|
|
||
| self.ctx.settings._package_settings[canonical_name] = new_package_settings | ||
| self.ctx.settings._pbi_cache.clear() |
There was a problem hiding this comment.
I don't like this approach. You are abusing internal implementation details of the package settings framework. This code is fragile and will make it harder to modify package settings in the future.
I have a different and simpler suggestion:
- add
pre_built_override: set[NormalizedName]tofromager.packagesettings.Settingsclass - get the state of the flag in
fromager.packagesettings.PackageBuildInfo.__init__ - modify
fromager.packagesettings.PackageBuildInfo.pre_builtto return True if the current package is flagged aspre_built_override
There was a problem hiding this comment.
Thanks for pointing this out. The modified implementation is way simpler. Fixed it with the latest code
| cache_wheel_server_url=cache_wheel_server_url, | ||
| sdist_only=True, | ||
| skip_constraints=skip_constraints, | ||
| test_mode=test_mode, |
There was a problem hiding this comment.
bootstrap-parallel runs bootstrap in sdist-only mode. That means it's not compiling any wheels except for build system requirements. The test-mode flag does not affect the wheel building for most packages.
There was a problem hiding this comment.
Right, If you want to test actual wheel compilation failures, you'd need to run fromager bootstrap --test-mode package1 package2
However it will be still useful for bootstrap-parallel to identify some issues with the source distribution e.g. identifying of the source of a package is not available, or any-other issue with dependency resolution
|
As discussed with @tiran last week, I will create a design doc and get it reviewed first then will update this PR. cc @dhellmann |
LalatenduMohanty
force-pushed
the
bootstrap-testmode
branch
2 times, most recently
from
0cb15a6 to
54dfc06
Compare
|
@dhellmann @tiran Here is the design document for the current implementation : doc |
LalatenduMohanty
force-pushed
the
bootstrap-testmode
branch
from
54dfc06 to
877f2c5
Compare
Adds --test-mode flag that marks failed packages as pre-built and continues bootstrap until all packages are processed. Uses optimal n+1 retry logic and reports comprehensive failure summary. Enables discovery of all build failures rather than stopping on first failure, supporting mixed source/binary dependency workflows. Fixes python-wheel-build#713 Co-developed-with: Cursor IDE with Claude 4.0 Sonnet Signed-off-by: Lalatendu Mohanty <[email protected]>
3 participants
Adds --test-mode flag that marks failed packages as pre-built and continues bootstrap until all packages are processed. Uses optimal n+1 retry logic and reports comprehensive failure summary.
Enables discovery of all build failures rather than stopping on first failure, supporting mixed source/binary dependency workflows.
Fixes #713
Co-developed-with: Cursor IDE with Claude 4.0 Sonnet
Command prompts: https://gist.github.com/LalatenduMohanty/762baf9999a09ef3d2d3e63220b9c52e