Automate wheel updates through github-action #6629
Conversation
th0ma7
force-pushed
the
automate-wheel-pr-update
branch
from
06aec54 to
73618ed
Compare
|
@hgy59 and @mreid-tt this is a little outside my normal area of work... I'm hoping with this to automate wheel updates. One area I'm unsure it will work ok is in the crossenv definition files as there is a Your 👀 on this would be appreciated, thnx. EDIT: Exclude portion now in theory solved... as for the prefix, I'll need to further think about it. |
|
@th0ma7 This is outside my area of expertise as well, but if I’m understanding correctly, you're reviewing the requirements for various Python versions and triggering a PR whenever updates are available? That seems a bit heavy; wouldn’t this potentially lead to users installing multiple new Python versions every month? Could you help me understand the benefit of this approach? |
|
The idea is for the wheels requirements definitions for python to be auto updated. This in turn would test python to confirm it still builds ok. When we are ready to release a new version it becomes straight forward as the requirements have already been tested. |
Ah! Okay. That would include the requirements we have in the |
|
It takes so much time and manipulations to fully update things that automating the wheel requirements process would simplify things greatly. Personally I'd also automate python versions... But that could be for a later time. |
This is taken from PR SynoCommunity#6629 in order to automate wheel updates using github-action.
* python314: Initial package for version 3.14.0 * python310: Update from version 3.10.18 to 3.10.19 * python311: Update from version 3.11.13 to 3.11.14 * python312: Update from version 3.12.11 to 3.12.12 * python313: Update from version 3.13.7 to 3.13.8 * python314-wheels: Preliminary package wheel check * native/python310-314: Use requirement files instead in-Makefile This is taken from PR #6629 in order to automate wheel updates using github-action. * mariadb-connector-c: Enable libmysqlclient compatibility mode * pydantic-core: Update from version 2.27.2 to 2.41.4 In turn it was migrated to rust and no longer needs its own crossenv * python314: Add post_install to fix prefix in pkgconfig files * mariadb-connector-c: Use mysql compat mode and remove old mysql version * python31*: Update default crossenv pakage versions * python314: Mark as BROKEN pending PR #6766 * python314-wheels: Mark as BROKEN for now
th0ma7
force-pushed
the
automate-wheel-pr-update
branch
from
09f6d66 to
27ad8b1
Compare
|
hey @th0ma7, did an AI review of this for you... Detailed Review of PR #6629Purpose: Automates the generation of Dependabot configuration for Python requirements files in the spksrc repository. SummaryPR #6629 adds a GitHub Actions workflow and Python script to automatically generate and maintain Files Added
Strengths
Issues Found
Suggestions
TestingI ran the script locally and it found 30 requirements files and generated valid YAML. The output structure looks correct for Dependabot. VerdictGood addition for reducing manual maintenance. The main issue is losing the |
| push: | ||
| paths: | ||
| - 'spk/python*/crossenv/requirements-default.txt' | ||
| - 'spk/python*/src/requirements-*.txt' |
There was a problem hiding this comment.
| - 'spk/python*/src/requirements-*.txt' | |
| - 'spk/python*/src/requirements-*.txt' | |
| - 'spk/python*-wheels/src/requirements-*.txt' | |
| - 'spk/*/src/requirements-crossenv.txt' | |
| - 'spk/*/src/requirements-pure.txt' |
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.11' |
There was a problem hiding this comment.
| python-version: '3.11' | |
| python-version: '3.x' |
| body: | | ||
| ## Python Requirements Configuration Update | ||
|
|
||
| This PR updates the `.github/dependabot.yml` configuration file based on the current Python requirements files in the repository. |
There was a problem hiding this comment.
| This PR updates the `.github/dependabot.yml` configuration file based on the current Python requirements files in the repository. | |
| This PR updates the \`.github/dependabot.yml\` configuration file based on the current Python requirements files in the repository. |
| - `spk/python*/crossenv/requirements-default.txt` | ||
| - `spk/python*/src/requirements-*.txt` | ||
| - `native/python*/src/requirements.txt` |
There was a problem hiding this comment.
| - `spk/python*/crossenv/requirements-default.txt` | |
| - `spk/python*/src/requirements-*.txt` | |
| - `native/python*/src/requirements.txt` | |
| - \`spk/python*/crossenv/requirements-default.txt\` | |
| - \`spk/python*/src/requirements-*.txt\` | |
| - \`spk/python*-wheels/src/requirements-*.txt\` | |
| - \`spk/*/src/requirements-crossenv.txt\` | |
| - \`spk/*/src/requirements-pure.txt\` | |
| - \`native/python*/src/requirements.txt\` |
| ### Configuration | ||
| - **Schedule**: Weekly updates | ||
| - **Grouping**: All Python dependencies grouped together per package | ||
| - **Ignored packages**: pip, Cython, msgpack |
There was a problem hiding this comment.
| - **Ignored packages**: pip, Cython, msgpack | |
| - **Ignored packages**: pip, Cython, msgpack | |
| - **Preserved**: Non-pip ecosystems (github-actions) |
| """ | ||
| Generate Dependabot configuration for Python requirements files. | ||
| Scans the repository for requirements.txt files and creates a consolidated | ||
| dependabot.yml configuration with grouped updates. |
There was a problem hiding this comment.
| dependabot.yml configuration with grouped updates. | |
| dependabot.yml configuration with grouped updates while preserving | |
| non-pip ecosystem entries (e.g., github-actions). |
|
|
||
| Check that: | ||
| - Paths start with "/" (e.g., /spk/python311/src) | ||
| - All expected requirements files are included |
There was a problem hiding this comment.
| - All expected requirements files are included | |
| - All expected requirements files are included | |
| - Non-pip ecosystems (github-actions) are preserved |
| Find the repository root by looking for common indicators | ||
| like .git directory or spksrc directory. |
There was a problem hiding this comment.
| Find the repository root by looking for common indicators | |
| like .git directory or spksrc directory. | |
| Find the repository root by looking for .git directory. |
| return current_path | ||
|
|
||
| if os.path.isdir(os.path.join(current_path, 'spksrc')): | ||
| return current_path | ||
|
|
||
| for indicator in ['.gitignore', 'README.md', 'LICENSE']: | ||
| if os.path.isfile(os.path.join(current_path, indicator)): | ||
| if os.path.isdir(os.path.join(current_path, 'spksrc')): | ||
| return current_path |
There was a problem hiding this comment.
| return current_path | |
| if os.path.isdir(os.path.join(current_path, 'spksrc')): | |
| return current_path | |
| for indicator in ['.gitignore', 'README.md', 'LICENSE']: | |
| if os.path.isfile(os.path.join(current_path, indicator)): | |
| if os.path.isdir(os.path.join(current_path, 'spksrc')): | |
| return current_path | |
| return current_path |
| "spk/python*/src/requirements-abi3.txt", | ||
| "spk/python*/src/requirements-crossenv.txt", | ||
| "spk/python*/src/requirements-pure.txt", | ||
| "native/python*/src/requirements.txt" |
There was a problem hiding this comment.
| "native/python*/src/requirements.txt" | |
| "spk/python*-wheels/src/requirements-abi3.txt", | |
| "spk/python*-wheels/src/requirements-crossenv.txt", | |
| "spk/python*-wheels/src/requirements-pure.txt", | |
| "native/python*/src/requirements.txt", | |
| "spk/*/src/requirements-crossenv.txt", | |
| "spk/*/src/requirements-pure.txt", |
| return ignore_list | ||
|
|
||
|
|
||
| def generate_dependabot_config(repo_root): |
There was a problem hiding this comment.
| def generate_dependabot_config(repo_root): | |
| def load_existing_config(config_path): | |
| """ | |
| Load existing dependabot.yml and extract non-pip ecosystem entries. | |
| Returns: | |
| list: Non-pip ecosystem update entries to preserve | |
| """ | |
| if not os.path.exists(config_path): | |
| return [] | |
| with open(config_path) as f: | |
| existing = yaml.safe_load(f) | |
| if not existing or 'updates' not in existing: | |
| return [] | |
| return [u for u in existing['updates'] if u.get('package-ecosystem') != 'pip'] | |
| def generate_dependabot_config(repo_root, preserve_ecosystems=None): |
| def generate_dependabot_config(repo_root): | ||
| """ | ||
| Generate Dependabot configuration by scanning for Python requirements files. | ||
|
|
There was a problem hiding this comment.
| Args: | |
| repo_root: Path to repository root | |
| preserve_ecosystems: List of existing non-pip ecosystem entries to preserve | |
| Generate Dependabot configuration by scanning for Python requirements files. | ||
|
|
||
| Returns: | ||
| dict: Dependabot configuration structure |
There was a problem hiding this comment.
| dict: Dependabot configuration structure | |
| tuple: (Dependabot configuration dict, list of requirements files found) |
|
|
||
| Returns: | ||
| dict: Dependabot configuration structure | ||
| """ |
There was a problem hiding this comment.
| """ | |
| """ | |
| if preserve_ecosystems is None: | |
| preserve_ecosystems = [] | |
| for pattern in REQUIREMENTS_PATTERNS | ||
| ) | ||
|
|
||
| updates = [] |
There was a problem hiding this comment.
| updates = [] | |
| # Start with preserved non-pip ecosystems | |
| updates = list(preserve_ecosystems) |
| ) | ||
|
|
||
| updates = [] | ||
| requirements_found = [] |
There was a problem hiding this comment.
| requirements_found = [] | |
| requirements_found = [] | |
| seen_dirs = set() |
|
|
||
| # CRITICAL: Dependabot requires relative path from repo root with leading "/" | ||
| relative_dir = "/" + os.path.relpath(os.path.dirname(req_file), repo_root) | ||
|
|
There was a problem hiding this comment.
| # Avoid duplicate directory+file combinations | |
| dir_file_key = (relative_dir, filename) | |
| if dir_file_key in seen_dirs: | |
| continue | |
| seen_dirs.add(dir_file_key) | |
| requirements_found.append(req_file) | |
| repo_root = find_repo_root(script_dir) | ||
|
|
||
| print(f"Repository root: {repo_root}") | ||
|
|
There was a problem hiding this comment.
| # Load existing config to preserve non-pip ecosystems | |
| config_path = os.path.join(repo_root, ".github", "dependabot.yml") | |
| preserved = load_existing_config(config_path) | |
| print(f"Preserving {len(preserved)} non-pip ecosystem entries") | |
| print(f"Repository root: {repo_root}") | ||
|
|
||
| # Generate configuration | ||
| dependabot_config, requirements_found = generate_dependabot_config(repo_root) |
There was a problem hiding this comment.
| dependabot_config, requirements_found = generate_dependabot_config(repo_root) | |
| dependabot_config, requirements_found = generate_dependabot_config(repo_root, preserved) |
| output_path = os.path.join(repo_root, ".github", "dependabot.yml") | ||
| os.makedirs(os.path.dirname(output_path), exist_ok=True) |
There was a problem hiding this comment.
| output_path = os.path.join(repo_root, ".github", "dependabot.yml") | |
| os.makedirs(os.path.dirname(output_path), exist_ok=True) | |
| os.makedirs(os.path.dirname(config_path), exist_ok=True) |
| output_path = os.path.join(repo_root, ".github", "dependabot.yml") | ||
| os.makedirs(os.path.dirname(output_path), exist_ok=True) | ||
|
|
||
| with open(output_path, "w") as f: |
There was a problem hiding this comment.
| with open(output_path, "w") as f: | |
| with open(config_path, "w") as f: |
| yaml.dump(dependabot_config, f, sort_keys=False, default_flow_style=False, indent=2) | ||
|
|
||
| print(f"Generated dependabot.yml with {len(dependabot_config['updates'])} update configurations") | ||
| print(f"Output: {output_path}") |
There was a problem hiding this comment.
| print(f"Output: {output_path}") | |
| print(f"Output: {config_path}") |
|
I've added the suggested changes from the AI as follows: Suggested improvements for the Python requirements updater:
|
|
I highly recommend to restrict this automation to python and python-wheels packages. For all other python packages you can't just process the requirements files. You need to take the project.toml file of each wheel into concern to resolve the dependencies over all. Since It would need a redesign to use original requirements like And do we really want to update wheels in python packages without updating the package version? |
3 participants
Description
Intent is to automate wheel updates through github-action so a PR can be made for every wheel needing updating as found into our requirement files. IMPORTANT: Said PR would contain a wheel update for all package where it was found.
How it works:
.github/actions/generate_dependabot.pyscript generates a.github/dependabot.ymlincluding all detected requirement files such as:.github/workflows/generate-dependabot.ymlcalls thegenerate_dependabot.pyscript every week..github/dependabot.ymland creates associated PRExtra tid-bits:
pip,Cythonandmsgpacknative/python3*now uses arequirements.txtfile and capture pip version at processing timeRelates to : #6619
Checklist
all-supportedcompleted successfullyType of change