Skip to content

Update workflow to use uv and Python 3.10#227

Closed
Vishak-V wants to merge 27 commits intomainfrom
update_python_version_workflows
Closed

Update workflow to use uv and Python 3.10#227
Vishak-V wants to merge 27 commits intomainfrom
update_python_version_workflows

Conversation

@Vishak-V
Copy link

This PR updates the python version for the forcingprocessor to Python3.10 and uses uv for the workflows

Additions

Removals

Changes

Changed setup to use python 3.10 and update workflow accordingly

Testing

Screenshots

Notes

Todos

Checklist

  • PR has an informative and human-readable title
  • Changes are limited to a single goal (no scope creep)
  • Code can be automatically merged (no conflicts)
  • Code follows project standards (link if applicable)
  • Passes all existing automated tests
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • Visually tested in supported browsers and devices (see checklist below 👇)
  • Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
  • Reviewers requested with the Reviewers tool ➡️

Testing checklist

Target Environment support

  • Windows
  • Linux
  • Browser

Accessibility

  • Keyboard friendly
  • Screen reader friendly

Other

  • Is useable without CSS
  • Is useable without JS
  • Flexible from small to large screens
  • No linting errors or warnings
  • JavaScript tests are passing

@arpita0911patel arpita0911patel requested review from JordanLaserGit and Copilot and removed request for JordanLaserGit August 26, 2025 15:54
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the forcingprocessor project to use Python 3.10 and migrates the CI workflows from pip to uv for package management. The change modernizes the build toolchain by adopting a faster, more reliable package manager.

  • Updated minimum Python version requirement from 3.9 to 3.10
  • Migrated all GitHub Actions workflows from pip to uv for dependency installation
  • Added virtual environment creation and activation steps in CI workflows

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
forcingprocessor/setup.cfg Updated minimum Python version requirement to 3.10
.github/workflows/forcingprocessor_weights.yaml Updated to use Python 3.10 and uv package manager
.github/workflows/forcingprocessor_plotting.yaml Updated to use Python 3.10 and uv package manager
.github/workflows/forcingprocessor_output_opts.yaml Updated to use Python 3.10 and uv package manager
.github/workflows/forcingprocessor_gcs_sources.yaml Updated to use Python 3.10 and uv package manager
.github/workflows/forcingprocessor_aws_sources.yaml Updated to use Python 3.10 and uv package manager

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment on lines +43 to 51
source .venv/bin/activate
uv pip install -e ./forcingprocessor
uv pip install pytest

- name: Test with hf2ds
run: |
source .venv/bin/activate
cd forcingprocessor
python -m pytest -vv tests/test_hf2ds.py
Copy link

Copilot AI Aug 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The virtual environment activation command source .venv/bin/activate may not persist between shell commands in GitHub Actions. Consider using uv run instead of manual activation, or ensure each command that needs the virtual environment includes the activation in the same step.

Suggested change
source .venv/bin/activate
uv pip install -e ./forcingprocessor
uv pip install pytest
- name: Test with hf2ds
run: |
source .venv/bin/activate
cd forcingprocessor
python -m pytest -vv tests/test_hf2ds.py
uv pip install -e ./forcingprocessor
uv pip install pytest
- name: Test with hf2ds
run: |
cd forcingprocessor
uv run python -m pytest -vv tests/test_hf2ds.py

Copilot uses AI. Check for mistakes.
Comment on lines +41 to +45
pip install uv
uv venv
source .venv/bin/activate
uv pip install -e ./forcingprocessor
uv pip install pytest
Copy link

Copilot AI Aug 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The virtual environment activation command source .venv/bin/activate may not persist between shell commands in GitHub Actions. Consider using uv run instead of manual activation, or ensure each command that needs the virtual environment includes the activation in the same step.

Copilot uses AI. Check for mistakes.
Comment on lines +41 to +45
pip install uv
uv venv
source .venv/bin/activate
uv pip install -e ./forcingprocessor
uv pip install pytest
Copy link

Copilot AI Aug 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The virtual environment activation command source .venv/bin/activate may not persist between shell commands in GitHub Actions. Consider using uv run instead of manual activation, or ensure each command that needs the virtual environment includes the activation in the same step.

Copilot uses AI. Check for mistakes.
Comment on lines +41 to +45
pip install uv
uv venv
source .venv/bin/activate
uv pip install -e ./forcingprocessor
uv pip install pytest
Copy link

Copilot AI Aug 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The virtual environment activation command source .venv/bin/activate may not persist between shell commands in GitHub Actions. Consider using uv run instead of manual activation, or ensure each command that needs the virtual environment includes the activation in the same step.

Copilot uses AI. Check for mistakes.
Comment on lines +41 to +45
pip install uv
uv venv
source .venv/bin/activate
uv pip install -e ./forcingprocessor
uv pip install pytest
Copy link

Copilot AI Aug 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The virtual environment activation command source .venv/bin/activate may not persist between shell commands in GitHub Actions. Consider using uv run instead of manual activation, or ensure each command that needs the virtual environment includes the activation in the same step.

Copilot uses AI. Check for mistakes.
Copy link
Collaborator

@JordanLaserGit JordanLaserGit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Vishak-V for working on this!

I have just merged a pytest and workflow to main that will run the nrds forcing processing to test that bit directly. Ideally we merge that test into this branch before merging, so we can confirm the upgrade of python from 3.9 to 3.10 doesn't break things. I had observed forcing processing fail for python3.12 on the multiprocessed write to s3.

There looks to be a conflict with datastream-deps and the main branch, so that'll need to be resolved as well.

One suggestion (that we can deal with in a later issue if we prefer) is modifying the forcingprocessing and datastream python tests to use a github runner that has a python version that matches the python version defined in the setup.cfg as done here. This way in future python upgrades, we won't have to modify the workflow files.

@JordanLaserGit
Copy link
Collaborator

@harshavemula-ua @Vishak-V can we close this PR? I don't think we will merge this with DataStreamCLI and ForcingProcessor moving.

@harshavemula-ua
Copy link
Collaborator

@JordanLaserGit This update looks fine from my side. I’ve reviewed the changes and confirmed that none of my workflows or ongoing work are hampered by this PR. The adjustments to Python 3.10 and uv in the workflows don’t introduce conflicts with the areas I’m working on, and everything continues to build and run as expected in my scope. From my end, there are no blockers to proceeding with this change. ✅

@JordanLaserGit
Copy link
Collaborator

The repositories have already been split. Changes made here won't be carried over.

@JordanLaserGit
Copy link
Collaborator

@Vishak-V has already begun moving this into forcingprocessor, just need to do the same now for datastreamcli. CIROH-UA/forcingprocessor#15

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants