packaging: check pip/conda compatibility

[oliver-sanders]

An attempt to ensure that the conda-environment.yml file is kept in sync with the pip config file(s).

For context, see conda-forge/cylc-flow-feedstock#116 (comment)

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• Changelog entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[oliver-sanders]

That did the trick, the conda builds failed with this error:

ERROR conda.cli.main_run:execute(127): `conda run pip check` failed. (See above for error)
cylc-flow 8.6.0 has requirement protobuf<8,>=5, but you have protobuf 4.24.4.

Reflecting the incompatibility between the Conda and pip metadata.

This will help in detecting cases where pip deps have been updated, but conda deps haven't, but won't catch all cases.

[oliver-sanders]

Note, the test will fail until the dependency is fixed!

Could also fix the dep on this PR, but would like reviewers to see the failure first.

[MetRonnie]

Nice, makes sense to me to fix the dependency in this PR

[MetRonnie]

One thing I noticed when testing locally is that conda seems pretty keen on installing Python 3.12 even when the current is 3.14. Can we check that 3.14 is not being overridden?

Suggested change

	# make sure Cylc is functioning
	conda run -n test python --version
	# make sure Cylc is functioning

[oliver-sanders]

That would just be testing the conda-incubator/setup-miniconda@v3 action?

[MetRonnie]

I'm concerned the python-version we specify is not working (e.g. for 3 it looks like we might be getting 3.12: https://github.com/cylc/cylc-flow/actions/runs/18500486273/job/52715670153?pr=7013#step:4:1438

Possibly this issue: conda-incubator/setup-miniconda#114

[oliver-sanders]

Right.

We could hard pin the python version at the Conda level (i.e add python = <version> into the conda environment file) which should be as safe as the Conda environment itself (if that dependency brokering is broken, all bets are off).

[oliver-sanders]

Implemented the above suggestion in c91c402

Unfold the last section in the "build Conda env" step to reveal the version installed.

[oliver-sanders]

I we want to force our actions (including this one) to run with the latest Python, we'll have to explicitly state it.

But that's a right maintenance pain as we'll have to keep updating it :(

• Using an action to extract the version is possible, but ugly, but at least automated.
• Configuring it in a file is still a maintenance pain, but at least a centralised one.
• Setting it as a repository secret/variable is a maintenance pain and non-obvious, but at least centralised.

Orthogonal to this new check either way, but probs something we should look into soon to help spot leading edge issues sooner as current tests ain't gonna do that.

[MetRonnie]

Ah right, I was thinking of actions/setup-python which does this effortlessly. I guess updating the latest one once a year isn't too bad... But at present there is no point having a separate 3.12 and 3 in the matrix

[oliver-sanders]

With setup-python, 3.x always picks the latest Python version for the operating system, not necessarily the latest Python.

With Conda, 3 always picks the latest Python compatible with the preferable solve of the environment, not necessarily the latest Python.

Neither is perfect, but setup-python is more likely to give the preferred answer more of the time.

Specifying 3 isn't "pointless", once the dependency issue that this PR is identifying has been resolved, it will pick up 3.14 as intended.

[MetRonnie]

Ohhh it was choosing 3.12 because of the old protobuf pin, got it.

Still, it is useful to have a conda run -n test python --version or better yet conda list -n test step for debugging purposes

[oliver-sanders]

Yep. It isn't ideal that this can happen, am mulling possible workarounds for future...

[oliver-sanders]

You've seen the test fail as intended, so I've pushed the dependency fix in bfda2ee

(cancelling the other checks)

[oliver-sanders]

Et voila, tests pass and the Python is 3.14