Exorcise git+https dependencies from pyproject.toml

[pllim]

jwst/pyproject.toml

Line 26 in 37a76cf

"gwcs @ git+https://github.com/spacetelescope/gwcs.git@master",

jwst/pyproject.toml

Lines 35 to 37 in 37a76cf

	"stdatamodels @ git+https://github.com/spacetelescope/stdatamodels.git@main",
	"stcal @ git+https://github.com/spacetelescope/stcal.git@main",
	"stpipe @ git+https://github.com/spacetelescope/stpipe.git@main",

jwst/pyproject.toml

Line 97 in 37a76cf

"ci-watson @ git+https://github.com/spacetelescope/ci_watson.git@main",

Having these git+https hardwired as dependencies have the following cons:

1. PyPI would reject a release unless you remember to replace them, which increases burden for release managers.
2. They rebuild from source on local jwst installs unless you pass in --no-deps, which increases burden for developers.
3. Complicate delivery of nightly dev wheels (BLD: Nightly dev wheels #9788).
4. Decrease flexibility in the stack, sometimes unnecessarily.

Alternatives:

1. Bundle the code (if you must be completely inflexible).
2. Have flexible logic routes in this package to handle different versions/behaviors of these packages.
3. Something else I didn't think of at the time of writing.

[melanieclarke]

This sounds difficult to accomplish, since changes in these repositories are interlinked and we often need to update them in tandem to avoid causing failures.

What would "bundling the code" look like?

[pllim]

Re: bundling -- I am no expert but I have come across these. Not a fan of either but they got the job done at the time they were needed.

1. https://git-scm.com/book/en/v2/Git-Tools-Submodules (some of you may remember astropy-helpers)
2. Physically copying the code over from the other repo (but only what is needed) and package it as if it belongs in this repo. I think this is only practical if the code does not change often. Living example: https://github.com/astropy/astropy/tree/main/astropy/extern

With JWST in operations, you would think the stack API shouldn't change that much anymore, but I guess not. Do they really change so badly that "flexible logic routes" is not feasible?

[melanieclarke]

Do they really change so badly that "flexible logic routes" is not feasible?

Yes, they really do! All the time! The gwcs and ci_watson packages might be feasible to support with some kind of bundling, but jwst, stdatamodels, stcal, and stpipe are deeply intertwined. They're not really independent packages. If we wanted to try to support multiple versions of our internal dependencies, we'd have to add a lot more complexity to our code. For an operational support package, I think that's both unnecessary and risky to attempt.

I don't think either git submodules or extern packaging would be satisfactory substitutes for the current method. I've found git submodules to be really difficult to work with, and the code definitely changes too often for extern packaging to be practical.

[pllim]

I heard of monorepo (https://www.atlassian.com/git/tutorials/monorepos) but I would have no idea how that would work here. 😬

I cannot speak for GWCS but I am pretty sure ci-watson doesn't need to be pinned to main all the time.

[pllim]

jwst, stdatamodels, stcal, and stpipe are deeply intertwined

That does complicate things. I wonder if @zacharyburnett or @jhunkeler have ideas.

[melanieclarke]

I cannot speak for GWCS but I am pretty sure ci-watson doesn't need to be pinned to main all the time.

Both of these are pinned to release versions when we do a jwst release, so if they are on main now, it's because at some point this build there was a breaking change we needed to support in the jwst code. If the relevant releases are available now, we could go ahead and update pyproject.toml to point to them.

[pllim]

Ah, I just opened #9800 for ci-watson. Do you want me to also attach gwcs pin fix to that PR?

[melanieclarke]

Ah, I just opened #9800 for ci-watson. Do you want me to also attach gwcs pin fix to that PR?

We can, if the necessary release is available, but I don't think it's urgent. We'll be wrapping up this build in a couple weeks anyway.

[emolter]

Chiming in to say that I regularly prompt internal stsci people to test changes I've made by installing the main branch (or a PR branch) using pip install git+https://github.com/spacetelescope/jwst.git@main, and it's extremely convenient that this also installs the main branches of stdatamodels and stcal. As Melanie said, often these changes are deeply intertwined. If these were unpinned I would very much like to have a one-liner replacement that is just as easy to use, and preferably uses a tool that most Python users are already familiar with (e.g. pip).

[zacharyburnett]

Chiming in to say that I regularly prompt internal stsci people to test changes I've made by installing the main branch (or a PR branch) using pip install git+https://github.com/spacetelescope/jwst.git@main, and it's extremely convenient that this also installs the main branches of stdatamodels and stcal. As Melanie said, often these changes are deeply intertwined. If these were unpinned I would very much like to have a one-liner replacement that is just as easy to use, and preferably uses a tool that most Python users are already familiar with (e.g. pip).

In my opinion it's better to be explicit. Would this command be acceptable?

pip install git+https://github.com/spacetelescope/jwst.git@main git+https://github.com/spacetelescope/stcal.git@main git+https://github.com/spacetelescope/stdatamodels.git@main

[emolter]

Thanks, yes that would work, although I'd caveat that with saying being explicit at the expense of simplicity is better for some reasons and worse for other reasons. It'd be easy to just simply forget to explicitly install the dependencies, especially for a non-developer, and they may have more difficulty understanding what went wrong.