BLD: Nightly dev wheels

[pllim]

This PR adds nightly dev wheels to enable user to install recent enough dev version without building from source. These should be the same wheels combo we would release later.

Cannot really test until after merge. 🎲

No need to run RT.

Tasks

• If you have a specific reviewer in mind, tag them.
• add a build milestone, i.e. Build 12.0 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see changelog readme for instructions)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.54%. Comparing base (6f198d4) to head (e9b4da7).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9788      +/-   ##
==========================================
- Coverage   82.54%   82.54%   -0.01%     
==========================================
  Files         366      366              
  Lines       37248    37246       -2     
==========================================
- Hits        30748    30745       -3     
- Misses       6500     6501       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[braingram]

Does anaconda allow uploading a wheel with a "git+*" dependency?

[pllim]

I don't see why not. Installing the wheels is another story though.

Why are we pinning to git+ anyway? It is pretty annoying because everytime I reinstall locally, it rebuilds those no matter what.

[braingram]

I don't see why not.

Thanks, I asked because pypi does not allow these.

Installing the wheels is another story though.

If installing will still trigger building stcal from source what benefit is there in uploading a nightly for jwst?

Why are we pinning to git+ anyway? It is pretty annoying because everytime I reinstall locally, it rebuilds those no matter what.

Often PRs with (let's say) stcal and jwst are paired. This is typically handled by having the jwst requirement point to stcal main (since there is often no released version that is compatible).

For your local installation perhaps it's worth trying --no-deps?

[pllim]

what benefit is there in uploading a nightly for jwst?

Not having to build the C stuff from this repo, maybe. I really don't think we should pin to git+ as we also have stdevdeps requirements file. If the goal is to make things easier for DMS or INS to test dev version, having the git+ pinning isn't helping anyway.

If you really want to pair up stcal so specifically, maybe a git submodule is what you want.

[braingram]

Not having to build the C stuff from this repo, maybe. I really don't think we should pin to git+ as we also have stdevdeps requirements file. If the goal is to make things easier for DMS or INS to test dev version, having the git+ pinning isn't helping anyway.

Thanks. Is that the goal here? Is there a ticket or conversation discussing this?

[pllim]

Is that the goal here? Is there a ticket or conversation discussing this?

Tyler said there is a ticket at tag-up, so maybe he could clarify.

This PR started out as an "academic exercise."

[tapastro]

The general idea here is good - SDP used to pull yml environment files from Artifactory to do ~weekly tests of SDP+CAL integration. That said, I don't believe we do (or want to do) any uploads to conda; we've moved away from conda channels generally, and it'd probably be simplest to upload wheels/ymls to Artifactory instead. This would mimic behavior that Roman has started to use for their multicomponent nightly testing architecture, which queries Artifactory for dev environment files.

DMD/SDP is not yet looking to implement this, though. If you'd like, feel free to edit the PR to upload to Artifactory. I'll move the PR to draft until we have a use case for the wheels.

[zacharyburnett]

I do agree with @pllim that we should ideally be moving away from defining git+ dependencies, so if uploading nightly wheels to Artifactory is a first step toward that then I'm all for it

[pllim]

It is hosted at anaconda but it is not a conda package. You would install it like this:

pip install jwst --pre --upgrade --extra-index-url https://pypi.anaconda.org/stsci/simple/

[pllim]

You can see example uploads from synphot here: https://pypi.anaconda.org/stsci/simple/synphot

[nden]

We need to deliver entire environments for "nightly" testing to SDP. Is the current RegressionTests scheduled action write out the files to Artifactory or was this dropped with the change to Jenkins?

[pllim]

If you mean https://github.com/spacetelescope/RegressionTests/blob/main/.github/workflows/jwst.yml , I see uploads for nightly test results which I don't think is the same as nightly builds.

[pllim]

Anyways... if you don't like this proposal, feel free to close without merge. I opened a separate issue about git+https topic that came up (#9796).

[pllim]

upload wheels/ymls to Artifactory

That would naturally exclude non-staff members from being able to use them, so not my fav option, but I don't make the call here. Also, Artifactory stuff is beyond me, so I'll leave that to someone else if you want to pursue this route instead.