[MAINT] - Identify current release process steps and pain points

[viniciusdc]

The current release process has inefficiencies, particularly regarding testing reliability and time consumption, which slow the release cycle. While automation has simplified many steps, there is still a lack of trust in test results, leading to hesitancy in pushing releases quickly.

Understanding where these bottlenecks occur is crucial for following up on decisions to mitigate or address such gaps.

Goal

• Document the current release process steps from start to finish, and -- if needed -- update the associated docs for visibility;
• Identify pain points caused by the current release process;

Success criteria

• The documentation should accurately reflect each step of the release process.
• Ideally, if a maintainer has not previously performed a release, they should test the new release workflow.

[viniciusdc]

Here, I will summarize the current process for Nebari's release process/schedule and, by the end, list the most painful points we've found while performing these processes.

Nebari’s release schedule is expected to move quickly: as soon as one release is finalized, we've already begun planning and discussing the next. For tracking purposes, we kick off each release by creating a GitHub issue (using our Release Checklist template) to track its overall progress as well as a corresponding milestone where we tag any PRs that will be included in this release, the main goal is to have that issue as a source of metric for how long the release ultimately took to be completed and visibility across any blocking issue that might've been found during it. Also, for internal communication on each release, we assign one of the core maintainers to handle the release process, named a Release Captain.

Current overall process

Typically, the whole process for cutting a release mounts to the following three major points:

1. Open the Release Issue and Milestone:

• Use the Release Checklist template to open the release issue to keep us on track.
  • Assign the release captain, expected release date, and release version.
• Start the conversation: which issues and PRs are we focusing on?
  • This usually is pretty straightforward, including the most recent merged PRs and those already in an approved/to-be-approved state.

2. Plan & Assign

• Make or update the milestone for this release.
• Gather all the PRs/issues we want to include and update the RELEASE.md notes as well as the upgrade command for this release; Nowadays its important to have this before the RC cycle begins since, if there are any automatic upgrade step occouring on this release (migration) then we need to test it within the RCs;

3. RC Cycle

• Freeze any unrelated new PRs, or branch off if it’s a hotfix (in case we need cherry-picking commits).
• Test, test, test: full deployments in the cloud, verifying dependencies, ensuring backward compatibility.
• Add or fix anything we discover along the way.

4. Final Release

• Once the RC looks solid, we tag the release.
• Make sure Docker images, nebari-dask, PyPI, and Conda-Forge updates happen in the right order.
• High fives all around!

During the RC cycle, newly discovered bugs or requirements can prompt us to add PRs to the release milestone (e.g., Release 2025.x.y). This means the milestone remains flexible throughout the RC phase. At the same time, it is not ideal to include too many changes during this stage since this might require retesting components and integration repeatedly.

The RC Cycle

Our focus during the whole process resides on the RC cycle section since it ends up becoming the most time-consuming step due to testing -- which can take around 80% of the entire process, due to in part its retroative nature and the extent of changes include in this RC.

• We cut a pre-release from main using the GitHub web UI , internally freeze PRs not intended for the release from being merged with the DO-NOT-MERGE tag, and then test a full Nebari deployment that simulates real user interactions.
  • This includes all tests under our testing-checklist template;
• If core Nebari services are upgraded, we must verify that all dependencies still work correctly;
• if there are infrastructure changes, we need to review how Terraform will handle them for existing deployments (for instance, whether it upgrades resources in place, replaces them, or migrates data).
• Ensuring backward compatibility is essential, mainly when the release includes infrastructure updates, though purely service-focused releases tend to move faster.

When the release candidate is stable enough to finalize, the release captain will see in the tracking issue, the following steps, guiding the final cut of this release:

_If there were changes to the following packages, handle their releases before cutting a new release for Nebari_
- [ ] [Cut PyPI release for `nebari-workflow-controller`](https://github.com/nebari-dev/nebari-workflow-controller)
- [ ] [Cut PyPI release for `argo-jupyter-scheduler`](https://github.com/nebari-dev/argo-jupyter-scheduler)

_These steps must be actioned in the order they appear in this checklist._

- [ ] [Tag, build and push docker images](https://github.com/nebari-dev/nebari-docker-images/releases/new)
- [ ] [Update and cut release for `nebari-dask` meta package on Conda-Forge.](https://github.com/conda-forge/nebari-dask-feedstock)
- [ ] Update `CURRENT_RELEASE` (and any other tags) in the [`constants.py`](https://github.com/nebari-dev/nebari/blob/main/src/_nebari/constants.py#L1)
- [ ] [Cut PyPI release via GHA release workflow.](https://github.com/nebari-dev/nebari/releases/new)
  - Avoid appending `v` to tag.
    - Copy release notes from `RELEASE.md`.
- [ ] [Merge automated release PR for `nebari` on Conda-Forge.](https://github.com/conda-forge/nebari-feedstock)
- [ ] Merge release branch into `main`

It's important to notice in this template the presence of the following:

These steps must be actioned in the order they appear in this checklist.

This is of utmost importance since Nebari relies on a few indirect dependencies that must be concluded before the final release tag is done. Since our Docker images live in a separate repo, any environment dep. Such as the nebari-dask package, needs to be addressed before the final images are built and tagged.

Release cadence

From a broader perspective of our release cadence, each release takes about two weeks from the time its corresponding tracking issue is opened until completion. Although we lack absolute certainty due to limited data from previous issues, we can still illustrate a best-case scenario for the timeline between creating the release issue and making the final cut. The following graph provides a visualization of that process.

[viniciusdc]

Now that we have the context in mind for the release process, here's most of the current pain points I found while performing the most recent releases:

Pain Points

• RC cycle and testing overhead: Around 80% of the release process time is spent cutting a pre-release, freezing PRs, and conducting full deployments to ensure everything works as expected. This involves simulating user interactions and verifying backward compatibility, particularly when infrastructure changes require checking Terraform behaviors for existing clusters.

  • Since most of this user-journey testing follows a very similar if not equal process as in the release testing checklist, we should make sure all the points in that checklist are automated and provide a way for the release captain to quickly rerun those tests a live cluster -- since we also depend on cloud provider deployments;

• Fragmentation between repos: Nebari’s core code and Docker images live in different repositories, and this separation can introduce issues if Dockerfile updates or dependent packages (e.g., nebari-dask) get out of sync with the main codebase. Misaligned release timings may lead to mismatched versions.

  • We haven't had any issues with this recently, but it has occurred in the past. Hopefully,y [MAINT] - Move docker-images repo back to main nebari's repo #2871 will address this.

• Upgrade command gaps: Although we test nebari upgrade at a high level, there isn’t a built-in mechanism to confirm that it hasn’t broken anything. Each new scenario could require specialized validation, which adds extra complexity and potential for missed issues.

  • We recently included an upgrade test action to handle this command execution, which immensely helped. However, this heavily depends on a cluster's existence, making the whole debugging feedback loop take longer than required. Some guidelines on writing unit tests for the upgrade command and examples of mock responses could help with this...

• Insufficient test coverage: Some problems only surface during final RC testing because we lack comprehensive automated tests and corner-case checks. This gap isn’t necessarily the fault of authors or reviewers but rather a limitation of current testing policies.

  • One such example, though this was a big corner case, was the upgrade of the conda-store that had some fields removed, leading to issues with old databases that still had such fields.
  • PRs should now be enforced to have tests alongside themsince we plan to automate the release testing step; if all that has changed is correctly covered, this will increase our coconfidence with the test results.

• Limited metrics and tracking: We don’t have a solid way to measure how long each release step takes, deployments, and sometimes the release checklist issue is created after discussions or testing have already started. Consequently, our recorded timelines and data can be misleading.

[marcelovilla]

Thanks for this detailed overview of our current release process, @viniciusdc!

I completely agree with the pain points you mentioned, and I’d go even further to say that our insufficient test coverage is the most limiting factor. This gap has resulted in significant time spent manually deploying clusters and testing both existing and new functionality, but even worse, it has also led to the need for hotfixes due to undetected bugs making it into the final release. We're not yet at a stage where we can have full confidence in our releases, given the number of use cases and critical user journeys that need to be thoroughly tested.

Here are the key areas I believe we need to address to achieve more thorough test coverage—and, in turn, a more efficient release process¹:

• Test parity between local and cloud deployments: currently, upgrade tests are only working for local deployments. Extending them to work on cloud deployments would be greatly beneficial. Furthermore, the way Nebari is deployed is different between local and cloud integration tests (see [ENH] - Move integration tests to its own test suit #2855 and [ENH] - Use Nebari CLI instead of Python and Pytest fixtures to deploy Nebari in our cloud integration tests #2950).
• Identifying critical user journeys and add tests for them: I'll open an issue for this and we already have a milestone with items we want to prioritize.
• Allowing tests to run on existing clusters and enabling the deployment of clusters with arbitrary configurations for testing: @soapy1 has already made some progress on the former in Run integration tests against deployment #2962

Once we achieve broader test coverage, the release process will become significantly faster, allowing us to cut smaller releases at a more frequent pace. As your release cadence figure shows, releases have been slow over the past year—we've struggled to release more than once per month, and at times, only once every two or three months. That said, we could still attempt to cut smaller releases anyways. Our usual releases include a large number of changes, which not only makes testing more difficult but also reduces our confidence in the release. We need to find the right balance, ensuring each release is substantial enough to justify the time spent on manual testing, but not so large that it becomes overwhelming. Achieving this requires better planning, with well-defined and properly scoped milestones for each release.

Footnotes

1. This should probably be included in one of the many existing related to improving our tests but I'm adding it here for visibility. ↩

[marcelovilla]

As an outcome of this issue, we'll create and scope new milestones to address the different pain points.

C.C.: @viniciusdc

[dcmcand]

We also need to call out the need to update constants.py