Skip to content

templates/rebase: untag old packages at branching instead of GA #2043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dustymabe merged 2 commits into from
Oct 31, 2025
+50 −46
Merged

templates/rebase: untag old packages at branching instead of GA #2043

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4af81e4
templates/rebase: untag old pkgs at first change checkpoint instead o…
marmijo Oct 8, 2025
5b36f1e
templates/rebase: add step to update build-args.conf
marmijo Oct 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 50 additions & 46 deletions .github/ISSUE_TEMPLATE/rebase.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,54 @@
- [ ] Test Week ([template](https://github.com/coreos/fedora-coreos-tracker/issues/new?template=test-week.md&title=tracker:+FN+Test+Week))
- [ ] Communications Tracker ([example](https://github.com/coreos/fedora-coreos-tracker/issues/1655))

## At the first change checkpoint

### Untag old packages

`koji untag` N-2 packages from the pool (at some point we'll have GC in place to do this for us, but for now we must remember to do this manually or otherwise distRepo will fail once the signed packages are GC'ed). For example the following snippet finds all RPMs signed by the Fedora 32 key and untags them. Use this process:

- [ ] Find the key short hash. Usually found [here](https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/templates/pungi.rpm.conf.j2). Then:

```
f32key=12c944d0
key=$f32key
echo > untaglist # create or empty out file
for build in $(koji list-tagged --quiet coreos-pool | cut -f1 -d' '); do
if koji buildinfo $build | grep -i $key 1>/dev/null; then
Copy link
Member Author

@marmijo marmijo Oct 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added -i to make the search case-insensitive when matching $key

echo "Adding $build to untag list"
echo "${build}" >> untaglist
fi
done
```

Now we have a list of builds to untag. But we need a few more sanity checks.

- [ ] Make sure none of the builds are used in `N` based FCOS. Check by running:

```
f32key=12c944d0
key=$f32key
podman run -it --rm quay.io/fedora/fedora-coreos:testing-devel rpm -qai | grep -i -B 9 $key
Copy link
Member Author

@marmijo marmijo Oct 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added -i to make the search case-insensitive when matching $key

podman rmi quay.io/fedora/fedora-coreos:testing-devel
```

If there are any RPMs signed by the old key they'll need to be investigated. Maybe they shouldn't be used any longer. Or maybe they're still needed. One example of this is the shim RPM where the same build could be used for many Fedora releases. In this case you'll need to untag the RPM from `coreos-pool`, run a `koji distrepo`, which will remove that RPM from the repo metadata, and then re-tag it into the pool. The RPM in the repo will now be signed with a newer signing key.



- [ ] After verifying the list looks good, untag:

```
# use xargs so we don't exhaust bash string limit
cat untaglist | xargs -L50 koji untag-build -v coreos-pool
```

- [ ] Now that untagging is done, give a heads up to rpm-ostree developers that N-2 packages have been untagged and that they may need to update their CI compose tests to freeze on a newer FCOS commit.

- [ ] Remove the N-2 signing key from the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the 33/34/35 keys. You'll most likely have to get someone from releng to run the second command (`edit-tag`).
- `koji taginfo coreos-pool`
- `koji edit-tag coreos-pool -x tag2distrepo.keys="9570ff31 45719a39 9867c58f"`

## At Branching

Branching is when a new stream is "branched" off of `rawhide`. This eventually becomes the next major Fedora (N).
Expand Down Expand Up @@ -57,6 +105,7 @@ Example PR: https://github.com/coreos/coreos-installer/pull/1113
- [ ] Bump `releasever` in `manifest.yaml`
- [ ] Add the `fedora-candidate-compose` repo in `manifest.yaml` ([example PR](https://github.com/coreos/fedora-coreos-config/pull/2706))
- [ ] Update the repos in `manifest.yaml` if needed
- [ ] Bump the Fedora version and `BUILDER_IMG` tag in `build-args.conf`
- [ ] Run `cosa fetch --dry-run --update-lockfile`
- this updates the x86_64 lockfile - the others will get updated when `bump-lockfile` runs.
- in the future we may support [this](https://github.com/coreos/coreos-assembler/issues/3088) in `cosa fetch` directly
Expand Down Expand Up @@ -94,6 +143,7 @@ If the packages in `next-devel` don't exactly match the last `next` release that
- [ ] Sync the lockfiles for all arches from `next-devel`
- [ ] Bump the base Fedora version in `ci/buildroot/Dockerfile`
- [ ] Bump the Fedora version for the test containers in `tests/kola/data/commonlib.sh`
- [ ] Bump the Fedora version and `BUILDER_IMG` tag in `build-args.conf`
- [ ] PR the result


Expand Down Expand Up @@ -127,52 +177,6 @@ We prefer to disable `next-devel` when there is no difference between `testing-d
- [ ] Ship `stable`
- [ ] Set a new update barrier for the final release of N-1 on `stable`. In the barrier entry set a link to [the docs](https://docs.fedoraproject.org/en-US/fedora-coreos/update-barrier-signing-keys/). See [discussion](https://github.com/coreos/fedora-coreos-tracker/issues/480#issuecomment-1247314065)

### Untag old packages

`koji untag` N-2 packages from the pool (at some point we'll have GC in place to do this for us, but for now we must remember to do this manually or otherwise distRepo will fail once the signed packages are GC'ed). For example the following snippet finds all RPMs signed by the Fedora 32 key and untags them. Use this process:

- [ ] Find the key short hash. Usually found [here](https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/templates/pungi.rpm.conf.j2). Then:

```
f32key=12c944d0
key=$f32key
echo > untaglist # create or empty out file
for build in $(koji list-tagged --quiet coreos-pool | cut -f1 -d' '); do
if koji buildinfo $build | grep $key 1>/dev/null; then
echo "Adding $build to untag list"
echo "${build}" >> untaglist
fi
done
```

Now we have a list of builds to untag. But we need a few more sanity checks.

- [ ] Make sure none of the builds are used in `N` based FCOS. Check by running:

```
f32key=12c944d0
key=$f32key
podman run -it --rm quay.io/fedora/fedora-coreos:testing-devel rpm -qai | grep -B 9 $key
podman rmi quay.io/fedora/fedora-coreos:testing-devel
```

If there are any RPMs signed by the old key they'll need to be investigated. Maybe they shouldn't be used any longer. Or maybe they're still needed. One example of this is the shim RPM where the same build could be used for many Fedora releases. In this case you'll need to untag the RPM from `coreos-pool`, run a `koji distrepo`, which will remove that RPM from the repo metadata, and then re-tag it into the pool. The RPM in the repo will now be signed with a newer signing key.



- [ ] After verifying the list looks good, untag:

```
# use xargs so we don't exhaust bash string limit
cat untaglist | xargs -L50 koji untag-build -v coreos-pool
```

- [ ] Now that untagging is done, give a heads up to rpm-ostree developers that N-2 packages have been untagged and that they may need to update their CI compose tests to freeze on a newer FCOS commit.

- [ ] Remove the N-2 signing key from the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the 33/34/35 keys. You'll most likely have to get someone from releng to run the second command (`edit-tag`).
- `koji taginfo coreos-pool`
- `koji edit-tag coreos-pool -x tag2distrepo.keys="9570ff31 45719a39 9867c58f"`

### Open ticket for the next Fedora rebase

- [ ] Create a new ticket from the [rebase template](https://github.com/coreos/fedora-coreos-tracker/issues/new?assignees=&labels=area%2Fplatforms%2C+kind%2Fenhancement&template=rebase.md&title=tracker:+Rebase+onto+Fedora+N)
Expand Down