RFE: point `local` repo for rawhide chroots at `fNN-build` instead of `rawhide`

[decathorpe]

The rawhide buildroot in rawhide is no longer a symlink to fNN-build, and is out of sync with the actual koji buildroot by up to 1-2 hours. So the local repo in mock configs for rawhide should probably point at fNN-build repos instead of the rawhide one now.

[praiskup]

I'm curious why this change happened @ Fedora, do you have more info?

Pointing at correct fNN is not an easy task (maybe impossible). In the best case, there would be large time windows pointing to off-by-one releases by mistake with --enablerepo=local.

[decathorpe]

It was explained to me this way by nirik (and I apologize if I get things wrong in my retelling):

• The rawhide repo in koji used to be not a real thing but just a symlink to the fNN-build tag that corresponded to rawhide at the time. This is why the mock local repo used to work fine - because the local repo pointed at the rawhide repo in koji, which was just a symlink to the actual current rawhide buildroot.
• This was brittle because this symlink setup isn't supported by koji. For example, a user running koji request-repo rawhide would cause koji to nuke the symlink and put an actual repo there, which would then increasingly get out of sync with the actual rawhide buildroot in fNN-build.
• This setup was changed (I think when koji 1.35 was deployed) to make rawhide always be a proper repo instead of a symlink, but together with the changes to how repositories are regenerated in koji v1.35, this resulted in the content of rawhide always being a random amount of time (up to like ~2 hours) behind in time compared to the contents of the actual rawhide buildroot in fNN-build.

What makes this worse: Running koji wait-repo fNN-build --request --build <nvr> to make sure the rawhide buildroot contains a specific build only ensures that fNN-build contains this build. But mock's --enablerepo=local points at rawhide, not fNN-build - which has (potentially) different contents that cannot be wait-repo --request'd like this (because the rawhide repo is set up differently in some way that I don't really understand).

So, as I understand it, currently the only way to do local mock builds against the actual rawhide buildroot in koji is not to use --enablerepo=local, but to create your own mock config based on the rawhide one, and replace rawhide with f44-build (and keep that up-to-date when the next Fedora version is branched off).

Or, well, alternatively, always wait <~2 hours for things to propagate to the stale rawhide repo. Which makes working on stuff quite slow.

[praiskup]

The 2 hours de-sync is just nothing, compared to the mock-core-configs release time when we are shipping updated mock configs to Fedora (the process of releasing stuff with Fedora N+1 takes weeks).

A weird question, but don't you want to add another repository, also disabled, pointing at fNN-build repo e.g. f44-build, and users could just continue using local unless they know what they are doing (and opt-in for --enablerepo=f44-build)?

[decathorpe]

I don't think that makes sense - the repo currently pointed to by the local repo from mock configs is just worse in every way than using the actual koji buildroot repo.

I also don't see how this would be additional work?

The rawhide template already needs changes to tweak the release number (like here: https://github.com/rpm-software-management/mock/blob/main/mock-core-configs/etc/mock/templates/fedora-rawhide.tpl#L10 ) - can't this number be reused in the repo definition for the local repo?

[praiskup]

I also don't see how this would be additional work?

Not really a work, the problem is that we have to bump the version at some point in time during the branching process. So far, we didn't care when exactly, it just needed to be done "soon enough" before branching - so mock-core-configs have enough time to get Bodhi Karma. See e.g. 46cb155 or 74bdc61.

With the rawhide symlink, the rawhide configuration worked fine all the time, both before and after updating to the new mock-core-configs.

The new approach means that [local] isn't working, or is pointing at an invalid NN variant, for a pretty long time period. Getting karma is often a matter of two weeks (mock is now on critical path).

[decathorpe]

I don't think it being temporarily broken at branch time is a good tradeoff for it being permanently wrong ...

[praiskup]

I don't think it being temporarily broken at branch time is a good tradeoff for it being permanently wrong ...

Could we re-introduce the symlink we had before (as, e.g., rawhide-devel?), alongside the rawhide repo - just to avoid the trade-off or any regressions? Has this been discussed anywhere with the community, how do folks feel about this?

[decathorpe]

I don't think it has been more widely discussed than multiple people asking "am I mad or why is this so unreliable now?" on Matrix (releng / infra / build-sys channels). I can ask again.

[nirik]

Yeah, what @decathorpe said. ;)

We used to have the rawhide tag as just a symlink on the filesystem and changed it at branching time.
However, with the new koji on demand repos this no longer worked because people requested repos of it and kojira nuked the link. ;( So, we made it a real tag and set on it 'repo.auto : True' which is supposed to mean 'The system regularly requests repos for such tags." However, rawhide is a busy place, so the native fNN-build tag updates more quickly (especially since people --request repos from it more often).

The only reason we have a 'rawhide' tag at all is that it's a target in the case of people doing 'koji build rawhide... '
fedpkg still uses this I think, so we can't just remove it and make it a link. ;(

So, yeah, we could make another named thing be a link. I don't like rawhide-devel, sounds confusing.
How about 'rawhide-build-link' or 'rawhide-build-latest' or 'rawhide-repo-link' or something?
Of course naming is hard.

[carlwgeorge]

The only reason we have a 'rawhide' tag at all is that it's a target in the case of people doing 'koji build rawhide... '
fedpkg still uses this I think, so we can't just remove it and make it a link. ;(

I don't believe fedpkg build from the rawhide branch or koji build rawhide... have any interaction with the rawhide tag. They use the rawhide target, which currently builds in the f44-build tag. As far as I know, the rawhide tag is only needed for the generation of a repo at https://kojipkgs.fedoraproject.org/repos/rawhide/ for the local repo in mock-core-configs. For EPEL 10 we replicated that structure with epel10z and epel10 tags to get https://kojipkgs.fedoraproject.org/repos/epel10z/ and https://kojipkgs.fedoraproject.org/repos/epel10/ repos.

Given that, I think we could switch back to a regular symlink if we figure out a way to keep kojira from nuking it. A name such as rawhide-devel or rawhide-local or rawhide-latest would work I think, as I don't expect maintainers to request repos for that name. Or maybe we keep the name as rawhide and just put it somewhere on the filesystem where kojira won't touch it?

[nirik]

Yes, of course tags and targets are different, but I was pretty sure there were reasons we wanted the tag.

The only one that I can think of right now is that rawhide tag produces src repos (unlike fNN-build).
In particular that was/is used by koschei and other tools that need to get src.rpms. If we moved that to the normal build tags they would be much slower.

Yeah, we could place it somewhere else (ie, another one thats a link), but it might be confusing.