Improve package updates reporting and add more build deps to SDK

[krnowak]

CI: http://jenkins.infra.kinvolk.io:8080/job/container/job/sdk/1956/cldsv/

This PR affects SDK only - it has some dependencies specified explicitly (for the package update automation) and it will now have all the build dependencies of the board packages, so the board-package-build job won't be building stuff for SDK anymore.

[krnowak]

CI passed.

[github-actions]

Build action triggered: https://github.com/flatcar/scripts/actions/runs/13675214714

[chewi]

I think there might be a better way, at least for including the build dependencies. In Portage 3.0.66, I changed --root-deps to install build dependencies to ROOT as well as / for all EAPIs. It previously installed them to ROOT instead of /, but this was never a good idea, so it was ignored for EAPI 7 and later.

[krnowak]

I think there might be a better way, at least for including the build dependencies. In Portage 3.0.66, I changed --root-deps to install build dependencies to ROOT as well as / for all EAPIs. It previously installed them to ROOT instead of /, but this was never a good idea, so it was ignored for EAPI 7 and later.

The point here is that we want those build dependencies to be a part of the SDK, but it is not about where the build dependencies are installed (ROOT or /), but rather when they are being built. I want all the build dependencies of the board packages to be built and installed during the SDK build job. But there were still some build dependencies that were absent in SDK, so they were built during board packages job.

[chewi]

Ah, I see. --onlydeps --onlydeps-with-rdeps=n almost works, but it includes DEPEND, which is unhelpful in this case. Maybe I could add a --onlydeps-with-ddeps option or something? It's annoying that a lot of the existing Portage options lump DEPEND and BDEPEND together as build dependencies.

[krnowak]

Ah, I see. --onlydeps --onlydeps-with-rdeps=n almost works, but it includes DEPEND, which is unhelpful in this case. Maybe I could add a --onlydeps-with-ddeps option or something?

I assume you meant --onlydeps-with-bdeps. Anyway. Do I understand it correctly that this flag would be something we could use in stage4 of SDK build, where we could replace emerge coreos-devel/sdk-depends with emerge --onlydeps-with-bdeps coreos-devel/board-packages; emerge coreos-devel/some-extra-pkgs-for-sdk? That'd be cool, actually.

It's annoying that a lot of the existing Portage options lump DEPEND and BDEPEND together as build dependencies.

Ah, well, that's the nature of the evolving software, isn't it? BDEPEND is relatively new.

[chewi]

I assume you meant --onlydeps-with-bdeps.

No, --onlydeps includes all the dependency types, then you filter out RDEPEND with --onlydeps-with-rdeps=n. Therefore, we would also filter out DEPEND with --onlydeps-with-ddeps=n. There's also --onlydeps-with-ideps for IDEPEND, which is filtered out by default when rdeps are, but we'd actually want to include those, so we'd also put =y for that.

Do I understand it correctly that this flag would be something we could use in stage4 of SDK build, where we could replace emerge coreos-devel/sdk-depends with emerge --onlydeps-with-bdeps coreos-devel/board-packages; emerge coreos-devel/some-extra-pkgs-for-sdk? That'd be cool, actually.

Yes. I've already coded it up and it works. Just gathering opinions on the option name.

Ah, well, that's the nature of the evolving software, isn't it? BDEPEND is relatively new.

New in 2018. 😛 I introduced it though, so I suppose it's my fault!

[krnowak]

I assume you meant --onlydeps-with-bdeps.

No, --onlydeps includes all the dependency types, then you filter out RDEPEND with --onlydeps-with-rdeps=n. Therefore, we would also filter out DEPEND with --onlydeps-with-ddeps=n. There's also --onlydeps-with-ideps for IDEPEND, which is filtered out by default when rdeps are, but we'd actually want to include those, so we'd also put =y for that.

Got it. But wouldn't it be better to have --onlydeps=<CSV-of-dep-types>? So you could have dep types like rdeps, bdeps, ddeps, ideps and pdeps and you pass some variation of them to --only-deps. Possibly with a shortcut like --only-deps=all meaning --only-deps=rdeps,bdeps,ddeps,ideps,pdeps.

Do I understand it correctly that this flag would be something we could use in stage4 of SDK build, where we could replace emerge coreos-devel/sdk-depends with emerge --onlydeps-with-bdeps coreos-devel/board-packages; emerge coreos-devel/some-extra-pkgs-for-sdk? That'd be cool, actually.

Yes. I've already coded it up and it works. Just gathering opinions on the option name.

Cool. Do you want to add this feature to flatcar as a user-patch instead of a part of this PR? Or do we merge this PR and use your feature when it's merged into portage?

Ah, well, that's the nature of the evolving software, isn't it? BDEPEND is relatively new.

New in 2018. 😛 I introduced it though, so I suppose it's my fault!

That's why I said "relatively". :P

[chewi]

Got it. But wouldn't it be better to have --onlydeps=? So you could have dep types like rdeps, bdeps, ddeps, ideps and pdeps and you pass some variation of them to --only-deps. Possibly with a shortcut like --only-deps=all meaning --only-deps=rdeps,bdeps,ddeps,ideps,pdeps.

I thought that too, but we'd still need the old options for compatibility, and it would probably just be even more confusing than it already is.

Cool. Do you want to add this feature to flatcar as a user-patch instead of a part of this PR? Or do we merge this PR and use your feature when it's merged into portage?

The former. I've had one response that said ddeps is fine. Good enough.

[chewi]

I've checked the patch applies and pushed it. I'll leave you to plumb it in. I guess that should happen in catalyst_sdk.sh?

PORTAGE_CONFIGROOT="$ROOT" emerge --root="$ROOT" --sysroot="$ROOT" --onlydeps --onlydeps-with-rdeps=n --onlydeps-with-ddeps=n --onlydeps-with-ideps=y coreos-devel/board-packages

I haven't pushed it upstream yet, so please let me know if it works.

[krnowak]

I've checked the patch applies and pushed it. I'll leave you to plumb it in. I guess that should happen in catalyst_sdk.sh?

PORTAGE_CONFIGROOT="$ROOT" emerge --root="$ROOT" --sysroot="$ROOT" --onlydeps --onlydeps-with-rdeps=n --onlydeps-with-ddeps=n --onlydeps-with-ideps=y coreos-devel/board-packages

I haven't pushed it upstream yet, so please let me know if it works.

Thanks, I'll try it out. I used a different command (run_merge, as provided by catalyst's chroot-functions.sh) though:

run_merge --onlydeps --onlydeps-with-rdeps=n --onlydeps-with-ddeps=n --onlydeps-with-ideps=y coreos-devel/board-packages

Let's see how it goes, I updated a link to the Jenkins build in the first message above.

[chewi]

Thanks, I'll try it out. I used a different command (run_merge, as provided by catalyst's chroot-functions.sh) though:

run_merge --onlydeps --onlydeps-with-rdeps=n --onlydeps-with-ddeps=n --onlydeps-with-ideps=y coreos-devel/board-packages

ROOT in that context refers to the SDK, not the board root. Although you're not actually installing any packages to the board root, you ideally want to use its profile etc. You therefore still need to PORTAGE_CONFIGROOT="$ROOT" run_merge --root="$ROOT" --sysroot="$ROOT" like build_target_toolchain() in catalyst_toolchains.sh does. I'd also do this at the end.

[krnowak]

Reverted to listing the packages explicitly - the portage patch did not work as expected.