New Yocto & Swift versions, repo maintenance

[xtremekforever]

@jeremy-prater, we have started working on this repo to provide some small maintenance tasks. The first thing we've done and succeeded at is updating the Yocto version from Dunfell (3.1) to Scarthgap (5.0) while maintaining the Swift version at 5.7.1. My question is, would you be willing to continue maintaining/hosting meta-swift under here (jeremy-prater/meta-swift), and we can simply PR things to you? Or shall we just fork and maintain our own meta-swift separately?

A couple of ideas/goals we have for this layer are:

1. Maintain support for a few recent versions of Yocto, starting at Scarthgap and going up (5.1, 5.2, 5.3, and so on).
   a. One thing we want to do is completely remove the hardcoded SWIFT_GCC_VERSION and make it dynamic instead.
2. Update the layer to support 6.1 in a clean and nice way, leveraging Swift 6.1 #9 but cleaning it up as much as possible (removing LLVM cmake files entirely). Maybe @colemancda can provide input on this part.
3. Add more libraries and modules to the layer. Such as: a. Add xctest and swift-testing recipes that can be included (or not) in an image, or for given Swift app recipes.
   b. Enable more Swift modules as well, such as StringProcessing, RegexParser, and more. I have a full list of modules that can be built for armv7 here: https://github.com/xtremekforever/swift-armv7/releases/tag/6.1.2.

There is more we "could" do, but I think this would be a great start and something that could really be meaningful and useable for potential users of the layer. So, what do you think- shall we work together to maintain this? Are you agreeable to the idea?

[compenguy]

@xtremetekforever I use a fork of this meta-swift, but I would love to go back to using @jeremy-prater 's. I'm still on kirkstone, but I would wholeheartedly support getting your proposed changes merged in here if they worked for me.

I would even be happy to help test and do cleanup on your fork to get it ready for PR, when/if you have something.

Are you wanting to see #9 merged, and then your changes would go on top of that?

[compenguy]

   a. One thing we want to do is completely remove the hardcoded `SWIFT_GCC_VERSION` and make it dynamic instead.

I've been looking at this, and tried a couple different solutions:

1. mapping DISTRO_VERSION values to gcc versions, but DISTRO_VERSION is a value that yocto encourages developers to override with their own value
2. using an anonymous python function to invoke gcc --version, but that's problematic because we need the value of this at recipe parse time, but you can't expect sysroot-native to be present at recipe parse time
3. iterating over keys in the bitbake context dictionary d, looking for one with the string prefix RECIPE_MAINTAINER:pn-gcc-source-, and parsing it out of that

Solution 3 is the most practical and reliable option I've found so far. It's not pretty, but it works.

-SWIFT_GCC_VERSION = "13.3.0"
+# Determine SWIFT_GCC_VERSION by examining bitbake's context dictionary key RECIPE_MAINTAINER:pn-gcc-source-<version>
+python () {
+    gcc_src_maint_pkg = [x for x in d if x.startswith("RECIPE_MAINTAINER:pn-gcc-source-")][0]
+    bb.note(f"maintainer gcc source pkg: {gcc_src_maint_pkg}")
+    gcc_ver = gcc_src_maint_pkg.rpartition("-")[2]
+    bb.note(f"gcc source pkg version: {gcc_ver}")
+
+    d.setVar("SWIFT_GCC_VERSION", gcc_ver)
+}

[lhoward]

1. Maintain support for a few recent versions of Yocto, starting at Scarthgap and going up (5.1, 5.2, 5.3, and so on).
   a. One thing we want to do is completely remove the hardcoded SWIFT_GCC_VERSION and make it dynamic instead.

I've applied @compenguy's patch from this PR, seems to work.

2. Update the layer to support 6.1 in a clean and nice way, leveraging Swift 6.1 #9 but cleaning it up as much as possible (removing LLVM cmake files entirely). Maybe @colemancda can provide input on this part.

I think the real way to fix this is to build Swift from scratch. But this means building both Swift 5.9 and 6.1. Downloading the LLVM CMake files from an external URL (as is done in the current main branch) is worse than embedding them IMO, as that URL is unlikely to be as stable as Apple's own.

4. Add more libraries and modules to the layer. Such as: a. Add xctest and swift-testing recipes that can be included (or not) in an image, or for given Swift app recipes.> b. Enable more Swift modules as well, such as StringProcessing, RegexParser, and more. I have a full list of modules that can be built for armv7 here: https://github.com/xtremekforever/swift-armv7/releases/tag/6.1.2.

xctest and swift-testing are in my PR, FWIW, as are StringProcessing, RegexParser and a few other things:

FILES:${PN} = "\
    ${libdir}/swift/linux/libswift_RegexParser.so \
    ${libdir}/swift/linux/libswiftSwiftPrivateThreadExtras.so \
    ${libdir}/swift/linux/libswift_Concurrency.so \
    ${libdir}/swift/linux/libswift_Differentiation.so \
    ${libdir}/swift/linux/libswiftDifferentiationUnittest.so \
    ${libdir}/swift/linux/libswiftDistributed.so \
    ${libdir}/swift/linux/libswiftRegexBuilder.so \
    ${libdir}/swift/linux/libswiftObservation.so \
    ${libdir}/swift/linux/libswiftSwiftOnoneSupport.so \
    ${libdir}/swift/linux/libswiftSwiftPrivateLibcExtras.so \
    ${libdir}/swift/linux/libswiftRuntimeUnittest.so \
    ${libdir}/swift/linux/libswift_StringProcessing.so \
    ${libdir}/swift/linux/libswiftGlibc.so \
    ${libdir}/swift/linux/libswiftCore.so \
    ${libdir}/swift/linux/libswift_Builtin_float.so \
    ${libdir}/swift/linux/libswiftSwiftPrivate.so \
    ${libdir}/swift/linux/libswiftSynchronization.so \
    ${libdir}/swift/linux/libswiftStdlibUnittest.so \

[xtremekforever]

   a. One thing we want to do is completely remove the hardcoded `SWIFT_GCC_VERSION` and make it dynamic instead.

I've been looking at this, and tried a couple different solutions:

1. mapping DISTRO_VERSION values to gcc versions, but DISTRO_VERSION is a value that yocto encourages developers to override with their own value
2. using an anonymous python function to invoke gcc --version, but that's problematic because we need the value of this at recipe parse time, but you can't expect sysroot-native to be present at recipe parse time
3. iterating over keys in the bitbake context dictionary d, looking for one with the string prefix RECIPE_MAINTAINER:pn-gcc-source-, and parsing it out of that

Solution 3 is the most practical and reliable option I've found so far. It's not pretty, but it works.

-SWIFT_GCC_VERSION = "13.3.0"
+# Determine SWIFT_GCC_VERSION by examining bitbake's context dictionary key RECIPE_MAINTAINER:pn-gcc-source-<version>
+python () {
+    gcc_src_maint_pkg = [x for x in d if x.startswith("RECIPE_MAINTAINER:pn-gcc-source-")][0]
+    bb.note(f"maintainer gcc source pkg: {gcc_src_maint_pkg}")
+    gcc_ver = gcc_src_maint_pkg.rpartition("-")[2]
+    bb.note(f"gcc source pkg version: {gcc_ver}")
+
+    d.setVar("SWIFT_GCC_VERSION", gcc_ver)
+}

One thing we figured out on the Swift 5.7.1 master branch is that SWIFT_GCC_VERSION becomes necessary if this function is left in the recipe:

do_create_gcc_version_symlinks() {
    GCC_VERSION=`basename ${STAGING_DIR_TARGET}/usr/include/c++/*`

    if [ ! -L "${STAGING_DIR_TARGET}/usr/lib/${TARGET_SYS}/current" ]; then
        cd ${STAGING_DIR_TARGET}/usr/lib/${TARGET_SYS}
        ln -s -r ${GCC_VERSION} current
    fi

    if [ ! -L "${STAGING_DIR_TARGET}/usr/include/c++/current" ]; then
        cd ${STAGING_DIR_TARGET}/usr/include/c++
        ln -s -r ${GCC_VERSION} current
    fi

    if [ ! -L "${STAGING_DIR_NATIVE}/usr/lib/${TARGET_SYS}/gcc/${TARGET_SYS}/current" ]; then
        cd ${STAGING_DIR_NATIVE}/usr/lib/${TARGET_SYS}/gcc/${TARGET_SYS}
        ln -s -r ${GCC_VERSION} current
    fi
}

And, you just give all the flags the paths to current:

EXTRA_INCLUDE_FLAGS = "\
    -I${STAGING_DIR_TARGET}/usr/include/c++/current/${TARGET_SYS} \
    -I${STAGING_DIR_TARGET}/usr/include/c++/current"
TARGET_LDFLAGS += "-w -fuse-ld=lld -L${STAGING_DIR_TARGET}/usr/lib/${TARGET_SYS}/current"
SWIFT_C_FLAGS = "-w -fuse-ld=lld -target ${SWIFT_TARGET_NAME} \
    --sysroot ${STAGING_DIR_TARGET} \
    -B${STAGING_DIR_TARGET}/usr/lib/${TARGET_SYS}/current \
    -L${STAGING_DIR_TARGET}/usr/lib/${TARGET_SYS}/current \
    ${EXTRA_INCLUDE_FLAGS}"

I'm not saying this is the "best" solution, but it's worked well for 5.7.1. Would this be a consideration for the 6.1 changes as well?

[xtremekforever]

1. Maintain support for a few recent versions of Yocto, starting at Scarthgap and going up (5.1, 5.2, 5.3, and so on).
   a. One thing we want to do is completely remove the hardcoded SWIFT_GCC_VERSION and make it dynamic instead.

I've applied @compenguy's patch from this PR, seems to work.

2. Update the layer to support 6.1 in a clean and nice way, leveraging Swift 6.1 #9 but cleaning it up as much as possible (removing LLVM cmake files entirely). Maybe @colemancda can provide input on this part.

I think the real way to fix this is to build Swift from scratch. But this means building both Swift 5.9 and 6.1. Downloading the LLVM CMake files from an external URL (as is done in the current main branch) is worse than embedding them IMO, as that URL is unlikely to be as stable as Apple's own.

4. Add more libraries and modules to the layer. Such as: a. Add xctest and swift-testing recipes that can be included (or not) in an image, or for given Swift app recipes.> b. Enable more Swift modules as well, such as StringProcessing, RegexParser, and more. I have a full list of modules that can be built for armv7 here: https://github.com/xtremekforever/swift-armv7/releases/tag/6.1.2.

xctest and swift-testing are in my PR, FWIW, as are StringProcessing, RegexParser and a few other things:

FILES:${PN} = "
${libdir}/swift/linux/libswift_RegexParser.so
${libdir}/swift/linux/libswiftSwiftPrivateThreadExtras.so
${libdir}/swift/linux/libswift_Concurrency.so
${libdir}/swift/linux/libswift_Differentiation.so
${libdir}/swift/linux/libswiftDifferentiationUnittest.so
${libdir}/swift/linux/libswiftDistributed.so
${libdir}/swift/linux/libswiftRegexBuilder.so
${libdir}/swift/linux/libswiftObservation.so
${libdir}/swift/linux/libswiftSwiftOnoneSupport.so
${libdir}/swift/linux/libswiftSwiftPrivateLibcExtras.so
${libdir}/swift/linux/libswiftRuntimeUnittest.so
${libdir}/swift/linux/libswift_StringProcessing.so
${libdir}/swift/linux/libswiftGlibc.so
${libdir}/swift/linux/libswiftCore.so
${libdir}/swift/linux/libswift_Builtin_float.so
${libdir}/swift/linux/libswiftSwiftPrivate.so
${libdir}/swift/linux/libswiftSynchronization.so
${libdir}/swift/linux/libswiftStdlibUnittest.so \

Excellent @lhoward . I think we'd like to incorporate almost all of this into the meta-swift fork, but we're taking it one step at a time as a learning experience. I just don't know if @jeremy-prater will have any motivation to merge this stuff into his repo, so that's why we've branched off here (https://github.com/xavgru12/meta-swift) to continue developing the layer and adding things piece by piece.

For sure, we'll want to download the LLVM sources instead of including the cmake files directly, and it's relatively easy to just run the cmake configure step without actually building it.

[lhoward]

Sounds great. From my perspective as long as it still supports the things we need (like SSH_AUTH_SOCK forwarding etc) I'll be very happy to switch over to your fork.

[xtremekforever]

What does the SSH_AUTH_SOCK forwarding do? Not familiar with this...

[lhoward]

I need the SSH socket from the host environment to leak into the build environment so I can check out Swift package dependencies that are in private repositories.

[compenguy]

@xtremekforever I don't want to speak for @jeremy-prater , but I suspect as long as we settle on one place to do all the work, and have compatible expectations for yocto release support management, it doesn't matter much where it lives.

I think the worst of all possible worlds is having three separate forks that pick up features and fixes piecemeal, and people duplicate work.

In my day job, I am working off a private fork of @jeremy-prater 's work from back in the 5.1 days, with an independent implementation of swift 5.7 support for kirkstone (although it's based heavily on colemancda's and your work). I'd like to not do all the heavy lifting myself, and just pool efforts with other interested parties, like you guys.

[lhoward]

Agreed. It's just a question of, if things we depend on get dropped along the way (moreso in the classes rather the recipes, as recipes can be bbappend-ed), then I'll probably keep maintaining my fork just because it's a core part of our product's build system.

[xtremekforever]

I agree. Well, let's see if Jeremy wants to host all the stuff here...! @jeremy-prater

[jeremy-prater]

Just merged @xtremekforever clean up PR. I guess the next appropriate step would be to make branches for kirkstone and scarthgap to support those layer versions. And then update to swift 6.x?

I'm not sure about swift 6.x support on armv7 devices though...

[lhoward]

Just merged @xtremekforever clean up PR. I guess the next appropriate step would be to make branches for kirkstone and scarthgap to support those layer versions. And then update to swift 6.x?

I'm not sure about swift 6.x support on armv7 devices though...

We're working on it (well, @xtremekforever is). At the very least I believe we need to disable some features in stdlib.

[compenguy]

I'm not sure about swift 6.x support on armv7 devices though...

We're working on it (well, @xtremekforever is). At the very least I believe we need to disable some features in stdlib.

I think there are further caveats for armv7, like swift-crypto symbol collisions because they do symbol renaming in a vendored BoringSSL lib for arm64 and x86_64, but not armv7.