Fix and slim down CI

[simeonschaub]

Right now, we're running 45 jobs on every push and PR, 17 of which are either failing or timing out. We should slim that down a bit, I don't think we need to be testing every possible combination of memory types, OSes, CPU-architectures as well as local and jll pocl.

One kind of failures are due to the local pocl builds. It seems like the build broke at some point. Can we fix that upstream? Otherwise, I would propose pinning pocl to a version known to build.

The other issues are due to tests timing out on Mac, @maleadt do you have any insights as to why they're timing out? Since they didn't fail at some point, I suspect this might be an upstream pocl issue as well? If that is the case, can we again pin pocl to an older version on Mac?

[maleadt]

The include issue should be easy enough to fix upstream. We should probably document how to set-up a local PoCL dev env (it's annoying having to do so through Yggdrasil). I currently have this helper build script:

#!/usr/bin/env julia

using Pkg
Pkg.activate(@__DIR__)

using LLVM_full_jll,
      SPIRV_Tools_jll, SPIRV_LLVM_Translator_jll,
      OpenCL_jll, OpenCL_Headers_jll,
      Hwloc_jll, CMake_jll

sourcedir = @__DIR__
builddir = joinpath(@__DIR__, "build")
destdir = joinpath(@__DIR__, "target")

prefix = []
for jll in [SPIRV_Tools_jll, SPIRV_LLVM_Translator_jll, OpenCL_jll, OpenCL_Headers_jll, Hwloc_jll]
    push!(prefix, jll.artifact_dir)
end

# XXX: -DLLVM_SPIRV_LIB="" is to force PoCL to use the SPIR-V translator library
#      because the API is not ODR safe on macOS when statically linking LLVM

withenv("LD_LIBRARY_PATH" => joinpath(Sys.BINDIR, Base.PRIVATE_LIBDIR)) do
    if !isdir(builddir)
        mkpath(builddir)
        run(```$(cmake()) -B $builddir -S $sourcedir
                -GNinja
                -DCMAKE_CXX_FLAGS="-fdiagnostics-color=always"
                -DCMAKE_C_FLAGS="-fdiagnostics-color=always"
                -DCMAKE_BUILD_TYPE=Debug
                -DCMAKE_PREFIX_PATH="$(join(prefix, ";"))"
                -DENABLE_EXAMPLES:Bool=OFF
                -DPOCL_DEBUG_MESSAGES:Bool=ON
                -DCMAKE_INSTALL_PREFIX=$destdir
                -DWITH_LLVM_CONFIG=$(LLVM_full_jll.artifact_dir)/tools/llvm-config
                -DKERNELLIB_HOST_CPU_VARIANTS=distro
                -DENABLE_ICD:BOOL=ON
                -DSTATIC_LLVM:Bool=ON
                -DLLVM_SPIRV_LIB=""```)
    end

    run(```$(cmake()) --build $builddir --parallel $(Sys.CPU_THREADS) --target install```)
end

... to be executed in the PoCL tree with this environment:

[deps]
CMake_jll = "3f4e10e2-61f2-5801-8945-23b9d642d0e6"
Hwloc_jll = "e33a78d0-f292-5ffc-b300-72abe9b543c8"
LLVM_full_jll = "a3ccf953-465e-511d-b87f-60a6490c289d"
OpenCL_Headers_jll = "a7aa756b-2b7f-562a-9e9d-e94076c5c8ee"
OpenCL_jll = "6cb37087-e8b6-5417-8430-1f242f1e46e4"
SPIRV_LLVM_Translator_jll = "4a5d46fc-d8cf-5151-a261-86b458210efb"
SPIRV_Tools_jll = "6ac6d60f-d740-5983-97d7-a4482c0689f4"

[compat]
SPIRV_LLVM_Translator_jll = "20"
LLVM_full_jll = "20"

... which I then add to OpenCL's LocalPreferences.toml (after having added a pocl_jll direct dependency) under:

[pocl_jll]
libpocl_path = "/path/to/pocl/target/lib/libpocl.so"

[simeonschaub]

Hmm, I can't reproduce the build failures we are seeing on CI locally. I checked and I don't have hwloc installed otherwise, so I am not sure why it would pass on my machine and fail on all CI machines

[maleadt]

I don't think we need to be testing every possible combination of memory types, OSes, CPU-architectures as well as local and jll pocl.

I kinda agree, but it's hard to pick a selection. When testing was cheap enough, I figured I'd just add all combinations.

The other issues are due to tests timing out on Mac, @maleadt do you have any insights as to why they're timing out? Since they didn't fail at some point, I suspect this might be an upstream pocl issue as well? If that is the case, can we again pin pocl to an older version on Mac?

Hard to tell. I'd say it's more likely to be an issue on our side, but I haven't been able to reproduce locally.

[christiangnrd]

I suggest only testing one memory backend on mac until we figure out what's going on with the hangs. That would bring us from 12 down to 4 tests to wait out.

Things I've noticed on CI:

• The hanging tests don't seem to always be the same ones
• Hanging tests sometimes pass taking much longer than usual, or the normal amount of time

Locally:

• I seem to be getting array out of bounds errors on the large mapreducedims tests Edit: This is happening on my local branch porting some optimizations from other backends, not main branch
• Whenever the tests are running, a process called XprotectService starts using lots of cpu.

Maybe XprotectService is what's causing the hangs, and maybe pocl/pocl#1971 is causing the out of bounds errors?

Maybe I should open a new issue.