ci: Windows CI (take 1) (#1930)

First step at getting full Windows CI: This PR gets us as far as doing
a Windows build of OSL for each Windows flavor on GHA.  This is an
important part of making it much harder for us to inadvertently break
the build on Windows.

We currently only build, but we don't run the testsuite. This is
in progress and we don't pass all tests quite yet, but I didn't want
to wait any longer to get this much in our CI.

This is just about getting our own CI runs for Windows and still relies
on helper scripts being run as part of the CI to build or install the
dependencies before the main build starts. We want eventually to get
to the point where OSL's build system can itself auto-build all the
important dependencies if not found and it's not something most
users need to worry about.

A few notes about specific changes in this patch that came along
for the ride:

* Port many build_blah.bash scripts (or recent changes thereto) from
  OIIO to OSL to synchronize them between the two projects and move
  improvements made on the OIIO side to OSL's versions.
* Augment the build_llvm.bash script with logic to work on Windows,
  too, using pre-built binaries associated with LLVM releases. Note
  that not all of their releases have Windows binaries pre-built,
  so you have to pick one of the right releases for it to work.
* Fix a problem with the way we propagate C++ std level flags from the
  main build to the bitcode-generating pass. On Linux and Mac,
  `-std=c++17` works identically on gcc/clang that we use for OSL
  itself and the clang we use to generate precompiled bitcode. But on
  Windows, the standard-setting flags for MSVC we use to build the cpp
  is different from the flags needed for clang when compiling to
  bicode.
* Modify the ci.yml to allow special branch names to exclude platforms,
  to simplify iterating over many CI runs when only caring about one
  platform. If the branch name contains the substring "windows-only",
  the Linux and Mac CI test variants will be skipped, and analogously
  for "macos-only" and "linux-only".

---------

Signed-off-by: Larry Gritz <[email protected]>
Co-authored-by: Brecht Van Lommel <[email protected]>

Author: lgritz

.github/workflows/build-steps.yml

@@ -120,9 +120,6 @@ jobs:
           path: ./ccache
           key: ${{github.job}}-${{inputs.nametag}}-${{steps.ccache_cache_keys.outputs.date}}
           restore-keys: ${{github.job}}-
-      - name: Setup Nuget.exe (Windows only)
-        if: runner.os == 'Windows'
-        uses: nuget/setup-nuget@a21f25cd3998bf370fde17e3f1b4c12c175172f9 # v2.0.0
       - name: Build setup
         shell: bash
         run: |

.github/workflows/ci.yml

@@ -40,6 +40,7 @@ permissions: read-all
 jobs:
 
   aswf-old:
+    if: ${{ ! contains(github.ref, 'windows-only') && ! contains(github.ref, 'macos-only') }}
     name: "VFX${{matrix.vfxyear}} ${{matrix.desc}}"
     strategy:
       fail-fast: false
@@ -202,6 +203,7 @@ jobs:
 
 
   linux:
+    if: ${{ ! contains(github.ref, 'windows-only') && ! contains(github.ref, 'macos-only') }}
     name: "${{matrix.desc}}"
     uses: ./.github/workflows/build-steps.yml
     with:
@@ -356,7 +358,8 @@ jobs:
             batched: b8_AVX2_noFMA
             setenvs: export OSL_CMAKE_FLAGS="-DSTOP_ON_WARNING=OFF -DEXTRA_CPP_ARGS=-fp-model=consistent"
                             OPENIMAGEIO_CMAKE_FLAGS=-DBUILD_FMT_VERSION=7.1.3
-                            USE_OPENVDB=0 OPENCOLORIO_CXX=g++
+                            USE_OPENVDB=0
+                            OPENCOLORIO_CMAKE_FLAGS="-DCMAKE_CXX_COMPILER=g++"
           - desc: icx/C++17 llvm14 py3.10 oiio-3.0 avx2
             nametag: linux-icx
             runner: ubuntu-latest
@@ -371,8 +374,9 @@ jobs:
             pybind11_ver: v2.10.0
             simd: avx2,f16c
             batched: b8_AVX2_noFMA
-            setenvs: export OSL_CMAKE_FLAGS="-DSTOP_ON_WARNING=OFF" USE_OPENVDB=0
-                            OPENCOLORIO_CXX=g++
+            setenvs: export OSL_CMAKE_FLAGS="-DSTOP_ON_WARNING=OFF"
+                            USE_OPENVDB=0
+                            OPENCOLORIO_CMAKE_FLAGS="-DCMAKE_CXX_COMPILER=g++"
 
           - desc: Debug gcc7/C++17 llvm11 py3.8 oiio2.5 exr3.1 sse4
             nametag: linux-debug-gcc7-llvm11
@@ -482,6 +486,7 @@ jobs:
 
 
   macos:
+    if: ${{ ! contains(github.ref, 'windows-only') && ! contains(github.ref, 'linux-only') }}
     name: "${{matrix.desc}}"
     uses: ./.github/workflows/build-steps.yml
     with:
@@ -546,3 +551,83 @@ jobs:
             openimageio_ver: main
 
 
+  windows:
+    if: ${{ ! contains(github.ref, 'linux-only') && ! contains(github.ref, 'macos-only') }}
+    name: "${{matrix.desc}}"
+    uses: ./.github/workflows/build-steps.yml
+    with:
+      nametag: ${{ matrix.nametag || 'unnamed!' }}
+      runner: ${{ matrix.runner }}
+      container: ${{ matrix.container }}
+      cc_compiler: ${{ matrix.cc_compiler }}
+      cxx_compiler: ${{ matrix.cxx_compiler }}
+      cxx_std: ${{ matrix.cxx_std || '17' }}
+      batched: ${{ matrix.batched }}
+      build_type: ${{ matrix.build_type || 'Release' }}
+      depcmds: ${{ matrix.depcmds }}
+      extra_artifacts: ${{ matrix.extra_artifacts }}
+      fmt_ver: ${{ matrix.fmt_ver }}
+      opencolorio_ver: ${{ matrix.opencolorio_ver }}
+      openexr_ver: ${{ matrix.openexr_ver }}
+      openimageio_ver: ${{ matrix.openimageio_ver }}
+      pybind11_ver: ${{ matrix.pybind11_ver }}
+      python_ver: ${{ matrix.python_ver }}
+      setenvs: ${{ matrix.setenvs }}
+      simd: ${{ matrix.simd }}
+      skip_build: ${{ matrix.skip_build }}
+      skip_tests: ${{ matrix.skip_tests }}
+      abi_check: ${{ matrix.abi_check }}
+      build_docs: ${{ matrix.build_docs }}
+      generator: ${{ matrix.generator }}
+      ctest_args: ${{ matrix.ctest_args }}
+      ctest_test_timeout: ${{ matrix.ctest_test_timeout || '800' }}
+      coverage: ${{ matrix.coverage || 0 }}
+      sonar: ${{ matrix.sonar || 0 }}
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - desc: Windows-2019 VS2019
+            runner: windows-2019
+            nametag: windows-2019
+            generator: "Visual Studio 16 2019"
+            python_ver: "3.9"
+            opencolorio_ver: v2.3.2
+            openexr_ver: v3.3.2
+            openimageio_ver: release
+            skip_tests: 1
+            setenvs: export LLVM_VERSION=18.1.8
+                            OSL_CMAKE_FLAGS="-DUSE_LLVM_BTCODE=ON"
+                            PUGIXML_VERSION=v1.14
+                            OpenImageIO_BUILD_MISSING_DEPS="Freetype;TIFF;libdeflate;libjpeg-turbo"
+
+          #
+          # Windows 2022 / MSVS 17 case is below. It's not working yet,
+          # work in progress.
+          #
+          - desc: Windows-2022 VS2022
+            runner: windows-2022
+            nametag: windows-2022
+            generator: "Visual Studio 17 2022"
+            python_ver: "3.9"
+            opencolorio_ver: v2.3.2
+            openexr_ver: v3.3.2
+            openimageio_ver: release
+            skip_tests: 1
+            setenvs: export LLVM_VERSION=18.1.8
+                            OSL_CMAKE_FLAGS="-DUSE_LLVM_BTCODE=ON"
+                            PUGIXML_VERSION=v1.14
+                            OpenImageIO_BUILD_MISSING_DEPS="Freetype;TIFF;libdeflate;libjpeg-turbo"
+          - desc: Windows-2025 VS2022
+            runner: windows-2025
+            nametag: windows-2025
+            generator: "Visual Studio 17 2022"
+            python_ver: "3.9"
+            opencolorio_ver: v2.3.2
+            openexr_ver: v3.3.2
+            openimageio_ver: release
+            skip_tests: 1
+            setenvs: export LLVM_VERSION=18.1.8
+                            OSL_CMAKE_FLAGS="-DUSE_LLVM_BTCODE=ON"
+                            PUGIXML_VERSION=v1.14
+                            OpenImageIO_BUILD_MISSING_DEPS="Freetype;TIFF;libdeflate;libjpeg-turbo"

src/build-scripts/build_libpng.bash

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+
+# Utility script to download and build libpng
+#
+# Copyright Contributors to the Open Shading Language project.
+# SPDX-License-Identifier: BSD-3-Clause
+# https://github.com/AcademySoftwareFoundation/OpenShadingLanguage
+
+# Exit the whole script if any command fails.
+set -ex
+
+# Repo and branch/tag/commit of libpng to download if we don't have it yet
+LIBPNG_REPO=${LIBPNG_REPO:=https://github.com/glennrp/libpng.git}
+LIBPNG_VERSION=${LIBPNG_VERSION:=v1.6.35}
+
+# Where to put libpng repo source (default to the ext area)
+LIBPNG_SRC_DIR=${LIBPNG_SRC_DIR:=${PWD}/ext/libpng}
+# Temp build area (default to a build/ subdir under source)
+LIBPNG_BUILD_DIR=${LIBPNG_BUILD_DIR:=${LIBPNG_SRC_DIR}/build}
+# Install area for libpng (default to ext/dist)
+LIBPNG_INSTALL_DIR=${LIBPNG_INSTALL_DIR:=${PWD}/ext/dist}
+#LIBPNG_CONFIG_OPTS=${LIBPNG_CONFIG_OPTS:=}
+
+pwd
+echo "libpng install dir will be: ${LIBPNG_INSTALL_DIR}"
+
+mkdir -p ./ext
+pushd ./ext
+
+# Clone libpng project from GitHub and build
+if [[ ! -e ${LIBPNG_SRC_DIR} ]] ; then
+    echo "git clone ${LIBPNG_REPO} ${LIBPNG_SRC_DIR}"
+    git clone ${LIBPNG_REPO} ${LIBPNG_SRC_DIR}
+fi
+cd ${LIBPNG_SRC_DIR}
+
+
+echo "git checkout ${LIBPNG_VERSION} --force"
+git checkout ${LIBPNG_VERSION} --force
+
+if [[ -z $DEP_DOWNLOAD_ONLY ]]; then
+    time cmake -S . -B ${LIBPNG_BUILD_DIR} -DCMAKE_BUILD_TYPE=Release \
+               -DCMAKE_INSTALL_PREFIX=${LIBPNG_INSTALL_DIR} \
+               -DPNG_EXECUTABLES=OFF \
+               -DPNG_TESTS=OFF \
+               ${LIBPNG_CONFIG_OPTS}
+    time cmake --build ${LIBPNG_BUILD_DIR} --config Release --target install
+fi
+
+# ls -R ${LIBPNG_INSTALL_DIR}
+popd
+
+
+# Set up paths. These will only affect the caller if this script is
+# run with 'source' rather than in a separate shell.
+export PNG_ROOT=$LIBPNG_INSTALL_DIR
+

src/build-scripts/build_llvm.bash

@@ -12,21 +12,36 @@ set -ex
 echo "Building LLVM"
 uname
 
+: ${LLVM_VERSION:=18.1.8}
+: ${LLVM_INSTALL_DIR:=${PWD}/llvm-install}
+mkdir -p $LLVM_INSTALL_DIR || true
 
 if [[ `uname` == "Linux" ]] ; then
-    : ${LLVM_VERSION:=14.0.0}
-    : ${LLVM_INSTALL_DIR:=${PWD}/llvm-install}
     : ${LLVM_DISTRO_NAME:=ubuntu-18.04}
     LLVMTAR=clang+llvm-${LLVM_VERSION}-x86_64-linux-gnu-${LLVM_DISTRO_NAME}.tar.xz
     echo LLVMTAR = $LLVMTAR
     curl --location https://github.com/llvm/llvm-project/releases/download/llvmorg-${LLVM_VERSION}/${LLVMTAR} -o $LLVMTAR
     ls -l $LLVMTAR
     tar xf $LLVMTAR
     rm -f $LLVMTAR
-    echo "Installed ${LLVM_VERSION} in ${LLVM_INSTALL_DIR}"
-    mkdir -p $LLVM_INSTALL_DIR && true
     mv clang+llvm*/* $LLVM_INSTALL_DIR
-    export LLVM_DIRECTORY=$LLVM_INSTALL_DIR
-    export PATH=${LLVM_INSTALL_DIR}/bin:$PATH
-    # ls -a $LLVM_DIRECTORY
+elif [[ `uname -s` == "Windows" || "${RUNNER_OS}" == "Windows" ]] ; then
+    echo "Installing Windows LLVM"
+    : ${LLVM_DISTRO_NAME:=ubuntu-18.04}
+    LLVMTAR=clang+llvm-${LLVM_VERSION}-x86_64-pc-windows-msvc.tar.xz
+    echo LLVMTAR = $LLVMTAR
+    curl --location https://github.com/llvm/llvm-project/releases/download/llvmorg-${LLVM_VERSION}/${LLVMTAR} -o $LLVMTAR
+    ls -l $LLVMTAR
+    tar xf $LLVMTAR
+    rm -f $LLVMTAR
+    mv clang+llvm*/* $LLVM_INSTALL_DIR
+else
+    echo Bad uname `uname`
 fi
+
+echo "Installed LLVM ${LLVM_VERSION} in ${LLVM_INSTALL_DIR}"
+ls -a $LLVM_INSTALL_DIR || true
+ls -a $LLVM_INSTALL_DIR/* || true
+export LLVM_DIRECTORY=$LLVM_INSTALL_DIR
+export PATH=${LLVM_INSTALL_DIR}/bin:$PATH
+export LLVM_ROOT=${LLVM_INSTALL_DIR}

src/build-scripts/build_opencolorio.bash

@@ -19,10 +19,8 @@ OPENCOLORIO_SOURCE_DIR=${OPENCOLORIO_SOURCE_DIR:=${LOCAL_DEPS_DIR}/OpenColorIO}
 OPENCOLORIO_BUILD_DIR=${OPENCOLORIO_BUILD_DIR:=${LOCAL_DEPS_DIR}/OpenColorIO-build}
 OPENCOLORIO_INSTALL_DIR=${OPENCOLORIO_INSTALL_DIR:=${LOCAL_DEPS_DIR}/dist}
 OPENCOLORIO_BUILD_TYPE=${OPENCOLORIO_BUILD_TYPE:=Release}
-OPENCOLORIO_CXX=${OPENCOLORIO_CXX:=g++}
-OPENCOLORIO_CXX_FLAGS=${OPENCOLORIO_CXX_FLAGS:="-Wno-unused-function -Wno-deprecated-declarations -Wno-cast-qual -Wno-write-strings"}
 # Just need libs:
-OPENCOLORIO_BUILDOPTS="-DOCIO_BUILD_APPS=OFF -DOCIO_BUILD_NUKE=OFF \
+OPENCOLORIO_CMAKE_FLAGS+=" -DOCIO_BUILD_APPS=OFF -DOCIO_BUILD_NUKE=OFF \
                        -DOCIO_BUILD_DOCS=OFF -DOCIO_BUILD_TESTS=OFF \
                        -DOCIO_BUILD_GPU_TESTS=OFF \
                        -DOCIO_BUILD_PYTHON=OFF -DOCIO_BUILD_PYGLUE=OFF \
@@ -46,13 +44,11 @@ echo "git checkout ${OPENCOLORIO_VERSION} --force"
 git checkout ${OPENCOLORIO_VERSION} --force
 echo "Building OpenColorIO from commit" `git rev-parse --short HEAD`
 
-time cmake -S . -B ${OPENCOLORIO_BUILD_DIR} \
+cmake -S ${OPENCOLORIO_SOURCE_DIR} -B ${OPENCOLORIO_BUILD_DIR} \
            -DCMAKE_BUILD_TYPE=${OPENCOLORIO_BUILD_TYPE} \
            -DCMAKE_INSTALL_PREFIX=${OPENCOLORIO_INSTALL_DIR} \
-           -DCMAKE_CXX_COMPILER=${OPENCOLORIO_CXX} \
-           -DCMAKE_CXX_FLAGS="${OPENCOLORIO_CXX_FLAGS}" \
-           ${OPENCOLORIO_BUILDOPTS}
-time cmake --build ${OPENCOLORIO_BUILD_DIR} --config Release --target install
+           ${OPENCOLORIO_CMAKE_FLAGS}
+time cmake --build ${OPENCOLORIO_BUILD_DIR} --target install --config ${OPENCOLORIO_BUILD_TYPE}
 popd
 
 # ls -R ${OPENCOLORIO_INSTALL_DIR}

src/build-scripts/build_openexr.bash

@@ -1,18 +1,17 @@
 #!/usr/bin/env bash
 
-# Utility script to download and build OpenEXR & IlmBase
-#
+# Utility script to download and build OpenEXR & Imath
+
 # Copyright Contributors to the Open Shading Language project.
 # SPDX-License-Identifier: BSD-3-Clause
 # https://github.com/AcademySoftwareFoundation/OpenShadingLanguage
 
-
 # Exit the whole script if any command fails.
 set -ex
 
 # Which OpenEXR to retrieve, how to build it
 OPENEXR_REPO=${OPENEXR_REPO:=https://github.com/AcademySoftwareFoundation/openexr.git}
-OPENEXR_VERSION=${OPENEXR_VERSION:=v3.1.11}
+OPENEXR_VERSION=${OPENEXR_VERSION:=v3.2.4}
 
 # Where to install the final results
 LOCAL_DEPS_DIR=${LOCAL_DEPS_DIR:=${PWD}/ext}
@@ -28,10 +27,10 @@ pwd
 echo "Building OpenEXR ${OPENEXR_VERSION}"
 echo "OpenEXR build dir will be: ${OPENEXR_BUILD_DIR}"
 echo "OpenEXR install dir will be: ${OPENEXR_INSTALL_DIR}"
-echo "OpenEXR Build type is ${OPENEXR_BUILD_TYPE}"
+echo "OpenEXR build type is ${OPENEXR_BUILD_TYPE}"
 echo "CMAKE_PREFIX_PATH is ${CMAKE_PREFIX_PATH}"
 
-# Clone OpenEXR project (including IlmBase) from GitHub and build
+# Clone OpenEXR project (including Imath) from GitHub and build
 if [[ ! -e ${OPENEXR_SOURCE_DIR} ]] ; then
     echo "git clone ${OPENEXR_REPO} ${OPENEXR_SOURCE_DIR}"
     git clone ${OPENEXR_REPO} ${OPENEXR_SOURCE_DIR}
@@ -42,14 +41,13 @@ pushd ${OPENEXR_SOURCE_DIR}
 git checkout ${OPENEXR_VERSION} --force
 echo "Building OpenEXR from commit" `git rev-parse --short HEAD`
 
-cmake   -S . -B ${OPENEXR_BUILD_DIR} \
+cmake   -S ${OPENEXR_SOURCE_DIR} -B ${OPENEXR_BUILD_DIR} \
         -DCMAKE_BUILD_TYPE=${OPENEXR_BUILD_TYPE} \
         -DCMAKE_INSTALL_PREFIX="${OPENEXR_INSTALL_DIR}" \
         -DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-        -DILMBASE_PACKAGE_PREFIX="${OPENEXR_INSTALL_DIR}" \
+        -DBUILD_SHARED_LIBS=${OpenEXR_LOCAL_BUILD_SHARED_LIBS:=ON} \
         -DOPENEXR_BUILD_UTILS=0 \
         -DBUILD_TESTING=0 \
-        -DPYILMBASE_ENABLE=0 \
         -DOPENEXR_VIEWERS_ENABLE=0 \
         -DINSTALL_OPENEXR_EXAMPLES=0 \
         -DOPENEXR_INSTALL_EXAMPLES=0 \
@@ -64,9 +62,12 @@ popd
 
 # Set up paths. These will only affect the caller if this script is
 # run with 'source' rather than in a separate shell.
-export ILMBASE_ROOT=$OPENEXR_INSTALL_DIR
-export OPENEXR_ROOT=$OPENEXR_INSTALL_DIR
-export ILMBASE_LIBRARY_DIR=$OPENEXR_INSTALL_DIR/lib
-export OPENEXR_LIBRARY_DIR=$OPENEXR_INSTALL_DIR/lib
-export LD_LIBRARY_PATH=$OPENEXR_ROOT/lib:$LD_LIBRARY_PATH
+export OpenEXR_ROOT=$OPENEXR_INSTALL_DIR
+# Environment variables are case insensitive on Windows, don't set
+# it a second time with different case. This seems to trigger a bug
+# where the next cmake command fails to find the compiler.
+if [[ $(uname -s) != "Windows" && "${RUNNER_OS}" != "Windows" ]]; then
+    export OPENEXR_ROOT=$OPENEXR_INSTALL_DIR
+fi
+export LD_LIBRARY_PATH=$OpenEXR_ROOT/lib:$LD_LIBRARY_PATH