[Bug]: dlib binaries/targets built via MSVC cause error when consumed via clang[++]: `no such file or directory: '/bigobj'`

[nyoma-diamond]

What Operating System(s) are you seeing this problem on?

Windows

dlib version

20.0.0

Python version

3.12.7

Compiler

MSVC 19.44.35217 & clang 19.1.5

Expected Behavior

Binaries/targets/configuration files generated using MSVC are usable in projects compiled using clang[++]

Current Behavior

If dlib is built using MSVC, the /bigobj option is always set in the configuration target, causing it to be erroneously passed to the compiler of the consuming project if the consumer is not using MSVC (i.e., using clang[++] instead).

Steps to Reproduce

1. Build dlib using MSVC
2. Link a consuming project to dlib (via find_package(dlib CONFIG REQUIRED) and target_link_libraries(${PROJECT_NAME} PUBLIC dlib::dlib))
3. Build consuming project using clang[++] (e.g., using CMAKE_CXX_COMPILER set to clang++)

Anything else?

When dlib is built using MSVC ([clang-]cl.exe), but then linked to using another (compatible) non-MSVC compiler (i.e., clang[++]), the /bigobj setting persists outside of the relevant context (dlib's compile time), causing an error.

In my case, I'm building dlib using MSVC (cl.exe to be specific) for development on my system / within my development environment. However, when compiling my project I intend to use clang[++]. Other libraries handle this just fine, but in dlib the /bigobj option persists when attempting to run clang[++] on my project, causing the following error:

Cannot get compiler information:
	Compiler exited with error code 1: C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\Llvm\x64\bin\clang++.exe -xc++ -isystemC:/Users/Nyoma/Documents/MyProject/build/debug/vcpkg_installed/x64-windows/include -O0 -std=gnu++20 -D_DEBUG -D_DLL -D_MT -Xclang --dependent-lib=msvcrtd -g -Xclang -gcodeview /bigobj -fpch-preprocess -v -dD -E
	clang version 19.1.5
	Target: x86_64-pc-windows-msvc
	Thread model: posix
	InstalledDir: C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\Llvm\x64\bin
	clang++: error: no such file or directory: '/bigobj'

The problem appears to come from the contents active_compile_opts being injected into the generated dlib.cmake at compile time for dlib, rather than at build time for the consumer, causing /bigopt to be used even when the context has switched to no longer use MSVC.

In the generated dlib.cmake file:

set_target_properties(dlib::dlib PROPERTIES
  INTERFACE_COMPILE_FEATURES "cxx_std_14"
  INTERFACE_COMPILE_OPTIONS "\$<\$<COMPILE_LANGUAGE:CXX>:/bigobj>"  # <-- Here
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "LAPACK::LAPACK;BLAS::BLAS;\$<LINK_ONLY:ws2_32>;\$<LINK_ONLY:winmm>;\$<\$<NOT:\$<CONFIG:DEBUG>>:${VCPKG_IMPORT_PREFIX}/lib/libpng16.lib>;\$<\$<CONFIG:DEBUG>:${VCPKG_IMPORT_PREFIX}/debug/lib/libpng16d.lib>;\$<\$<NOT:\$<CONFIG:DEBUG>>:${VCPKG_IMPORT_PREFIX}/lib/zlib.lib>;\$<\$<CONFIG:DEBUG>:${VCPKG_IMPORT_PREFIX}/debug/lib/zlibd.lib>;\$<\$<NOT:\$<CONFIG:DEBUG>>:${VCPKG_IMPORT_PREFIX}/lib/jpeg.lib>;\$<\$<CONFIG:DEBUG>:${VCPKG_IMPORT_PREFIX}/debug/lib/jpeg.lib>"
)

This is caused by /bigobj being appended to the active_compile_opts variable in dlib/cmake_utils/set_compiler_specific_options.cmake:

if(MSVC)
   #...
   list(APPEND active_compile_opts "/bigobj")
   #...
endif()

and then added to the target compile properties via the following in dlib/CMakeLists.txt:

if((MSVC AND CMAKE_VERSION VERSION_LESS 3.11))
   target_compile_options(dlib PUBLIC ${active_compile_opts})
   target_compile_options(dlib PRIVATE ${active_compile_opts_private})
else()
   target_compile_options(dlib PUBLIC $<$<COMPILE_LANGUAGE:CXX>:${active_compile_opts}>)
   target_compile_options(dlib PRIVATE $<$<COMPILE_LANGUAGE:CXX>:${active_compile_opts_private}>)
endif()

((Side note 1: set_compiler_specific_options.cmake also appends /MP into active_compile_opts right after /bigobj, but it doesn't seem to appear in dlib.cmake or the clang++ call for which the error occurs. Not really sure why))

This appears to effectively apply the assumption that if MSVC was used to compile dlib initially, then any consumer must also be using MSVC, which is not necessarily true. This also introduces the problem of being unable to distribute pre-compiled dlib binaries with their generated config/target files for Windows unless we mandate that the consumer is using MSVC. If this behavior is expected for some reason, it should really be documented, but in practice it would be much better to just remove this restriction altogether. ((Side note 2: shouldn't $<$<COMPILE_LANGUAGE:CXX>: be applied to each element of active_compile_opts[_private]? The current approach only applies it to the first option if multiple are present. edit: I misread the syntax, this is fine)).

In practice, I believe this would be solved by restructuring the relevant checks so that the necessary target_compile_options calls are performed by dlib/cmake_utils/dlibConfig.cmake.in / dlibConfig.cmake when dlib is linked/consumed rather than initially compiled, and/or via via CMAKE_C[XX]_FLAGS, as these are the approaches I see from other libraries (namely CGAL). ((Side note 3: Despite having functionality for it, other dependencies that I'm using don't seem to add /bigobj to INTERFACE_COMPILE_OPTIONS even when using MSVC exclusively. I'm not super well-versed in this side of CMake/compiler interaction so I'm not 100% clear what's necessary or not here, but I thought I'd mention it)). Edit: This is wrong/misdiagnosis where/why the flag propagates in dlib but not in other libraries. Actually resolved by changing target options/features from PUBLIC to PRIVATE. See: #3125 (comment)

This is somewhat related to #2921, which noticed an issue related /bigobj when using g++/MinGW, but failed to sufficiently explain their actual configuration details or investigate what was causing the problem. This lead to vcpkg being pointed to as a potential culprit (which I believe is almost certainly wrong), or possibly attempting to link MSVC binaries with MinGW (a much more likely cause in the case of #2921 specifically, but they don't provide enough details to verify whether or not they built dlib using MSVC or MinGW).

In my case, I'm running into this issue in a repeatable context without MinGW. Admittedly, I am running into this error via vcpkg, which @davisking said the port for which is not developed by the maintainers of this library directly (#2921 (comment)); however I expect this issue would persist even outside of the vcpkg context due to the /bigobj being directly applied to the dlib.cmake target when dlib is built (I just don't currently have a workflow set up to test this outside of vcpkg).

Edit: A significant realization is that this bug makes cross-compiling much more challenging on some setups (if possible at all). Because host-side development may occur using dlib compiled via MSVC, but then target a non-MSVC compatible system for distribution (e.g., Linux, MacOS) and thus require a different compiler/linker, the /bigobj compile option will cause a build failure

[nyoma-diamond]

Following up based on some further experimentation:

Could it be clarified why target_compile_options is PUBLIC for these compiler options at all? This ends up enforcing rules on the consumer which, as far as I know, shouldn't be necessary.

From my experimentation, this bug can be resolved by just changing

if((MSVC AND CMAKE_VERSION VERSION_LESS 3.11))
   target_compile_options(dlib PUBLIC ${active_compile_opts})
   target_compile_options(dlib PRIVATE ${active_compile_opts_private})
else()
   target_compile_options(dlib PUBLIC $<$<COMPILE_LANGUAGE:CXX>:${active_compile_opts}>)
   target_compile_options(dlib PRIVATE $<$<COMPILE_LANGUAGE:CXX>:${active_compile_opts_private}>)
endif()

to

target_compile_options(dlib PRIVATE $<$<COMPILE_LANGUAGE:CXX>:${active_compile_opts}>)

and making /MP stored to active_compile_opts instead of active_compile_opts_private in set_compiler_specific_options.cmake (removing active_compile_opts_private altogether). To my understanding, the restriction preventing using $<$<COMPILE_LANGUAGE:CXX>...> on MSVC with CMake < 3.11 only applies to PUBLIC/INTERFACE, not PRIVATE, so this greatly simplifies things.

Also, this type of misconfiguration seems like it might bleed into other options as well (not just /bigobj), as dlib_needed_public_cflags is also applied with PUBLIC. This is populated by FFMPEG_CFLAGS and JXL_CFLAGS retrieved by pkg_check_modules. Given that necessary FFMPEG/JXL details may change when switching compilers (e.g., in my case) or systems (if using pre-built binaries), I believe dlib_needed_public_cflags should actually become private. I think this also applies to dlib_needed_public_ldflags as well. I'd argue that handling requirements like this should be the responsibility of the consumer anyways; not dlib. It's not clear why these are set publicly by dlib as it creates unnecessary transitive restrictions.

Please let me know if I'm missing something here.

[davisking]

Options like /bigobj need to be set in client builds or things don't work in many real situations. I.e. people try to use dlib with MSVC and get build errors until /bigobj is added.

Sounds like the thing to do is to make those options not go to the client when the client isn't actually using MSVC. Send me a PR that makes that happen for your setup :) It's important to not mess up other existing users though, so things should not change for them.