Make header search path behavior determined by driver used rather than -fms-compatibility.

Use of `-fms-compatibility` to opt-in to Microsoft header search behavior was
found to cause incompatibility problems with some projects that use the GCC
compatible driver on Windows. This change introduces a new `-fheader-search`
option to control the behavior. Valid values for the option include `gcc` and
`microsoft`. The `clang-cl` driver uses `-fheader-search=microsoft` by default;
all other drivers continue to use `-fheader-search=gcc` by default.

This change affects backward compatibility. The `-fheader-search=` option can
be used to opt in to previous behavior with some caveats. Clang 19 and earlier
followed the GCC header search behavior except when searching for header files
included by quoted inclusion (`#include "file.h"`) when `-fms-compatibility`
is specified; in that case, Microsoft's behavior of searching for the header
file in the directories of each file in the include stack was used. An option
to control this specific behavior is not provided, but could be added if found
to be necessary or useful.

Author: tahonermann

clang/docs/ReleaseNotes.rst

@@ -945,16 +945,17 @@ Windows Support
   allowed as VS2013 and prior allow it.
 
 - Clang now matches MSVC behavior regarding the handling of duplicate header
-  search paths when running in Microsoft compatibility mode. Historically,
-  Clang has mimicked gcc behavior in which user search paths are ordered before
-  system search paths, user search paths that duplicate a (later) system search
+  search paths when run via the ``clang-cl`` driver (by default) or with the
+  ``-fheader-search=microsoft`` option otherwise. Historically, Clang has
+  mimicked GCC behavior in which user search paths are ordered before system
+  search paths, user search paths that duplicate a (later) system search
   path are ignored, and search paths that duplicate an earlier search path of
   the same user/system kind are ignored. This ordering is not compatible with
   the ordering that MSVC uses when paths are duplicated across ``/I`` options
   and the ``INCLUDE`` environment variable.
 
   The order that MSVC uses and that Clang now replicates when the
-  ``-fms-compatibility`` option is enabled follows.
+  ``-fheader-search=microsoft`` option is enabled follows.
 
   - Paths specified by the ``/I`` and ``/external:I`` options are processed in
     the order that they appear. Paths specified by ``/I`` that duplicate a path
@@ -982,6 +983,10 @@ Windows Support
     Paths that duplicate an earlier path in the ``EXTERNAL_INCLUDE``
     environment variable are ignored.
 
+  The ``-fheader-search=gcc`` option can be used to opt in to GCC duplicate
+  header search path handling (which remains the default behavior for the GCC
+  compatible drivers).
+
 LoongArch Support
 ^^^^^^^^^^^^^^^^^
 

clang/docs/UsersManual.rst

@@ -5166,12 +5166,26 @@ follows:
 
 2. Consult the environment.
 
-    TODO: This is not yet implemented.
+    - `/external:env:[VARIABLE]`
 
-    This will consult the environment variables:
+      This command line option specifies a user identified environment variable
+      which is treated as a path delimited (`;`) list of system header paths.
 
-    - `WindowsSdkDir`
-    - `UCRTVersion`
+    - `INCLUDE`
+
+      This environment variable is treated as a path delimited (`;`) list of
+      system header paths.
+
+    - `EXTERNAL_INCLUDE`
+
+      This environment variable is treated as a path delimited (`;`) list of
+      system header paths.
+
+    The following environment variables will be consulted and used to form paths
+    to validate and load content from as appropriate:
+
+      - `WindowsSdkDir`
+      - `UCRTVersion`
 
 3. Fallback to the registry.
 
@@ -5183,6 +5197,8 @@ The Visual C++ Toolset has a slightly more elaborate mechanism for detection.
 
 1. Consult the command line.
 
+    Anything the user specifies is always given precedence.
+
     - `/winsysroot:`
 
     The `/winsysroot:` is used as an equivalent to `-sysroot` on Unix
@@ -5200,21 +5216,30 @@ The Visual C++ Toolset has a slightly more elaborate mechanism for detection.
 
 2. Consult the environment.
 
-    - `/external:[VARIABLE]`
+    - `/external:env:[VARIABLE]`
+
+      This command line option specifies a user identified environment variable
+      which is treated as a path delimited (`;`) list of external header search
+      paths. Additionally, any header search path provided by other means that
+      has a prefix that matches one of these paths is treated as an external
+      header search path.
+
+    - `INCLUDE`
 
-      This specifies a user identified environment variable which is treated as
-      a path delimiter (`;`) separated list of paths to map into `-imsvc`
-      arguments which are treated as `-isystem`.
+      This environment variable is treated as a path delimited (`;`) list of
+      header search paths.
 
-    - `INCLUDE` and `EXTERNAL_INCLUDE`
+    - `EXTERNAL_INCLUDE`
 
-      The path delimiter (`;`) separated list of paths will be mapped to
-      `-imsvc` arguments which are treated as `-isystem`.
+      This environment variable is treated as a path delimited (`;`) list of
+      external header search paths. Additionally, any header search path
+      provided by other means that has a prefix that matches one of these paths
+      is treated as an external header search path.
 
     - `LIB` (indirectly)
 
       The linker `link.exe` or `lld-link.exe` will honour the environment
-      variable `LIB` which is a path delimiter (`;`) set of paths to consult for
+      variable `LIB` which is a path delimited (`;`) set of paths to consult for
       the import libraries to use when linking the final target.
 
     The following environment variables will be consulted and used to form paths

clang/include/clang/Driver/Options.td

@@ -8240,6 +8240,17 @@ def fexperimental_max_bitint_width_EQ:
 // Header Search Options
 //===----------------------------------------------------------------------===//
 
+let Visibility = [ClangOption, CC1Option, CLOption] in {
+
+def fheader_search : Joined<["-"], "fheader-search=">, Group<clang_i_Group>,
+  HelpText<"Specify the method used to resolve included header files">,
+  Values<"gcc,microsoft">,
+  NormalizedValuesScope<"clang::HeaderSearchMode">,
+  NormalizedValues<["GCC", "Microsoft"]>,
+  MarshallingInfoEnum<HeaderSearchOpts<"Mode">, "GCC">;
+
+} // let Visibility = [ClangOption, CC1Option, CLOption]
+
 let Visibility = [CC1Option] in {
 
 def nostdsysteminc : Flag<["-"], "nostdsysteminc">,

clang/include/clang/Lex/HeaderSearchOptions.h

@@ -69,6 +69,11 @@ enum IncludeDirGroup {
 
 } // namespace frontend
 
+/// HeaderSearchMode - The method used to resolve included headers to files.
+/// This controls the order in which include paths are searched and how
+/// duplicate search paths are handled.
+enum class HeaderSearchMode { GCC, Microsoft };
+
 /// HeaderSearchOptions - Helper class for storing options related to the
 /// initialization of the HeaderSearch object.
 class HeaderSearchOptions {
@@ -103,6 +108,9 @@ class HeaderSearchOptions {
         : Prefix(Prefix), IsSystemHeader(IsSystemHeader) {}
   };
 
+  /// The header search mode to use.
+  HeaderSearchMode Mode = HeaderSearchMode::GCC;
+
   /// If non-empty, the directory to use as a "virtual system root" for include
   /// paths.
   std::string Sysroot;

clang/lib/Lex/InitHeaderSearch.cpp

@@ -98,7 +98,7 @@ class InitHeaderSearch {
                               const HeaderSearchOptions &HSOpts);
 
   /// Merges all search path lists into one list and send it to HeaderSearch.
-  void Realize(const LangOptions &Lang);
+  void Realize(const HeaderSearchOptions &HSOpts, const LangOptions &Lang);
 };
 
 }  // end anonymous namespace.
@@ -369,7 +369,7 @@ void InitHeaderSearch::AddDefaultIncludePaths(
 /// and system search paths. If partitioning is not needed, then call with
 /// Part1Begin equal to Part2Begin. The return value is the number of items
 /// removed from the first partition.
-static unsigned RemoveDuplicates(const LangOptions &Lang,
+static unsigned RemoveDuplicates(const HeaderSearchOptions &HSOpts,
                                  std::vector<DirectoryLookupInfo> &SearchList,
                                  unsigned Part1Begin, unsigned Part2Begin,
                                  bool Verbose) {
@@ -430,8 +430,8 @@ static unsigned RemoveDuplicates(const LangOptions &Lang,
       // A path that matches a later path specified by -iexternal is always
       // suppressed.
       DirToRemove = PrevIndex;
-    } else if (!Lang.MSVCCompat && PrevSrcKind == SrcMgr::C_User &&
-               CurSrcKind != SrcMgr::C_User) {
+    } else if (HSOpts.Mode != HeaderSearchMode::Microsoft &&
+               PrevSrcKind == SrcMgr::C_User && CurSrcKind != SrcMgr::C_User) {
       // When not in Microsoft compatibility mode, a user path that matches
       // a later system path is suppressed.
       DirToRemove = PrevIndex;
@@ -489,7 +489,8 @@ mapToUserEntries(const std::vector<DirectoryLookupInfo> &Infos) {
   return LookupsToUserEntries;
 }
 
-void InitHeaderSearch::Realize(const LangOptions &Lang) {
+void InitHeaderSearch::Realize(const HeaderSearchOptions &HSOpts,
+                               const LangOptions &Lang) {
   // Concatenate ANGLE+SYSTEM+AFTER chains together into SearchList.
   std::vector<DirectoryLookupInfo> SearchList;
   SearchList.reserve(IncludePath.size());
@@ -499,7 +500,7 @@ void InitHeaderSearch::Realize(const LangOptions &Lang) {
     if (Include.Group == Quoted)
       SearchList.push_back(Include);
   // Remove duplicate search paths within the quoted inclusion list.
-  RemoveDuplicates(Lang, SearchList, 0, 0, Verbose);
+  RemoveDuplicates(HSOpts, SearchList, 0, 0, Verbose);
   unsigned EndQuoted = SearchList.size();
 
   // Add search paths for angled inclusion next. Note that user paths and
@@ -516,7 +517,7 @@ void InitHeaderSearch::Realize(const LangOptions &Lang) {
   // Remove duplicate search paths within the angled inclusion list.
   // This may leave paths duplicated across the quoted and angled inclusion
   // sections.
-  RemoveDuplicates(Lang, SearchList, EndQuoted, EndQuoted, Verbose);
+  RemoveDuplicates(HSOpts, SearchList, EndQuoted, EndQuoted, Verbose);
   unsigned EndAngled = SearchList.size();
 
   // Add search paths for language dependent system paths next.
@@ -539,7 +540,7 @@ void InitHeaderSearch::Realize(const LangOptions &Lang) {
   // same file. This may result in earlier user paths being removed, and thus
   // requires updating the EndAngled index.
   unsigned NonSystemRemoved =
-      RemoveDuplicates(Lang, SearchList, EndQuoted, EndAngled, Verbose);
+      RemoveDuplicates(HSOpts, SearchList, EndQuoted, EndAngled, Verbose);
   EndAngled -= NonSystemRemoved;
 
   Headers.SetSearchPaths(extractLookups(SearchList), EndQuoted, EndAngled,
@@ -599,5 +600,5 @@ void clang::ApplyHeaderSearchOptions(HeaderSearch &HS,
       HS.getModuleMap().setBuiltinIncludeDir(*Dir);
   }
 
-  Init.Realize(Lang);
+  Init.Realize(HSOpts, Lang);
 }

clang/lib/Lex/PPDirectives.cpp

@@ -23,6 +23,7 @@
 #include "clang/Basic/TokenKinds.h"
 #include "clang/Lex/CodeCompletionHandler.h"
 #include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/HeaderSearchOptions.h"
 #include "clang/Lex/LexDiagnostic.h"
 #include "clang/Lex/LiteralSupport.h"
 #include "clang/Lex/MacroInfo.h"
@@ -999,7 +1000,9 @@ OptionalFileEntryRef Preprocessor::LookupFile(
     // MSVC searches the current include stack from top to bottom for
     // headers included by quoted include directives.
     // See: http://msdn.microsoft.com/en-us/library/36k2cdd4.aspx
-    if (LangOpts.MSVCCompat && !isAngled) {
+    if (getHeaderSearchInfo().getHeaderSearchOpts().Mode ==
+            HeaderSearchMode::Microsoft &&
+        !isAngled) {
       for (IncludeStackInfo &ISEntry : llvm::reverse(IncludeMacroStack)) {
         if (IsFileLexer(ISEntry))
           if ((FileEnt = ISEntry.ThePPLexer->getFileEntry()))

clang/lib/Serialization/ASTReader.cpp

@@ -6166,8 +6166,8 @@ bool ASTReader::ParseHeaderSearchOptions(const RecordData &Record,
                                          ASTReaderListener &Listener) {
   HeaderSearchOptions HSOpts;
   unsigned Idx = 0;
+  HSOpts.Mode = static_cast<HeaderSearchMode>(Record[Idx++]);
   HSOpts.Sysroot = ReadString(Record, Idx);
-
   HSOpts.ResourceDir = ReadString(Record, Idx);
   HSOpts.ModuleCachePath = ReadString(Record, Idx);
   HSOpts.ModuleUserBuildPath = ReadString(Record, Idx);

clang/lib/Serialization/ASTWriter.cpp

@@ -1673,6 +1673,7 @@ void ASTWriter::WriteControlBlock(Preprocessor &PP, StringRef isysroot) {
   const HeaderSearchOptions &HSOpts =
       PP.getHeaderSearchInfo().getHeaderSearchOpts();
 
+  Record.push_back(static_cast<unsigned>(HSOpts.Mode));
   AddString(HSOpts.Sysroot, Record);
   AddString(HSOpts.ResourceDir, Record);
   AddString(HSOpts.ModuleCachePath, Record);

clang/test/Driver/header-search-duplicates.c

@@ -5,9 +5,8 @@
 // RUN: rm -rf %t
 // RUN: split-file %s %t
 
-// This test exercises the clang driver using a target that does not implicitly
-// enable the -fms-compatibility option. The -nostdinc option is used to
-// suppress default search paths to ease testing.
+// This test uses the -nostdinc option to suppress default search paths to
+// ease testing.
 
 // Header search paths are categorized into the following general groups.
 // - Quoted: Search paths that are only used to resolve inclusion of header

clang/test/Driver/microsoft-header-search-duplicates.c

@@ -2,26 +2,8 @@
 // Microsoft compatibility mode. See header-search-duplicates.c for GCC
 // compatible behavior.
 
-// This test is intended to be usable to validate MSVC behavior using a .bat
-// test driver similar to the following. A failure to compile successfully
-// would indicate a problem with the test or, perhaps, a behavioral difference
-// across MSVC versions.
-//   @echo on
-//   setlocal
-//   rd /s/q test-msvc
-//   split-file microsoft-header-search-duplicates.c test-msvc
-//   pushd test-msvc
-//   REM Validate test 1:
-//   set INCLUDE=...
-//   set EXTERNAL_INCLUDE=...
-//   set EXTRA_INCLUDE=...
-//   cl.exe /c /showIncludes ... test1.c
-//   popd
-
-// This test exercises both the clang and clang-cl drivers using a target that
-// implicitly enables the '-fms-compatibility' option. The '-nobuiltininc',
-// '-nostdinc', and '/X ('-nostdlibinc') options are used to suppress implicit
-// header search paths to ease testing.
+// This test uses the '-nobuiltininc', '-nostdinc', and '/X ('-nostdlibinc')
+// options to suppress implicit header search paths to ease testing.
 
 // Header search paths are processed as follows:
 // 1) Paths specified by the '/I' and '/external:I' options are processed in
@@ -38,7 +20,7 @@
 // 3) Paths specified by the 'INCLUDE' environment variable are processed in
 //    order. Paths that duplicate a path from step 1, step 2, or an earlier
 //    path in the 'INCLUDE' environment variable are ignored.
-// 4) Paths specified by the 'EXTERNALINCLUDE' environment variable are
+// 4) Paths specified by the 'EXTERNAL_INCLUDE' environment variable are
 //    processed in order. Paths that duplicate a path from step 1, step 2,
 //    step 3, or an earlier path in the 'EXTERNAL_INCLUDE' environment
 //    variable are ignored.
@@ -51,6 +33,7 @@
 //
 // RUN: %clang \
 // RUN:     -target x86_64-pc-windows -v -fsyntax-only \
+// RUN:     -fheader-search=microsoft \
 // RUN:     -nostdinc \
 // RUN:     -I%t/test1/include/y \
 // RUN:     -I%t/test1/include/z \
@@ -87,6 +70,7 @@
 //
 // RUN: %clang \
 // RUN:     -target x86_64-pc-windows -v -fsyntax-only \
+// RUN:     -fheader-search=microsoft \
 // RUN:     -nostdinc \
 // RUN:     -iexternal %t/test2/include/z \
 // RUN:     -iexternal %t/test2/include/y \
@@ -123,6 +107,7 @@
 //
 // RUN: %clang \
 // RUN:     -target x86_64-pc-windows -v -fsyntax-only \
+// RUN:     -fheader-search=microsoft \
 // RUN:     -nostdinc \
 // RUN:     -iexternal %t/test3/include/w \
 // RUN:     -I%t/test3/include/z \
@@ -193,6 +178,7 @@
 // RUN: env EXTRA_INCLUDE2="%t/test4/include/z;%t/test4/include/y;%t/test4/include/x;%t/test4/include/w" \
 // RUN: %clang \
 // RUN:     -target x86_64-pc-windows -v -fsyntax-only \
+// RUN:     -fheader-search=microsoft \
 // RUN:     -nostdinc \
 // RUN:     -I%t/test4/include/w \
 // RUN:     -iexternal %t/test4/include/x \
@@ -262,6 +248,7 @@
 // RUN: env EXTERNAL_INCLUDE="%t/test5/include/z;%t/test5/include/y;%t/test5/include/w;%t/test5/include/v;%t/test5/include/u" \
 // RUN: %clang \
 // RUN:     -target x86_64-pc-windows -v -fsyntax-only \
+// RUN:     -fheader-search=microsoft \
 // RUN:     -nostdinc \
 // RUN:     -I%t/test5/include/u \
 // RUN:     -iexternal %t/test5/include/v \