[Clang] Support for MSVC compatible header search path ordering.

Clang has historically matched GCC's behavior for header search path order
and pruning of duplicate paths. That traditional behavior is to order user
search paths before system search paths, to ignore user search paths that
duplicate a (later) system search path, and to ignore search paths that
duplicate an earlier search path of the same user/system kind. This differs
from MSVC and can result in inconsistent header file resolution for `#include`
directives.

MSVC orders header search paths as follows:
1) Paths specified by the `/I` and `/external:I` options are processed in
   the order that they appear. Paths specified by `/I` that duplicate a path
   specified by `/external:I` are ignored regardless of the order of the
   options. Paths specified by `/I` that duplicate a path from a prior `/I`
   option are ignored. Paths specified by `/external:I` that duplicate a
   path from a later `/external:I` option are ignored.
2) Paths specified by `/external:env` are processed in the order that they
   appear. Paths that duplicate a path from a `/I` or `/external:I` option
   are ignored regardless of the order of the options. Paths that duplicate a
   path from a prior `/external:env` option or an earlier path from the same
   `/external:env` option are ignored.
3) Paths specified by the `INCLUDE` environment variable are processed in
   the order they appear. Paths that duplicate a path from a `/I`,
   `/external:I`, or `/external:env` option are ignored. Paths that
   duplicate an earlier path in the `INCLUDE` environment variable are
   ignored.
4) Paths specified by the `EXTERNAL_INCLUDE` environment variable are
   processed in the order they appear. Paths that duplicate a path from a
   `/I`, `/external:I`, or `/external:env` option are ignored. Paths that
   duplicate a path from the `INCLUDE` environment variable are ignored.
   Paths that duplicate an earlier path in the `EXTERNAL_INCLUDE
   environment variable are ignored.

Prior to this change, Clang handled the MSVC `/external:I` and `/external:env`
options and the paths present in the `INCLUDE` and `EXTERNAL_INCLUDE`
environment variables as though they were specified with the `-isystem` option.
The GCC behavior described above then lead to a command line such as
`/external:I dir1 /Idir2` having a header search order of `dir2` followed by
`dir1`; contrary to MSVC behavior.

Paths specified by the MSVC `/external:I` or `/external:env` options or by the
`EXTERNAL_INCLUDE` environment variable are not just used to nominate a header
search path. MSVC also uses these paths as external directory prefixes to mark
paths that are to be treated as system directories. When a header file is
resolved to a path for which one of these paths is a prefix match, that header
file is treated as a system header regardless of whether the header file was
resolved against a `/I` specified path. Note that it is not necessary for the
final path component of the external path to match a directory in the filesystem
in order to be used as an external directory prefix, though trailing path
separators are significant. For example:

   Include directive         Command line         System header
   ------------------------  ------------------   -------------
   #include <foobar/file.h>  /I. /external:Ifoo   Yes
   #include <foobar/file.h>  /I. /external:Ifoo\  No

This change adds support for the MSVC external path concept through the addition
of new driver options with the following option syntax.
   clang                     clang-cl
   ------------------------  -------------------
   -iexternal <dir>          /external:I <dir>
   -iexternal-env=<ENV>      /external:env:<ENV>

Paths specified by these options are treated as system paths. That is, whether
warnings are issued in header files found via these paths remains subject to
use of the `-Wsystem-headers` and `-Wno-system-headers` options. Note that the
MSVC `/external:W<N>` options are mapped to these options.

The MSVC behavior described above implies that (system) paths present in the
`INCLUDE` and `EXTERNAL_INCLUDE` environment variables do not suppress matching
user paths specified via `/I`. This contrasts with GCC's behavior, and Clang's
historical behavior, of suppressing user paths that match a system path
regardless of how each is specified. In order to support both behaviors, the
following driver option has been added. The option arguments shown reflect the
default behavior for each driver.

   clang                     clang-cl
   ------------------------  -------------------------
   -fheader-search=gcc       -fheader-search=microsoft

Use of the MSVC compatible header search path order by default for `clang-cl`
is a change in behavior with potential to cause problems for some projects that
build with `clang-cl`. Potentially impacted projects include those that specify
a header search path via either the `/I` option or in the `CPATH` environment
variable, but rely on the path being suppressed and/or treated as a system
header due to a duplicate path specified by another MSVC option or environment
variable or by the `-imsvc` option or one of the `-isystem` family of options.
Such projects can pass the `-fheader-search=gcc` option in their `clang-cl`
invocations to (mostly) restore previous behavior.

Clang emulates the MSVC behavior of resolving quoted header file inclusions
(e.g., `#include "file.h"`) by searching for a matching file in the directories
for the including files in the include stack. See Microsoft's documentation of
this feature in the Microsoft-specific section of
https://learn.microsoft.com/en-us/cpp/preprocessor/hash-include-directive-c-cpp.
Previously, this behavior was implicitly enabled when the `-fms-compatibility`
option is specified (implicitly for Windows targets). This change ties this
behavior to the `-fheader-search=microsoft` option instead. Projects that use
the `clang` (not `clang-cl`) driver to build code targeting Windows that depend
on this local header file lookup may be impacted by this change. Such projects
can try adding `-fheader-search=microsoft` to their `clang` invocations to
restore the prior behavior.

There is at least one behavior that MSVC exhibits that is not currently
replicated with these changes. Clang treats paths specified in the `INCLUDE`
environment variable as system directories; MSVC does not. Clang will therefore
suppress warnings in header files found via these paths by default while MSVC
will not. However, recent MSVC releases set the `EXTERNAL_INCLUDE` environment
variable to have the same paths as `INCLUDE` by default, so, for recent
releases, MSVC appears to suppress warnings for header files found via
`INCLUDE` by default when it does not. Note that many Microsoft provided header
files use `#pragma` directives to suppress warnings.

Author: tahonermann

clang/docs/ReleaseNotes.rst

@@ -1110,6 +1110,49 @@ Windows Support
   When `-fms-compatibility-version=18.00` or prior is set on the command line this Microsoft extension is still
   allowed as VS2013 and prior allow it.
 
+- Clang now matches MSVC behavior regarding the handling of duplicate header
+  search paths when run via the ``clang-cl`` driver (by default) or with the
+  ``-fheader-search=microsoft`` option otherwise. Historically, Clang has
+  mimicked GCC behavior in which user search paths are ordered before system
+  search paths, user search paths that duplicate a (later) system search
+  path are ignored, and search paths that duplicate an earlier search path of
+  the same user/system kind are ignored. This ordering is not compatible with
+  the ordering that MSVC uses when paths are duplicated across ``/I`` options
+  and the ``INCLUDE`` environment variable.
+
+  The order that MSVC uses and that Clang now replicates when the
+  ``-fheader-search=microsoft`` option is enabled follows.
+
+  - Paths specified by the ``/I`` and ``/external:I`` options are processed in
+    the order that they appear. Paths specified by ``/I`` that duplicate a path
+    specified by ``/external:I`` are ignored regardless of the order of the
+    options. Paths specified by ``/I`` that duplicate a path from a prior ``/I``
+    option are ignored. Paths specified by ``/external:I`` that duplicate a
+    path from a later ``/external:I`` option are ignored.
+
+  - Paths specified by ``/external:env`` are processed in the order that they
+    appear. Paths that duplicate a path from a ``/I`` or ``/external:I`` option
+    are ignored regardless of the order of the options. Paths that duplicate a
+    path from a prior ``/external:env`` option or an earlier path from the same
+    ``/external:env`` option are ignored.
+
+  - Paths specified by the ``INCLUDE`` environment variable are processed in
+    the order they appear. Paths that duplicate a path from a ``/I``,
+    ``/external:I``, or ``/external:env`` option are ignored. Paths that
+    duplicate an earlier path in the ``INCLUDE`` environment variable are
+    ignored.
+
+  - Paths specified by the ``EXTERNAL_INCLUDE`` environment variable are
+    processed in the order they appear. Paths that duplicate a path from a
+    ``/I``, ``/external:I``, or ``/external:env`` option are ignored. Paths that
+    duplicate a path from the ``INCLUDE`` environment variable are ignored.
+    Paths that duplicate an earlier path in the ``EXTERNAL_INCLUDE``
+    environment variable are ignored.
+
+  The ``-fheader-search=gcc`` option can be used to opt in to GCC duplicate
+  header search path handling (which remains the default behavior for the GCC
+  compatible drivers).
+
 LoongArch Support
 ^^^^^^^^^^^^^^^^^
 

clang/docs/UsersManual.rst

@@ -5196,12 +5196,26 @@ follows:
 
 2. Consult the environment.
 
-    TODO: This is not yet implemented.
+    - `/external:env:[VARIABLE]`
 
-    This will consult the environment variables:
+      This command line option specifies a user identified environment variable
+      which is treated as a path delimited (`;`) list of system header paths.
 
-    - `WindowsSdkDir`
-    - `UCRTVersion`
+    - `INCLUDE`
+
+      This environment variable is treated as a path delimited (`;`) list of
+      system header paths.
+
+    - `EXTERNAL_INCLUDE`
+
+      This environment variable is treated as a path delimited (`;`) list of
+      system header paths.
+
+    The following environment variables will be consulted and used to form paths
+    to validate and load content from as appropriate:
+
+      - `WindowsSdkDir`
+      - `UCRTVersion`
 
 3. Fallback to the registry.
 
@@ -5213,6 +5227,8 @@ The Visual C++ Toolset has a slightly more elaborate mechanism for detection.
 
 1. Consult the command line.
 
+    Anything the user specifies is always given precedence.
+
     - `/winsysroot:`
 
     The `/winsysroot:` is used as an equivalent to `-sysroot` on Unix
@@ -5230,21 +5246,30 @@ The Visual C++ Toolset has a slightly more elaborate mechanism for detection.
 
 2. Consult the environment.
 
-    - `/external:[VARIABLE]`
+    - `/external:env:[VARIABLE]`
+
+      This command line option specifies a user identified environment variable
+      which is treated as a path delimited (`;`) list of external header search
+      paths. Additionally, any header search path provided by other means that
+      has a prefix that matches one of these paths is treated as an external
+      header search path.
+
+    - `INCLUDE`
 
-      This specifies a user identified environment variable which is treated as
-      a path delimiter (`;`) separated list of paths to map into `-imsvc`
-      arguments which are treated as `-isystem`.
+      This environment variable is treated as a path delimited (`;`) list of
+      header search paths.
 
-    - `INCLUDE` and `EXTERNAL_INCLUDE`
+    - `EXTERNAL_INCLUDE`
 
-      The path delimiter (`;`) separated list of paths will be mapped to
-      `-imsvc` arguments which are treated as `-isystem`.
+      This environment variable is treated as a path delimited (`;`) list of
+      external header search paths. Additionally, any header search path
+      provided by other means that has a prefix that matches one of these paths
+      is treated as an external header search path.
 
     - `LIB` (indirectly)
 
       The linker `link.exe` or `lld-link.exe` will honour the environment
-      variable `LIB` which is a path delimiter (`;`) set of paths to consult for
+      variable `LIB` which is a path delimited (`;`) set of paths to consult for
       the import libraries to use when linking the final target.
 
     The following environment variables will be consulted and used to form paths

clang/include/clang/Driver/Options.td

@@ -4622,6 +4622,15 @@ def iapinotes_modules : JoinedOrSeparate<["-"], "iapinotes-modules">, Group<clan
 def idirafter : JoinedOrSeparate<["-"], "idirafter">, Group<clang_i_Group>,
   Visibility<[ClangOption, CC1Option]>,
   HelpText<"Add directory to AFTER include search path">;
+def iexternal : Separate<["-"], "iexternal">, Group<clang_i_Group>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Add directory to include search path with warnings suppressed">, MetaVarName<"<dir>">;
+def iexternal_system : Separate<["-"], "iexternal-system">, Group<clang_i_Group>,
+  Visibility<[CC1Option]>,
+  HelpText<"Add directory to include search path with warnings suppressed">, MetaVarName<"<dir>">;
+def iexternal_env_EQ : Joined<["-"], "iexternal-env=">, Group<clang_i_Group>,
+  Visibility<[ClangOption]>,
+  HelpText<"Add dirs in env var <var> to include search path with warnings suppressed">, MetaVarName<"<var>">;
 def iframework : JoinedOrSeparate<["-"], "iframework">, Group<clang_i_Group>,
   Visibility<[ClangOption, CC1Option]>,
   HelpText<"Add directory to SYSTEM framework search path">;
@@ -4664,6 +4673,10 @@ def isysroot : JoinedOrSeparate<["-"], "isysroot">, Group<clang_i_Group>,
 def isystem : JoinedOrSeparate<["-"], "isystem">, Group<clang_i_Group>,
   Visibility<[ClangOption, CC1Option]>,
   HelpText<"Add directory to SYSTEM include search path">, MetaVarName<"<directory>">;
+def isystem_env_EQ : Joined<["-"], "isystem-env=">, Group<clang_i_Group>,
+  MetaVarName<"<var>">,
+  Visibility<[ClangOption]>,
+  HelpText<"Add directoires in env var <var> to SYSTEM include search path">;
 def isystem_after : JoinedOrSeparate<["-"], "isystem-after">,
   Group<clang_i_Group>, Flags<[NoXarchOption]>, MetaVarName<"<directory>">,
   HelpText<"Add directory to end of the SYSTEM include search path">;
@@ -8281,6 +8294,17 @@ def fexperimental_max_bitint_width_EQ:
 // Header Search Options
 //===----------------------------------------------------------------------===//
 
+let Visibility = [ClangOption, CC1Option, CLOption] in {
+
+def fheader_search : Joined<["-"], "fheader-search=">, Group<clang_i_Group>,
+  HelpText<"Specify the method used to resolve included header files">,
+  Values<"gcc,microsoft">,
+  NormalizedValuesScope<"clang::HeaderSearchMode">,
+  NormalizedValues<["GCC", "Microsoft"]>,
+  MarshallingInfoEnum<HeaderSearchOpts<"Mode">, "GCC">;
+
+} // let Visibility = [ClangOption, CC1Option, CLOption]
+
 let Visibility = [CC1Option] in {
 
 def nostdsysteminc : Flag<["-"], "nostdsysteminc">,
@@ -8309,6 +8333,12 @@ def internal_isystem : Separate<["-"], "internal-isystem">,
   HelpText<"Add directory to the internal system include search path; these "
            "are assumed to not be user-provided and are used to model system "
            "and standard headers' paths.">;
+def internal_iexternal_system : Separate<["-"], "internal-iexternal-system">,
+  MetaVarName<"<directory>">,
+  HelpText<"Add directory to the internal system include search path with "
+           "external directory prefix semantics; these are assumed to not be "
+           "user-provided and are used to model system and standard headers' "
+           "paths.">;
 def internal_externc_isystem : Separate<["-"], "internal-externc-isystem">,
   MetaVarName<"<directory>">,
   HelpText<"Add directory to the internal system include search path with "
@@ -8552,9 +8582,12 @@ def _SLASH_diagnostics_classic : CLFlag<"diagnostics:classic">,
 def _SLASH_D : CLJoinedOrSeparate<"D", [CLOption, DXCOption]>,
   HelpText<"Define macro">, MetaVarName<"<macro[=value]>">, Alias<D>;
 def _SLASH_E : CLFlag<"E">, HelpText<"Preprocess to stdout">, Alias<E>;
-def _SLASH_external_COLON_I : CLJoinedOrSeparate<"external:I">, Alias<isystem>,
+def _SLASH_external_COLON_I : CLJoinedOrSeparate<"external:I">, Alias<iexternal>,
   HelpText<"Add directory to include search path with warnings suppressed">,
   MetaVarName<"<dir>">;
+def _SLASH_external_env : CLJoined<"external:env:">, Alias<iexternal_env_EQ>,
+  HelpText<"Add dirs in env var <var> to include search path with warnings suppressed">,
+  MetaVarName<"<var>">;
 def _SLASH_fp_contract : CLFlag<"fp:contract">, HelpText<"">, Alias<ffp_contract>, AliasArgs<["on"]>;
 def _SLASH_fp_except : CLFlag<"fp:except">, HelpText<"">, Alias<ffp_exception_behavior_EQ>, AliasArgs<["strict"]>;
 def _SLASH_fp_except_ : CLFlag<"fp:except-">, HelpText<"">, Alias<ffp_exception_behavior_EQ>, AliasArgs<["ignore"]>;
@@ -8790,9 +8823,6 @@ def _SLASH_volatile_Group : OptionGroup<"</volatile group>">,
 def _SLASH_EH : CLJoined<"EH">, HelpText<"Set exception handling model">;
 def _SLASH_EP : CLFlag<"EP">,
   HelpText<"Disable linemarker output and preprocess to stdout">;
-def _SLASH_external_env : CLJoined<"external:env:">,
-  HelpText<"Add dirs in env var <var> to include search path with warnings suppressed">,
-  MetaVarName<"<var>">;
 def _SLASH_FA : CLJoined<"FA">,
   HelpText<"Output assembly code file during compilation">;
 def _SLASH_Fa : CLJoined<"Fa">,

clang/include/clang/Driver/ToolChain.h

@@ -224,11 +224,12 @@ class ToolChain {
   /// \return The subdirectory path if it exists.
   std::optional<std::string> getTargetSubDirPath(StringRef BaseDir) const;
 
+public:
   /// \name Utilities for implementing subclasses.
   ///@{
   static void addSystemInclude(const llvm::opt::ArgList &DriverArgs,
                                llvm::opt::ArgStringList &CC1Args,
-                               const Twine &Path);
+                               const Twine &Path, bool Internal = true);
   static void addExternCSystemInclude(const llvm::opt::ArgList &DriverArgs,
                                       llvm::opt::ArgStringList &CC1Args,
                                       const Twine &Path);
@@ -238,13 +239,24 @@ class ToolChain {
                                       const Twine &Path);
   static void addSystemIncludes(const llvm::opt::ArgList &DriverArgs,
                                 llvm::opt::ArgStringList &CC1Args,
-                                ArrayRef<StringRef> Paths);
+                                ArrayRef<StringRef> Paths,
+                                bool Internal = true);
+  static bool addSystemIncludesFromEnv(const llvm::opt::ArgList &DriverArgs,
+                                       llvm::opt::ArgStringList &CC1Args,
+                                       StringRef Var, bool Internal = true);
+  static void addExternalSystemIncludes(const llvm::opt::ArgList &DriverArgs,
+                                        llvm::opt::ArgStringList &CC1Args,
+                                        ArrayRef<StringRef> Paths,
+                                        bool Internal = true);
+  static bool
+  addExternalSystemIncludesFromEnv(const llvm::opt::ArgList &DriverArgs,
+                                   llvm::opt::ArgStringList &CC1Args,
+                                   StringRef Var, bool Internal = true);
 
   static std::string concat(StringRef Path, const Twine &A, const Twine &B = "",
                             const Twine &C = "", const Twine &D = "");
   ///@}
 
-public:
   virtual ~ToolChain();
 
   // Accessors

clang/include/clang/Lex/HeaderSearch.h

@@ -276,6 +276,14 @@ class HeaderSearch {
   /// a system header.
   std::vector<std::pair<std::string, bool>> SystemHeaderPrefixes;
 
+  /// External directories are user specified directories that are to be treated
+  /// like system directories for the purposes of warning suppression. A header
+  /// file that has a path that matches one of these prefixes is promoted to a
+  /// system header regardless of which header search path was used to resolve
+  /// the \#include directive. These paths are normalized using
+  /// llvm::sys::fs::real_path().
+  llvm::StringSet<llvm::BumpPtrAllocator> ExternalDirectoryPrefixes;
+
   /// The hash used for module cache paths.
   std::string ModuleHash;
 
@@ -387,6 +395,9 @@ class HeaderSearch {
     SearchDirsUsage.push_back(false);
   }
 
+  /// Add an additional external directory prefix path.
+  bool AddExternalDirectoryPrefix(StringRef Path);
+
   /// Set the list of system header prefixes.
   void SetSystemHeaderPrefixes(ArrayRef<std::pair<std::string, bool>> P) {
     SystemHeaderPrefixes.assign(P.begin(), P.end());

clang/include/clang/Lex/HeaderSearchOptions.h

@@ -35,9 +35,19 @@ enum IncludeDirGroup {
   /// Paths for '\#include <>' added by '-I'.
   Angled,
 
+  /// Like Angled, but marks the directory as an external directory prefix.
+  /// This group is intended to match the semantics of the MSVC /external:I
+  /// option.
+  External,
+
   /// Like Angled, but marks system directories.
   System,
 
+  /// Like System, but marks the directory as an external directory prefix.
+  /// This group is intended to match the semantics of the MSVC
+  /// /external:env option.
+  ExternalSystem,
+
   /// Like System, but headers are implicitly wrapped in extern "C".
   ExternCSystem,
 
@@ -59,6 +69,11 @@ enum IncludeDirGroup {
 
 } // namespace frontend
 
+/// HeaderSearchMode - The method used to resolve included headers to files.
+/// This controls the order in which include paths are searched and how
+/// duplicate search paths are handled.
+enum class HeaderSearchMode { GCC, Microsoft };
+
 /// HeaderSearchOptions - Helper class for storing options related to the
 /// initialization of the HeaderSearch object.
 class HeaderSearchOptions {
@@ -93,6 +108,9 @@ class HeaderSearchOptions {
         : Prefix(Prefix), IsSystemHeader(IsSystemHeader) {}
   };
 
+  /// The header search mode to use.
+  HeaderSearchMode Mode = HeaderSearchMode::GCC;
+
   /// If non-empty, the directory to use as a "virtual system root" for include
   /// paths.
   std::string Sysroot;

clang/lib/Driver/Driver.cpp

@@ -1479,7 +1479,10 @@ Compilation *Driver::BuildCompilation(ArrayRef<const char *> ArgList) {
     if (VFS->setCurrentWorkingDirectory(WD->getValue()))
       Diag(diag::err_drv_unable_to_set_working_directory) << WD->getValue();
 
-  // Check for missing include directories.
+  // Check for missing include directories. Diagnostics should not be issued
+  // for directories specified with -iexternal, -iexternal-env=, or
+  // -iexternal-system since those options may be used to specify external
+  // directory prefixes that don't necessarily match an existing path.
   if (!Diags.isIgnored(diag::warn_missing_include_dirs, SourceLocation())) {
     for (auto IncludeDir : Args.getAllArgValues(options::OPT_I_Group)) {
       if (!VFS->exists(IncludeDir))