[Clang] Match MSVC handling of duplicate header search paths in Microsoft compatibility modes.

clang/docs/ReleaseNotes.rst

@@ -389,6 +389,49 @@ Android Support
 Windows Support
 ^^^^^^^^^^^^^^^
 
+- Clang now matches MSVC behavior regarding the handling of duplicate header
+  search paths when run via the ``clang-cl`` driver (by default) or with the
+  ``-fheader-search=microsoft`` option otherwise. Historically, Clang has
+  mimicked GCC behavior in which user search paths are ordered before system
+  search paths, user search paths that duplicate a (later) system search
+  path are ignored, and search paths that duplicate an earlier search path of
+  the same user/system kind are ignored. This ordering is not compatible with
+  the ordering that MSVC uses when paths are duplicated across ``/I`` options
+  and the ``INCLUDE`` environment variable.
+
+  The order that MSVC uses and that Clang now replicates when the
+  ``-fheader-search=microsoft`` option is enabled follows.
+
+  - Paths specified by the ``/I`` and ``/external:I`` options are processed in
+    the order that they appear. Paths specified by ``/I`` that duplicate a path
+    specified by ``/external:I`` are ignored regardless of the order of the
+    options. Paths specified by ``/I`` that duplicate a path from a prior ``/I``
+    option are ignored. Paths specified by ``/external:I`` that duplicate a
+    path from a later ``/external:I`` option are ignored.
+
+  - Paths specified by ``/external:env`` are processed in the order that they
+    appear. Paths that duplicate a path from a ``/I`` or ``/external:I`` option
+    are ignored regardless of the order of the options. Paths that duplicate a
+    path from a prior ``/external:env`` option or an earlier path from the same
+    ``/external:env`` option are ignored.
+
+  - Paths specified by the ``INCLUDE`` environment variable are processed in
+    the order they appear. Paths that duplicate a path from a ``/I``,
+    ``/external:I``, or ``/external:env`` option are ignored. Paths that
+    duplicate an earlier path in the ``INCLUDE`` environment variable are
+    ignored.
+
+  - Paths specified by the ``EXTERNAL_INCLUDE`` environment variable are
+    processed in the order they appear. Paths that duplicate a path from a
+    ``/I``, ``/external:I``, or ``/external:env`` option are ignored. Paths that
+    duplicate a path from the ``INCLUDE`` environment variable are ignored.
+    Paths that duplicate an earlier path in the ``EXTERNAL_INCLUDE``
+    environment variable are ignored.
+
+  The ``-fheader-search=gcc`` option can be used to opt in to GCC duplicate
+  header search path handling (which remains the default behavior for the GCC
+  compatible drivers).
+
 LoongArch Support
 ^^^^^^^^^^^^^^^^^
 

clang/docs/UsersManual.rst

@@ -5418,12 +5418,26 @@ follows:
 
 2. Consult the environment.
 
-    TODO: This is not yet implemented.
+    - `/external:env:[VARIABLE]`
 
-    This will consult the environment variables:
+      This command line option specifies a user identified environment variable
+      which is treated as a path delimited (`;`) list of system header paths.
 
-    - `WindowsSdkDir`
-    - `UCRTVersion`
+    - `INCLUDE`
+
+      This environment variable is treated as a path delimited (`;`) list of
+      system header paths.
+
+    - `EXTERNAL_INCLUDE`
+
+      This environment variable is treated as a path delimited (`;`) list of
+      system header paths.
+
+    The following environment variables will be consulted and used to form paths
+    to validate and load content from as appropriate:
+
+      - `WindowsSdkDir`
+      - `UCRTVersion`
 
 3. Fallback to the registry.
 
@@ -5435,6 +5449,8 @@ The Visual C++ Toolset has a slightly more elaborate mechanism for detection.
 
 1. Consult the command line.
 
+    Anything the user specifies is always given precedence.
+
     - `/winsysroot:`
 
     The `/winsysroot:` is used as an equivalent to `-sysroot` on Unix
@@ -5452,21 +5468,30 @@ The Visual C++ Toolset has a slightly more elaborate mechanism for detection.
 
 2. Consult the environment.
 
-    - `/external:[VARIABLE]`
+    - `/external:env:[VARIABLE]`
+
+      This command line option specifies a user identified environment variable
+      which is treated as a path delimited (`;`) list of external header search
+      paths. Additionally, any header search path provided by other means that
+      has a prefix that matches one of these paths is treated as an external
+      header search path.
+
+    - `INCLUDE`
 
-      This specifies a user identified environment variable which is treated as
-      a path delimiter (`;`) separated list of paths to map into `-imsvc`
-      arguments which are treated as `-isystem`.
+      This environment variable is treated as a path delimited (`;`) list of
+      header search paths.
 
-    - `INCLUDE` and `EXTERNAL_INCLUDE`
+    - `EXTERNAL_INCLUDE`
 
-      The path delimiter (`;`) separated list of paths will be mapped to
-      `-imsvc` arguments which are treated as `-isystem`.
+      This environment variable is treated as a path delimited (`;`) list of
+      external header search paths. Additionally, any header search path
+      provided by other means that has a prefix that matches one of these paths
+      is treated as an external header search path.
 
     - `LIB` (indirectly)
 
       The linker `link.exe` or `lld-link.exe` will honour the environment
-      variable `LIB` which is a path delimiter (`;`) set of paths to consult for
+      variable `LIB` which is a path delimited (`;`) set of paths to consult for
       the import libraries to use when linking the final target.
 
     The following environment variables will be consulted and used to form paths

clang/include/clang/Driver/Options.td

@@ -4811,6 +4811,19 @@ def iapinotes_modules : JoinedOrSeparate<["-"], "iapinotes-modules">, Group<clan
 def idirafter : JoinedOrSeparate<["-"], "idirafter">, Group<clang_i_Group>,
   Visibility<[ClangOption, CC1Option]>,
   HelpText<"Add directory to AFTER include search path">;
+def iexternal : Separate<["-"], "iexternal">, Group<clang_i_Group>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Add directory to include search path with warnings suppressed">, MetaVarName<"<dir>">;
+def iexternal_system : Separate<["-"], "iexternal-system">, Group<clang_i_Group>,
+  Visibility<[CC1Option]>,
+  HelpText<"Add directory to include search path with warnings suppressed">, MetaVarName<"<dir>">;
+def iexternal_env_EQ : Joined<["-"], "iexternal-env=">, Group<clang_i_Group>,
+  Visibility<[ClangOption]>,
+  HelpText<"Add dirs in env var <var> to include search path with warnings suppressed">, MetaVarName<"<var>">;
+def iexternal_anglebrackets : Flag<["-"], "iexternal-anglebrackets">, Group<clang_i_Group>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Treat all #include paths enclosed in angle brackets as including a system header">,
+  MarshallingInfoFlag<HeaderSearchOpts<"AngleBracketsImpliesSystemHeader">>;
 def iframework : JoinedOrSeparate<["-"], "iframework">, Group<clang_i_Group>,
   Visibility<[ClangOption, CC1Option]>,
   HelpText<"Add directory to SYSTEM framework search path">;
@@ -4853,6 +4866,10 @@ def isysroot : JoinedOrSeparate<["-"], "isysroot">, Group<clang_i_Group>,
 def isystem : JoinedOrSeparate<["-"], "isystem">, Group<clang_i_Group>,
   Visibility<[ClangOption, CC1Option]>,
   HelpText<"Add directory to SYSTEM include search path">, MetaVarName<"<directory>">;
+def isystem_env_EQ : Joined<["-"], "isystem-env=">, Group<clang_i_Group>,
+  MetaVarName<"<var>">,
+  Visibility<[ClangOption]>,
+  HelpText<"Add directoires in env var <var> to SYSTEM include search path">;
 def isystem_after : JoinedOrSeparate<["-"], "isystem-after">,
   Group<clang_i_Group>, Flags<[NoXarchOption]>, MetaVarName<"<directory>">,
   HelpText<"Add directory to end of the SYSTEM include search path">;
@@ -8620,6 +8637,17 @@ def fexperimental_max_bitint_width_EQ:
 // Header Search Options
 //===----------------------------------------------------------------------===//
 
+let Visibility = [ClangOption, CC1Option, CLOption] in {
+
+def fheader_search : Joined<["-"], "fheader-search=">, Group<clang_i_Group>,
+  HelpText<"Specify the method used to resolve included header files">,
+  Values<"gcc,microsoft">,
+  NormalizedValuesScope<"clang::HeaderSearchMode">,
+  NormalizedValues<["GCC", "Microsoft"]>,
+  MarshallingInfoEnum<HeaderSearchOpts<"Mode">, "GCC">;
+
+} // let Visibility = [ClangOption, CC1Option, CLOption]
+
 let Visibility = [CC1Option] in {
 
 def nostdsysteminc : Flag<["-"], "nostdsysteminc">,
@@ -8653,6 +8681,12 @@ def internal_isystem : Separate<["-"], "internal-isystem">,
   HelpText<"Add directory to the internal system include search path; these "
            "are assumed to not be user-provided and are used to model system "
            "and standard headers' paths.">;
+def internal_iexternal_system : Separate<["-"], "internal-iexternal-system">,
+  MetaVarName<"<directory>">,
+  HelpText<"Add directory to the internal system include search path with "
+           "external directory prefix semantics; these are assumed to not be "
+           "user-provided and are used to model system and standard headers' "
+           "paths.">;
 def internal_externc_isystem : Separate<["-"], "internal-externc-isystem">,
   MetaVarName<"<directory>">,
   HelpText<"Add directory to the internal system include search path with "
@@ -8906,9 +8940,14 @@ def _SLASH_diagnostics_classic : CLFlag<"diagnostics:classic">,
 def _SLASH_D : CLJoinedOrSeparate<"D", [CLOption, DXCOption]>,
   HelpText<"Define macro">, MetaVarName<"<macro[=value]>">, Alias<D>;
 def _SLASH_E : CLFlag<"E">, HelpText<"Preprocess to stdout">, Alias<E>;
-def _SLASH_external_COLON_I : CLJoinedOrSeparate<"external:I">, Alias<isystem>,
+def _SLASH_external_COLON_I : CLJoinedOrSeparate<"external:I">, Alias<iexternal>,
   HelpText<"Add directory to include search path with warnings suppressed">,
   MetaVarName<"<dir>">;
+def _SLASH_external_env : CLJoined<"external:env:">, Alias<iexternal_env_EQ>,
+  HelpText<"Add dirs in env var <var> to include search path with warnings suppressed">,
+  MetaVarName<"<var>">;
+def _SLASH_external_anglebrackets : CLFlag<"external:anglebrackets">, Alias<iexternal_anglebrackets>,
+  HelpText<"Treat all #include paths enclosed in angle brackets as including a system header">;
 def _SLASH_fp_contract : CLFlag<"fp:contract">, HelpText<"">, Alias<ffp_contract>, AliasArgs<["on"]>;
 def _SLASH_fp_except : CLFlag<"fp:except">, HelpText<"">, Alias<ffp_exception_behavior_EQ>, AliasArgs<["strict"]>;
 def _SLASH_fp_except_ : CLFlag<"fp:except-">, HelpText<"">, Alias<ffp_exception_behavior_EQ>, AliasArgs<["ignore"]>;
@@ -9148,9 +9187,6 @@ def _SLASH_d2epilogunwindrequirev2 : CLFlag<"d2epilogunwindrequirev2">,
 def _SLASH_EH : CLJoined<"EH">, HelpText<"Set exception handling model">;
 def _SLASH_EP : CLFlag<"EP">,
   HelpText<"Disable linemarker output and preprocess to stdout">;
-def _SLASH_external_env : CLJoined<"external:env:">,
-  HelpText<"Add dirs in env var <var> to include search path with warnings suppressed">,
-  MetaVarName<"<var>">;
 def _SLASH_FA : CLJoined<"FA">,
   HelpText<"Output assembly code file during compilation">;
 def _SLASH_Fa : CLJoined<"Fa">,

clang/include/clang/Driver/ToolChain.h

@@ -219,6 +219,7 @@ class ToolChain {
   /// \return The subdirectory path if it exists.
   std::optional<std::string> getTargetSubDirPath(StringRef BaseDir) const;
 
+public:
   /// \name Utilities for implementing subclasses.
   ///@{
   static void addSystemFrameworkInclude(const llvm::opt::ArgList &DriverArgs,
@@ -236,16 +237,28 @@ class ToolChain {
                                          ArrayRef<StringRef> Paths);
   static void addSystemIncludes(const llvm::opt::ArgList &DriverArgs,
                                 llvm::opt::ArgStringList &CC1Args,
-                                ArrayRef<StringRef> Paths);
+                                ArrayRef<StringRef> Paths,
+                                bool Internal = true);
+  static bool addSystemIncludesFromEnv(const llvm::opt::ArgList &DriverArgs,
+                                       llvm::opt::ArgStringList &CC1Args,
+                                       StringRef Var, bool Internal = true);
+  static void addExternalSystemIncludes(const llvm::opt::ArgList &DriverArgs,
+                                        llvm::opt::ArgStringList &CC1Args,
+                                        ArrayRef<StringRef> Paths,
+                                        bool Internal = true);
+  static bool
+  addExternalSystemIncludesFromEnv(const llvm::opt::ArgList &DriverArgs,
+                                   llvm::opt::ArgStringList &CC1Args,
+                                   StringRef Var, bool Internal = true);
 
   static std::string concat(StringRef Path, const Twine &A, const Twine &B = "",
                             const Twine &C = "", const Twine &D = "");
   ///@}
 
-public:
   static void addSystemInclude(const llvm::opt::ArgList &DriverArgs,
                                llvm::opt::ArgStringList &CC1Args,
-                               const Twine &Path);
+                               const Twine &Path, bool Internal = true);
+
   virtual ~ToolChain();
 
   // Accessors

clang/include/clang/Lex/HeaderSearch.h

@@ -276,6 +276,15 @@ class HeaderSearch {
   /// a system header.
   std::vector<std::pair<std::string, bool>> SystemHeaderPrefixes;
 
+  /// External directories are user specified directories that are to be treated
+  /// like system directories for the purposes of warning suppression. A header
+  /// file that has a path that matches one of these prefixes is promoted to a
+  /// system header regardless of which header search path was used to resolve
+  /// the \#include directive. llvm::sys::path::remove_dots() is used to
+  /// normalize these paths by removing "." and ".." path components and
+  /// duplicate path separators. Trailing path separators are retained.
+  llvm::StringSet<llvm::BumpPtrAllocator> ExternalDirectoryPrefixes;
+
   /// The hash used for module cache paths.
   std::string ModuleHash;
 
@@ -401,6 +410,9 @@ class HeaderSearch {
     SearchDirsUsage.push_back(false);
   }
 
+  /// Add an additional external directory prefix path.
+  bool AddExternalDirectoryPrefix(StringRef Path);
+
   /// Set the list of system header prefixes.
   void SetSystemHeaderPrefixes(ArrayRef<std::pair<std::string, bool>> P) {
     SystemHeaderPrefixes.assign(P.begin(), P.end());

clang/include/clang/Lex/HeaderSearchOptions.h

@@ -35,9 +35,19 @@ enum IncludeDirGroup {
   /// Paths for '\#include <>' added by '-I'.
   Angled,
 
+  /// Like Angled, but marks the directory as an external directory prefix.
+  /// This group is intended to match the semantics of the MSVC /external:I
+  /// option.
+  External,
+
   /// Like Angled, but marks system directories.
   System,
 
+  /// Like System, but marks the directory as an external directory prefix.
+  /// This group is intended to match the semantics of the MSVC
+  /// /external:env option.
+  ExternalSystem,
+
   /// Like System, but headers are implicitly wrapped in extern "C".
   ExternCSystem,
 
@@ -59,6 +69,11 @@ enum IncludeDirGroup {
 
 } // namespace frontend
 
+/// HeaderSearchMode - The method used to resolve included headers to files.
+/// This controls the order in which include paths are searched and how
+/// duplicate search paths are handled.
+enum class HeaderSearchMode { GCC, Microsoft };
+
 /// HeaderSearchOptions - Helper class for storing options related to the
 /// initialization of the HeaderSearch object.
 class HeaderSearchOptions {
@@ -93,6 +108,9 @@ class HeaderSearchOptions {
         : Prefix(Prefix), IsSystemHeader(IsSystemHeader) {}
   };
 
+  /// The header search mode to use.
+  HeaderSearchMode Mode = HeaderSearchMode::GCC;
+
   /// If non-empty, the directory to use as a "virtual system root" for include
   /// paths.
   std::string Sysroot;
@@ -207,6 +225,11 @@ class HeaderSearchOptions {
   LLVM_PREFERRED_TYPE(bool)
   unsigned Verbose : 1;
 
+  /// Whether header files specified in angle brackets should be treated as
+  /// system headers.
+  LLVM_PREFERRED_TYPE(bool)
+  unsigned AngleBracketsImpliesSystemHeader : 1;
+
   /// If true, skip verifying input files used by modules if the
   /// module was already verified during this build session (see
   /// \c BuildSessionTimestamp).
@@ -289,6 +312,7 @@ class HeaderSearchOptions {
         ModuleFileHomeIsCwd(false), EnablePrebuiltImplicitModules(false),
         UseBuiltinIncludes(true), UseStandardSystemIncludes(true),
         UseStandardCXXIncludes(true), UseLibcxx(false), Verbose(false),
+        AngleBracketsImpliesSystemHeader(false),
         ModulesValidateOncePerBuildSession(false),
         ModulesValidateSystemHeaders(false),
         ModulesForceValidateUserHeaders(true),

clang/lib/Driver/Driver.cpp

@@ -1534,7 +1534,10 @@ Compilation *Driver::BuildCompilation(ArrayRef<const char *> ArgList) {
     if (VFS->setCurrentWorkingDirectory(WD->getValue()))
       Diag(diag::err_drv_unable_to_set_working_directory) << WD->getValue();
 
-  // Check for missing include directories.
+  // Check for missing include directories. Diagnostics should not be issued
+  // for directories specified with -iexternal, -iexternal-env=, or
+  // -iexternal-system since those options may be used to specify external
+  // directory prefixes that don't necessarily match an existing path.
   if (!Diags.isIgnored(diag::warn_missing_include_dirs, SourceLocation())) {
     for (auto IncludeDir : Args.getAllArgValues(options::OPT_I_Group)) {
       if (!VFS->exists(IncludeDir))