Append paths in global config to progdb in configureCompiler v2 (#10826)

yutotakano · jasagredo · web-flow · commit 74601f46289d · 2025-03-31T09:35:23.000Z
* Append paths in global config to progdb in `configureCompiler`

* Add test to check global extra-prog-path is used in `configureCompiler`

We add a new PackageTest for cabal-install, which makes sure that the
extra-prog-path in the global config is used when resolving pkg-config
instead of anything already on PATH. We do this by specifying a "bad"
pkg-config to extra-prog-path that returns exit code 1.

On Windows, Cabal logs are identical in the following two cases:
- the override didn't work, and pkg-config was not found on the system
- the override did work, and querying it failed (expectedly)
To work around this, we add a "good" pkg-config script to PATH before-
hand, so that cabal succeeds in the first case and the logs differ.

This test is a stripped down reproduction of what happens when a user on
Windows adds MSYS2 paths to extra-prog-path but not the system PATH.
This is a reasonable setup (often, adding MSYS2 to system PATH is not
good practice), and is even created automatically by GHCup.

* Refactor `configureCompiler` path addition for cleaner variable names

* Remove unnecessary build target from ExtraProgPathGlobal test command

* Fix ExtraProgPathGlobal test failing on Mac/Unix for missing +x perms

* Rename pr-10790 changelog to pr-10826 as it superceded

* Add hashtag before PR/issue numbers in changelog for 10826

---------

Co-authored-by: Javier Sagredo &lt;jasataco@gmail.com&gt;
diff --git a/cabal-install/src/Distribution/Client/ProjectPlanning.hs b/cabal-install/src/Distribution/Client/ProjectPlanning.hs
@@ -476,6 +476,7 @@ configureCompiler
         { projectConfigHcFlavor
         , projectConfigHcPath
         , projectConfigHcPkg
+        , projectConfigProgPathExtra
         }
     , projectConfigLocalPackages =
       PackageConfig
@@ -499,25 +500,28 @@ configureCompiler
       )
       $ do
         liftIO $ info verbosity "Compiler settings changed, reconfiguring..."
-        let extraPath = fromNubList packageConfigProgramPathExtra
-        progdb <- liftIO $ prependProgramSearchPath verbosity extraPath [] defaultProgramDb
-        let progdb' = userSpecifyPaths (Map.toList (getMapLast packageConfigProgramPaths)) progdb
-        result@(_, _, progdb'') <-
+        progdb <-
+          liftIO $
+            -- Add paths in the global config
+            prependProgramSearchPath verbosity (fromNubList projectConfigProgPathExtra) [] defaultProgramDb
+              -- Add paths in the local config
+              >>= prependProgramSearchPath verbosity (fromNubList packageConfigProgramPathExtra) []
+              >>= pure . userSpecifyPaths (Map.toList (getMapLast packageConfigProgramPaths))
+        result@(_, _, progdb') <-
           liftIO $
             Cabal.configCompilerEx
               hcFlavor
               hcPath
               hcPkg
-              progdb'
+              progdb
               verbosity
-
         -- Note that we added the user-supplied program locations and args
         -- for /all/ programs, not just those for the compiler prog and
         -- compiler-related utils. In principle we don't know which programs
         -- the compiler will configure (and it does vary between compilers).
         -- We do know however that the compiler will only configure the
         -- programs it cares about, and those are the ones we monitor here.
-        monitorFiles (programsMonitorFiles progdb'')
+        monitorFiles (programsMonitorFiles progdb')
 
         -- Note: There is currently a bug here: we are dropping unconfigured
         -- programs from the 'ProgramDb' when we re-use the cache created by
diff --git a/cabal-testsuite/PackageTests/ExtraProgPathGlobal/MyLibrary.hs b/cabal-testsuite/PackageTests/ExtraProgPathGlobal/MyLibrary.hs
@@ -0,0 +1 @@
+module MyLibrary () where
diff --git a/cabal-testsuite/PackageTests/ExtraProgPathGlobal/cabal.project b/cabal-testsuite/PackageTests/ExtraProgPathGlobal/cabal.project
@@ -0,0 +1 @@
+packages: *.cabal
diff --git a/cabal-testsuite/PackageTests/ExtraProgPathGlobal/my.cabal b/cabal-testsuite/PackageTests/ExtraProgPathGlobal/my.cabal
@@ -0,0 +1,20 @@
+name: CheckExtraProgPathGlobal
+version: 0.1
+license: BSD3
+license-file: LICENSE
+author: Yuto Takano
+maintainer: Yuto Takano
+synopsis: Check Extra Prog Path Global
+category: PackageTests
+build-type: Simple
+cabal-version: 2.0
+
+description:
+    Check that Cabal recognizes an override of the prog path.
+
+Library
+    pkgconfig-depends: zlib
+    default-language: Haskell2010
+    build-depends: base <5.0
+    exposed-modules:
+        MyLibrary
diff --git a/cabal-testsuite/PackageTests/ExtraProgPathGlobal/scripts-winpath/pkg-config b/cabal-testsuite/PackageTests/ExtraProgPathGlobal/scripts-winpath/pkg-config
@@ -0,0 +1,49 @@
+#!/bin/sh
+
+set -eu
+
+# ugly, but "good enough" for this test
+# This will need to be updated whenever cabal invokes pkg-config
+# in new ways
+case "$*" in
+  '--version')
+    echo 2.1.0  # whatever
+    ;;
+
+  '--variable pc_path pkg-config')
+    echo '.'
+    ;;
+
+  '--list-all')
+    printf 'zlib   zlib - zlib compression library\n'
+    # \256 = \xAE is the iso-8859-1 (latin-1) encoded version of U+00AE,
+    # i.e. the "registered sign": ®
+    # This resulted in problems, see #9608
+    printf 'vpl    Intel\256 Video Processing Library - Accelerated video decode, encode, and frame processing capabilities on Intel\256 GPUs\n'
+    # \360 = \xF0 is latin-1 for ð; this is orð, Icelandic for "word"/"words".
+    printf 'or\360   Icelandic characters\n'
+    ;;
+
+  '--modversion '*)
+    shift  # drop the --modversion
+    for arg; do
+      case "$arg" in
+        zlib) echo 1.3; ;;  # whatever
+        vpl) echo 2.10; ;;  # whatever
+        # No entry for orð here; let's not even try to match on that
+        *)
+          echo >&2 "Package $arg was not found in the pkg-config search path."
+          exit 1
+      esac
+    done
+    ;;
+
+  # Ignore some stuff we're not implementing
+  '--cflags '*) ;;
+  '--libs '*) ;;
+
+  *)
+    echo >&2 "pkg-config: unrecognised arguments $* (this is an incomplete shim)"
+    exit 1
+    ;;
+esac
diff --git a/cabal-testsuite/PackageTests/ExtraProgPathGlobal/scripts-winpath/pkg-config.exe b/cabal-testsuite/PackageTests/ExtraProgPathGlobal/scripts-winpath/pkg-config.exe
diff --git a/cabal-testsuite/PackageTests/ExtraProgPathGlobal/scripts-winpath/pkg-config.shim b/cabal-testsuite/PackageTests/ExtraProgPathGlobal/scripts-winpath/pkg-config.shim
@@ -0,0 +1,2 @@
+path = "FINDSH/sh.exe"
+args = "SCRIPTSWINPATHDIR/pkg-config"
diff --git a/cabal-testsuite/PackageTests/ExtraProgPathGlobal/scripts/pkg-config b/cabal-testsuite/PackageTests/ExtraProgPathGlobal/scripts/pkg-config
@@ -0,0 +1,3 @@
+#!/bin/sh
+
+exit 1;
diff --git a/cabal-testsuite/PackageTests/ExtraProgPathGlobal/scripts/pkg-config.exe b/cabal-testsuite/PackageTests/ExtraProgPathGlobal/scripts/pkg-config.exe
diff --git a/cabal-testsuite/PackageTests/ExtraProgPathGlobal/scripts/pkg-config.shim b/cabal-testsuite/PackageTests/ExtraProgPathGlobal/scripts/pkg-config.shim
@@ -0,0 +1,2 @@
+path = "FINDSH/sh.exe"
+args = "SCRIPTSDIR/pkg-config"
diff --git a/cabal-testsuite/PackageTests/ExtraProgPathGlobal/setup.out b/cabal-testsuite/PackageTests/ExtraProgPathGlobal/setup.out
@@ -0,0 +1,14 @@
+# cabal v2-build
+Configuration is affected by the following files:
+- cabal.project
+Warning: cannot determine version of <ROOT>/scripts/pkg-config :
+""
+Warning: cannot determine version of <ROOT>/scripts/pkg-config :
+""
+Resolving dependencies...
+Error: [Cabal-7107]
+Could not resolve dependencies:
+[__0] next goal: CheckExtraProgPathGlobal (user goal)
+[__0] rejecting: CheckExtraProgPathGlobal-0.1 (pkg-config package zlib-any is needed but no pkg-config executable was found or querying it failed)
+[__0] fail (backjumping, conflict set: CheckExtraProgPathGlobal)
+After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: CheckExtraProgPathGlobal (2)
diff --git a/cabal-testsuite/PackageTests/ExtraProgPathGlobal/setup.test.hs b/cabal-testsuite/PackageTests/ExtraProgPathGlobal/setup.test.hs
@@ -0,0 +1,60 @@
+import Test.Cabal.Prelude
+import System.Directory
+
+-- Test that extra-prog-path in the global cabal config (~/.cabal/config)
+-- overrides the path for pkg-config while resolving a dependency on zlib.
+-- Specifically, this test emulates the scenario on Windows where pkg-config
+-- doesn't exist natively and must be provided via extra-prog-path.
+main = cabalTest $ do
+    env <- getTestEnv
+    let
+      testDir = testCurrentDir env
+      tmpDir  = testTmpDir     env
+      -- Scripts is where we put a bad pkg-config
+      scripts = tmpDir </> "scripts"
+      -- Scripts-Winpath contains a mock pkg-config that returns good results.
+      -- We add this to PATH on Windows before executing the test so we can tell
+      -- apart when Cabal uses the override or not.
+      scripts_winpath = tmpDir </> "scripts-winpath"
+
+    -------------------------
+    -- Workaround for the fact that, on Windows, Cabal will only look for
+    -- .exe files to satisfy executable dependencies. So we have to create a
+    -- shim pkg-config.exe file in 'script'. This is a thin wrapper (which is
+    -- explicitly added to Git) that calls whatever is defined in the .shim
+    -- file. In our case, we rewrite the .shim file as below so that the
+    -- pkg-config script is executed using sh.
+    when isWindows $ do
+      mb_sh <- fmap takeDirectory <$> liftIO (findExecutable "sh")
+      case mb_sh of
+          Nothing -> skip "no sh"
+          Just sh -> do
+            let escape = concatMap (\c -> case c of '\\' -> "\\\\\\\\"; x -> [x])
+            void $ shell "sed" [ "-i", "-e", "s/FINDSH/" <> escape sh <> "/g", escape (scripts </> "pkg-config.shim"), escape (scripts_winpath </> "pkg-config.shim") ]
+            void $ shell "sed" [ "-i", "-e", "s/SCRIPTSDIR/" <> escape scripts <> "/g", escape (scripts </> "pkg-config.shim") ]
+            void $ shell "sed" [ "-i", "-e", "s/SCRIPTSWINPATHDIR/" <> escape scripts_winpath <> "/g", escape (scripts_winpath </> "pkg-config.shim") ]
+
+    -- End of Windows workaround
+    ------------------------------
+
+    -- On Windows, we want it to find the "scripts/pkg-config" (which will
+    -- return exit code 1), instead of what we add to the path
+    -- ("scripts-winpath/pkg-config", which will return exit code 0). This is
+    -- because Windows doesn't have a system pkg-config by default. If we didn't
+    -- add a known-to-be-good mock pkg-config in the path, we wouldn't be able
+    -- to tell apart from Cabal logs whether it wasn't able to find pkg-config
+    -- at all (test fail) or whether the override worked and it found the bad
+    -- one but couldn't query it (test success).
+    --
+    -- On other systems, we want it to find "scripts/pkg-config" (exit code 1)
+    -- instead of the system pkg-config (success).
+    let wrap_test = if isWindows then addToPath scripts_winpath else id
+
+    -- Add global config override in ~/.cabal/config (in the test environment).
+    liftIO $ appendFile (testUserCabalConfigFile env) $
+      "\nextra-prog-path: " ++ scripts
+    liftIO $ putStrLn $ testUserCabalConfigFile env
+
+    -- On correct behaviour, cabal should fail because it found our exit-code-1
+    -- pkg-config through the global extra-prog-path.
+    fails $ wrap_test $ cabal "v2-build" []
diff --git a/changelog.d/pr-10826 b/changelog.d/pr-10826
@@ -0,0 +1,11 @@
+synopsis: Prepend the global extra paths to the `ProgramDb` sooner
+packages: cabal-install
+prs: #10826
+issues: #9800
+significance: significant
+
+description: {
+
+- The initial configuration of programs was done without the `extra-prog-path` from the global cabal configuration. This meant that in some cases, some executables were not found. In particular this manifested as Windows users who didn't add the MSYS2 paths to the global PATH couldn't make use of `pkg-config`.
+
+}

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,2 @@`
	`1`	`+path = "FINDSH/sh.exe"`
	`2`	`+args = "SCRIPTSWINPATHDIR/pkg-config"`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,2 @@`
	`1`	`+path = "FINDSH/sh.exe"`
	`2`	`+args = "SCRIPTSDIR/pkg-config"`