Merge remote-tracking branch 'origin/master' into revert_bzip2_musl

Author: hamishmack

BUGLOG

@@ -0,0 +1,86 @@
+This file contains bugs we find while working on haskell.nix.  The format is as
+follow:
+<separator: 80 * '-'>
+YYYY-MM-DD: nix-job name
+
+<error>
+
+<discussion>
+
+--------------------------------------------------------------------------------
+2024-04-09 x86_64-linux.R2305.ghc8107.mingwW64.ghc
+
+/build/ghc62733_0/ghc_1.s:50:0: error:
+     Error: CFI instruction used without previous .cfi_startproc
+   |
+50 |         .cfi_escape 0x16, 0x07, 0x04, 0x77, 152, 65
+   | ^
+`x86_64-w64-mingw32-cc' failed in phase `Assembler'. (Exit code: 1)
+make[1]: *** [rts/ghc.mk:325: rts/dist/build/StgCRun.o] Error 1
+
+The source for this is
+> https://github.com/ghc/ghc/blob/1f02b7430b2fbab403d7ffdde9cfd006e884678e/rts/StgCRun.c#L433
+
+It appears that GCC C17 12.2.0 does _not_ emit .cfi_startproc / .cfi_endprocs
+whereas GCC C17 13.2.0 _does_. Specificall x86_64-w64-mingw32-cc. So this might
+be a cross compilation issue.
+
+The -g is hardcoded in
+https://github.com/ghc/ghc/blob/1f02b7430b2fbab403d7ffdde9cfd006e884678e/mk/config.mk.in#L361
+
+Turns out, this was disabled for anything but linux in https://github.com/ghc/ghc/commit/5b08e0c06e038448a63aa9bd7f163b23d824ba4b,
+hence we backport that patch to GHC-8.10 when targeting windows (to prevent mass rebuilds for
+other archs).
+
+--------------------------------------------------------------------------------
+2024-04-10 x86_64-linux.R2305.ghc902.mingwW64.ghc
+
+make[1]: *** [utils/hsc2hs/ghc.mk:22: utils/hsc2hs/dist-install/build/tmp/hsc2hs.exe] Error 1
+utils/runghc/dist-install/build/Main.o:fake:(.text+0x2a): relocation truncated to fit: R_X86_64_32S against `.text'
+utils/runghc/dist-install/build/Main.o:fake:(.text+0x46): relocation truncated to fit: IMAGE_REL_AMD64_ADDR32 against `.data'
+utils/runghc/dist-install/build/Main.o:fake:(.text+0x8b): relocation truncated to fit: R_X86_64_32S against symbol `stg_bh_upd_frame_info' defined in .text section in /build/ghc-9.0.2/rts/dist/build/libHSrts.a(Updates.o)
+utils/runghc/dist-install/build/Main.o:fake:(.text+0x95): relocation truncated to fit: IMAGE_REL_AMD64_ADDR32 against `.rdata'
+utils/runghc/dist-install/build/Main.o:fake:(.text+0xe3): relocation truncated to fit: R_X86_64_32S against symbol `stg_bh_upd_frame_info' defined in .text section in /build/ghc-9.0.2/rts/dist/build/libHSrts.a(Updates.o)
+utils/runghc/dist-install/build/Main.o:fake:(.text+0xed): relocation truncated to fit: IMAGE_REL_AMD64_ADDR32 against `.rdata'
+utils/runghc/dist-install/build/Main.o:fake:(.text+0x13b): relocation truncated to fit: R_X86_64_32S against symbol `stg_bh_upd_frame_info' defined in .text section in /build/ghc-9.0.2/rts/dist/build/libHSrts.a(Updates.o)
+utils/runghc/dist-install/build/Main.o:fake:(.text+0x145): relocation truncated to fit: IMAGE_REL_AMD64_ADDR32 against `.rdata'
+utils/runghc/dist-install/build/Main.o:fake:(.text+0x193): relocation truncated to fit: R_X86_64_32S against symbol `stg_bh_upd_frame_info' defined in .text section in /build/ghc-9.0.2/rts/dist/build/libHSrts.a(Updates.o)
+utils/runghc/dist-install/build/Main.o:fake:(.text+0x19d): relocation truncated to fit: IMAGE_REL_AMD64_ADDR32 against `.rdata'
+utils/runghc/dist-install/build/Main.o:fake:(.text+0x1eb): additional relocation overflows omitted from the output
+
+We notice `fake`, which is GHC failing to provide .file identifier in the source.
+We also see lots of R_X64_64_32S relocations, which are signed 32bit relocations.
+These fall with ASLR and high entropy base images from later binutils.
+
+The underlying issue is that GHC emits _absolute_ label loads (mov $... reg), instead
+of %rpi or other relative loads. This then leads to the linker emitting 32bit
+absolute relocation. With the final image being potentially loaded into high memory
+(e.g. dynamic base, and the base image being set to some high address), the linker
+starts falling over itself, because it simply can't resolve those absolute addresses
+in the 32bit slots.
+
+This was fixed in GHC upstream in https://gitlab.haskell.org/ghc/ghc/-/merge_requests/7449,
+while the patch in haskell.nix is a bit more pedestrian and just sets PIC on windows to
+always be on, and then uses the PIC pipeline.
+
+--------------------------------------------------------------------------------
+2024-06-18 x86_64-linux.unstable.ghc9101.ucrt64.tests.th-dlls-minimal.build
+
+0024:err:seh:call_stack_handlers invalid frame 00007FFFFF68EF18 (0000000000022000-0000000000220000)
+0024:err:seh:NtRaiseException Exception frame is not in stack limits => unable to dispatch exception.
+iserv-proxy: {handle: <socket: 11>}: GHCi.Message.remoteCall: end of file
+
+This is due to GHC mislinking GNU import libraries (dll.a). What happens is that
+GHC ends up creating GOT entries for function calls instead of PLT entries. The
+loader/linker in GHC for Windows has logic to lazy load .dll's as referenced. For
+this symbols get a dependency symbol attached, this could be a symbol indicating
+the DLL that needs to be loaded. While walking the dependencies to find the dll to
+load (or in some cases just the dependent symbol, not the dll), we override the
+symbol type with the one of the dependent symbol.  This however means we'll
+override the type of a symbol with the DATA type each time the symbol leads to a
+dllInstance to be loaded. Subsequently we end up creating a GOT entry instead of
+a PLT entry for the symbol, irrepsective of the original symbol being a code or
+data symbol. If code symbols end up getting GOT stubs, we see the above crash as
+the control flow jumps to the location of the stub, and instead of a PLT/jump
+island just lands in the address of the target symbol, which is in most cases
+non-sensical machine code.

build.nix

@@ -26,7 +26,11 @@ in rec {
 
   tools = pkgs.lib.optionalAttrs (ifdLevel >= 3) (
     pkgs.recurseIntoAttrs ({
-      cabal-latest = tool compiler-nix-name "cabal" { inherit evalPackages; };
+      cabal-latest = tool compiler-nix-name "cabal" ({
+        inherit evalPackages;
+      } // pkgs.lib.optionalAttrs (ghcFromTo "9.10" "9.12") {
+        cabalProjectLocal = builtins.readFile ./test/cabal.project.local;
+      });
     } // pkgs.lib.optionalAttrs (__compareVersions haskell.compiler.${compiler-nix-name}.version "9.8" < 0) {
       hlint-latest = tool compiler-nix-name "hlint" {
         inherit evalPackages;
@@ -58,10 +62,10 @@ in rec {
         inherit evalPackages;
         src = pkgs.haskell-nix.sources."hls-2.2";
       };
-    } // pkgs.lib.optionalAttrs (ghcFromTo "9.0" "9.8") {
-      "hls-26" = tool compiler-nix-name "haskell-language-server" {
+    } // pkgs.lib.optionalAttrs (ghcFromTo "9.0" "9.11") {
+      "hls-29" = tool compiler-nix-name "haskell-language-server" {
         inherit evalPackages;
-        src = pkgs.haskell-nix.sources."hls-2.6";
+        src = pkgs.haskell-nix.sources."hls-2.9";
       };
     })
   );

builder/comp-builder.nix

@@ -418,7 +418,8 @@ let
 
     nativeBuildInputs =
       [ghc buildPackages.removeReferencesTo]
-      ++ executableToolDepends;
+      ++ executableToolDepends
+      ++ (lib.optional stdenv.hostPlatform.isGhcjs buildPackages.nodejs);
 
     outputs = ["out"]
       ++ (lib.optional keepConfigFiles "configFiles")

builder/default.nix

@@ -13,7 +13,7 @@
 #
 # tl;dr: the builder must not re-introduce any reference to the build plan.
 
-{ pkgs, buildPackages, evalPackages, stdenv, lib, haskellLib, ghc, compiler-nix-name, fetchurl, nonReinstallablePkgs, hsPkgs, compiler }:
+{ pkgs, buildPackages, pkgsBuildBuild, evalPackages, stdenv, lib, haskellLib, ghc, compiler-nix-name, fetchurl, nonReinstallablePkgs, hsPkgs, compiler }:
 
 let
   # Builds a single component of a package.
@@ -85,7 +85,7 @@ let
 
   # Same as haskellPackages.shellFor in nixpkgs.
   shellFor = haskellLib.weakCallPackage pkgs ./shell-for.nix {
-    inherit hsPkgs ghcForComponent makeConfigFiles hoogleLocal haskellLib buildPackages evalPackages compiler;
+    inherit hsPkgs ghcForComponent makeConfigFiles hoogleLocal haskellLib pkgsBuildBuild evalPackages compiler;
     inherit (buildPackages) glibcLocales;
   };
 

builder/hspkg-builder.nix

@@ -31,11 +31,13 @@ let
 
   cabalFile = if package-description-override == null || bundledSrc != null then null else package-description-override;
 
-  defaultSetupSrc = if stdenv.hostPlatform.isGhcjs then ./Setup.ghcjs.hs else ./Setup.hs;
+  # New GHC JS backend run emcc itself without the need for custom Setup.hs
+  oldGhcjs = stdenv.hostPlatform.isGhcjs && builtins.compareVersions ghc.version "9.10" < 0;
+  defaultSetupSrc = if oldGhcjs then ./Setup.ghcjs.hs else ./Setup.hs;
 
   setup = if package.buildType == "Simple"
     then
-      if stdenv.targetPlatform.isGhcjs # TODO probably should be hostPlatform, but only HsColour used to build ghc will change (updating will require rebuilding all the ghcjs versions)
+      if oldGhcjs
         then
           buildPackages.haskell-nix.nix-tools-unchecked.exes.default-setup-ghcjs // { exeName = "default-setup-ghcjs"; }
         else

builder/make-config-files.nix

@@ -160,7 +160,7 @@ let
         fi
       ''}
     done
-    for p in ${lib.concatStringsSep " " (lib.remove "ghc" nonReinstallablePkgs')}; do
+    for p in ${lib.concatStringsSep " " nonReinstallablePkgs'}; do
       if [ -e $ghcDeps/envDeps/$p ]; then
         cat $ghcDeps/envDeps/$p >> $configFiles/ghc-environment
       fi

builder/shell-for.nix

@@ -1,4 +1,4 @@
-{ lib, stdenv, mkShell, glibcLocales, pkgconfig, ghcForComponent, makeConfigFiles, hsPkgs, hoogleLocal, haskellLib, buildPackages, evalPackages, compiler }:
+{ lib, stdenv, mkShell, glibcLocales, pkgconfig, ghcForComponent, makeConfigFiles, hsPkgs, hoogleLocal, haskellLib, pkgsBuildBuild, evalPackages, compiler }:
 
 { # `packages` function selects packages that will be worked on in the shell itself.
   # These packages will not be built by `shellFor`, but their
@@ -149,7 +149,7 @@ let
     # inherit (hsPkgs) hoogle;
   } // (
     lib.optionalAttrs (args ? tools && args.tools ? hoogle) {
-      hoogle = buildPackages.haskell-nix.hackage-tool (
+      hoogle = pkgsBuildBuild.haskell-nix.hackage-tool (
         haskellLib.versionOrModToMods args.tools.hoogle ++ [{
           name = "hoogle";
           compiler-nix-name = compiler.nix-name;
@@ -168,12 +168,12 @@ in
     nativeBuildInputs = [ ghcEnv.drv ]
       ++ nativeBuildInputs
       ++ mkDrvArgs.nativeBuildInputs or []
-      ++ lib.attrValues (buildPackages.haskell-nix.tools' evalPackages compiler.nix-name tools)
+      ++ lib.attrValues (pkgsBuildBuild.haskell-nix.tools' evalPackages compiler.nix-name tools)
       # If this shell is a cross compilation shell include
       # wrapper script for running cabal build with appropriate args.
       # Includes `--with-compiler` in case the `cabal.project` file has `with-compiler:` in it.
       ++ lib.optional (ghcEnv.targetPrefix != "") (
-            buildPackages.writeShellScriptBin "${ghcEnv.targetPrefix}cabal" ''
+            pkgsBuildBuild.writeShellScriptBin "${ghcEnv.targetPrefix}cabal" ''
               exec cabal \
                 --with-ghc=${ghcEnv.targetPrefix}ghc \
                 --with-compiler=${ghcEnv.targetPrefix}ghc \

changelog.md

@@ -1,6 +1,47 @@
 This file contains a summary of changes to Haskell.nix and `nix-tools`
 that will impact users.
 
+## Jun 5, 2024
+
+Haskell.nix now respects the `pre-existing` packages selected
+by the cabal planner.  The selection made by the planner
+is used to set `nonReinstallablePkgs`.
+
+Instead setting `nonReinstallablePkgs` and `reinstallableLibGhc`
+haskell.nix projects should add `constraints` to the cabal project.
+
+For instance to force the use of the `pre-exising` `text`
+package add:
+
+```
+  constraints: text installed
+```
+
+To make sure `text` is reinstalled use:
+
+```
+  constraints: text source
+```
+
+The `pre-existing` `ghc` will now be used by default as
+that is what `cabal` will choose (haskell.nix used to choose
+`reinstallableLibGhc=true` by default).
+
+To allow cabal to choose reinstalling `ghc` add:
+
+```
+  allow-boot-library-installs: True
+```
+
+To force cabal to choose reinstalling:
+
+```
+  constraints: ghc source
+  allow-boot-library-installs: True
+```
+
+It may also need `allow-newer: ghc:Cabal`
+
 ## Mar 27, 2023
 
 Haskell.nix will no longer parse the `cabal.project` file to

ci.nix

@@ -1,6 +1,6 @@
 # 'supportedSystems' restricts the set of systems that we will evaluate for. Useful when you're evaluating
 # on a machine with e.g. no way to build the Darwin IFDs you need!
-{ ifdLevel ? 3
+{ ifdLevel # This is passed in from flake.nix
 , checkMaterialization ? false
 , system ? builtins.currentSystem
 , evalSystem ? builtins.currentSystem or "x86_64-linux"
@@ -22,17 +22,22 @@
     "R2211" = inputs.nixpkgs-2211;
     "R2305" = inputs.nixpkgs-2305;
     "R2311" = inputs.nixpkgs-2311;
+    "R2405" = inputs.nixpkgs-2405;
     "unstable" = inputs.nixpkgs-unstable;
   };
 
   nixpkgsArgs = {
     # set checkMaterialization as per top-level argument
     overlays = [
       haskellNix.overlay
-      (_final: prev: {
+      (final: prev: {
         haskell-nix = prev.haskell-nix // {
           inherit checkMaterialization;
+          extraPkgconfigMappings = prev.haskell-nix.extraPkgconfigMappings or {} // {
+            "libsodium" = [ "libsodium-18" ];
+          };
         };
+        libsodium-18 = (final.callPackage (inputs.nixpkgs-2311 + "/pkgs/development/libraries/libsodium") {}).overrideAttrs (_: { dontDisableStatic = true; });
       })
     ];
     # Needed for dwarf tests
@@ -56,11 +61,7 @@
       # cabal-install and nix-tools plans.  When removing a ghc version
       # from here (so that is no longer cached) also remove ./materialized/ghcXXX.
       # Update supported-ghc-versions.md to reflect any changes made here.
-      nixpkgs.lib.optionalAttrs (nixpkgsName == "R2305") {
-        ghc810 = false;
-        ghc90 = false;
-        ghc92 = false;
-      } // nixpkgs.lib.optionalAttrs (nixpkgsName == "R2311") {
+      nixpkgs.lib.optionalAttrs (nixpkgsName == "R2405") {
         ghc94 = false;
         ghc96 = false;
         ghc98 = false;
@@ -73,7 +74,7 @@
         ghc96llvm = true;
         ghc98 = true;
         ghc98llvm = true;
-        ghc910X = true;
+        ghc910 = true;
         ghc911 = true;
       })));
   crossSystems = nixpkgsName: nixpkgs: compiler-nix-name:
@@ -88,9 +89,9 @@
        || (system == "aarch64-darwin" && !builtins.elem compiler-nix-name ["ghc884" "ghc902" "ghc928" "ghc948"])
        )) {
     inherit (lib.systems.examples) ghcjs;
-  } // lib.optionalAttrs (nixpkgsName != "unstable"
-      && (__match ".*llvm" compiler-nix-name == null)
-      && ((system == "x86_64-linux"  && !builtins.elem compiler-nix-name ["ghc884"])
+  } // lib.optionalAttrs (
+         (__match ".*llvm" compiler-nix-name == null)
+      && ((system == "x86_64-linux"  && !builtins.elem compiler-nix-name ["ghc884" "ghc91120240620"]) # Including GHC HEAD here because the patches for rts/RtsSymbols.c no longer apply and mingwW64 GHC build fails without them
        || (system == "x86_64-darwin" && builtins.elem compiler-nix-name []))) { # TODO add ghc versions when we have more darwin build capacity
     inherit (lib.systems.examples) mingwW64;
   } // lib.optionalAttrs (nixpkgsName == "unstable"
@@ -124,8 +125,6 @@ dimension "Nixpkgs version" nixpkgsVersions (nixpkgsName: pinnedNixpkgsSrc:
           ghc = pkgs.buildPackages.haskell-nix.compiler.${compiler-nix-name};
         } // pkgs.lib.optionalAttrs runTests {
           inherit (build) tests tools maintainer-scripts maintainer-script-cache;
-        } // pkgs.lib.optionalAttrs (ifdLevel >= 2) {
-          inherit (pkgs.haskell-nix.iserv-proxy-exes.${compiler-nix-name}) iserv-proxy;
         } // pkgs.lib.optionalAttrs (ifdLevel >= 3) {
           hello = (pkgs.haskell-nix.hackage-package { name = "hello"; version = "1.0.0.2"; inherit evalPackages compiler-nix-name; }).getComponent "exe:hello";
         });
@@ -149,7 +148,7 @@ dimension "Nixpkgs version" nixpkgsVersions (nixpkgsName: pinnedNixpkgsSrc:
         // pkgs.lib.optionalAttrs (ifdLevel >= 2 && crossSystemName != "ghcjs")
             pkgs.haskell-nix.iserv-proxy-exes.${compiler-nix-name}
         // pkgs.lib.optionalAttrs (ifdLevel >= 3) {
-          hello = (pkgs.haskell-nix.hackage-package { name = "hello"; version = "1.0.0.2"; inherit compiler-nix-name; }).getComponent "exe:hello";
+          hello = (pkgs.haskell-nix.hackage-package { name = "hello"; version = "1.0.0.2"; inherit evalPackages compiler-nix-name; }).getComponent "exe:hello";
         })
       ))
     )