Nix daemon disconnected unexpectedly on large closure for review

[siraben]

I'm trying to review NixOS/nixpkgs#371092, specifically building custom packages, and nixpkgs-review crashes when specifying several packages in the package set.

>>> nix-review pr --post-result --no-shell 371092 -p pkgsCross.aarch64-linux-mlibc.vim -p pkgsCross.aarch64-linux-mlibc.ed -p pkgsCross.aarch64-linux-mlibc.tmux -p pkgsCross.aarch64-linux-mlibc.git
$ git -c fetch.prune=false fetch --no-tags --force https://github.com/NixOS/nixpkgs master:refs/nixpkgs-review/0 pull/371092/merge:refs/nixpkgs-review/1
$ git worktree add /home/siraben/.cache/nixpkgs-review/pr-371092-18/nixpkgs d02d69ecf9e3cbaca78cbf3f3c45eacfffcec50d
Preparing worktree (detached HEAD d02d69ecf9e3)
Updating files: 100% (48761/48761), done.
HEAD is now at d02d69ecf9e3 Merge 410df666a21e88b335ad43b6c6ccada2dfbe5f30 into b8cf6031aa5767df06566ad84caf5e0f9d58a6d1

$ nom build --file /nix/store/xqz3ymbs5laqj5kzdi5bmzf409qx57ky-nixpkgs-review-3.3.0/lib/python3.12/site-packages/nixpkgs_review/nix/review-shell.nix --nix-path 'nixpkgs=/home/siraben/.cache/nixpkgs-review/pr-371092-18/nixpkgs nixpkgs-overlays=/tmp/tmpbhubbign' --extra-experimental-features 'nix-command no-url-literals' --no-link --keep-going --no-allow-import-from-derivation --option build-use-sandbox relaxed --argstr local-system x86_64-linux --argstr nixpkgs-path /home/siraben/.cache/nixpkgs-review/pr-371092-18/nixpkgs --argstr nixpkgs-config-path /tmp/tmpwon8huiu.nix --argstr attrs-path /home/siraben/.cache/nixpkgs-review/pr-371092-18/attrs.nix
error: Nix daemon disconnected unexpectedly (maybe it crashed?)
┏━ 1 Errors: 
┃ error: Nix daemon disconnected unexpectedly (maybe it crashed?)
┣━━━                                                            
┗━ ∑  Exited with 1 errors reported by nix at 23:49:26 after 2s

Link to currently reviewing PR:
https://github.com/NixOS/nixpkgs/pull/371092
--------- Report for 'x86_64-linux' ---------
8 packages failed to build:
pkgsCross.aarch64-linux-mlibc.ed pkgsCross.aarch64-linux-mlibc.git pkgsCross.aarch64-linux-mlibc.git.debug pkgsCross.aarch64-linux-mlibc.git.doc pkgsCross.aarch64-linux-mlibc.tmux pkgsCross.aarch64-linux-mlibc.tmux.man pkgsCross.aarch64-linux-mlibc.vim pkgsCross.aarch64-linux-mlibc.vim.xxd

Logs can be found under:
/home/siraben/.cache/nixpkgs-review/pr-371092-18/logs

The following command works, however:

>>> nixpkgs-review pr 371092 --package pkgsCross.loongarch64-linux-mlibc.pkgsStatic.bash --package pkgsCross.aarch64-linux-mlibc.bash --package pkgsCross.x86_64-linux-mlibc.pkgsStatic.bash --package pkgsCross.loongarch64-linux-mlibc.bash --package pkgsCross.riscv64-linux-mlibc.bash --package pkgsCross.x86_64-linux-mlibc.bash --package pkgsCross.riscv64-linux-mlibc.pkgsStatic.bash --package pkgsCross.aarch64-linux-mlibc.pkgsStatic.bash

Moreover following works just fine and displays with nom.

>>> nix-build -k -K --log-format internal-json -v --expr 'with import ./. {}; mkShell { packages = [ pkgsCross.aarch64-linux-mlibc.vim pkgsCross.aarch64-linux-mlibc.ed pkgsCross.aarch64-linux-mlibc.tmux pkgsCross.aarch64-linux-mlibc.git ]; }' |& nom --json

[SuperSandro2000]

What did the nix-daemon log? Is there a crash report under coredumpctl? What makes you think this is a nixpkgs-review issue and not a general nix issue?

[Mic92]

New nix version will log a stack trace these days as well if there is an exception being thrown.

[siraben]

I think this is a Nix issue, at least on Nix 2.28.3.

Main Assertion Failure:
-----------------------
nix-daemon: /nix/store/XHD38P32PVBZQMI0ZQ1WICLJY5B7YQDV-nix-util-2.28.3-dev/include/nix/util/callback.hh:45: void nix::Callback<T>::rethrow(const std::__exception_ptr::exception_ptr&) [with T = std::optional<std::__cxx11::basic_string<char> >]: Assertion `!prev' failed.

Stack Trace of Thread 4064405 (crash point):
--------------------------------------------
#0  0x00007fd49d674a9c __pthread_kill_implementation (libc.so.6 + 0x92a9c)
#1  0x00007fd49d622576 raise (libc.so.6 + 0x40576)
#2  0x00007fd49d60a935 abort (libc.so.6 + 0x28935)
#3  0x00007fd49d60a859 __assert_fail_base.cold (libc.so.6 + 0x28859)
#4  0x00007fd49d61aa56 __assert_fail (libc.so.6 + 0x38a56)
#5  0x00007fd49e093634 _ZN3nix8CallbackISt8optionalINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEE7rethrowERKNSt15__exception_ptr13exception_ptrE (libnixstore.so + 0x93634)
#6  0x00007fd49e0b4f00 _ZN3nix20HttpBinaryCacheStore7getFileERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_8CallbackISt8optionalIS6_EEE.cold (libnixstore.so + 0xb4f00)
#7  0x00007fd49e0f14f2 _ZN3nix16BinaryCacheStore21queryPathInfoUncachedERKNS_9StorePathENS_8CallbackISt10shared_ptrIKNS_13ValidPathInfoEEEE (libnixstore.so + 0xf14f2)
#8  0x00007fd49e22f03f _ZN3nix5Store13queryPathInfoERKNS_9StorePathENS_8CallbackINS_3refIKNS_13ValidPathInfoEEEEE (libnixstore.so + 0x22f03f)
#9  0x00007fd49e22bc7d _ZN3nix5Store13queryPathInfoERKNS_9StorePathE (libnixstore.so + 0x22bc7d)
#10 0x00007fd49e22c1db _ZN3nix5Store27querySubstitutablePathInfosERKSt3mapINS_9StorePathESt8optionalINS_14ContentAddressEESt4lessIS2_ESaISt4pairIKS2_S5_EEERS1_IS2_NS_21SubstitutablePathInfoES7_SaIS8_IS9_SF_EEE (libnixstore.so + 0x22c1db)
#11 0x00007fd49e1d4fbe _ZZN3nix5Store12queryMissingERKSt6vectorINS_11DerivedPathESaIS2_EERSt3setINS_9StorePathESt4lessIS8_ESaIS8_EESD_SD_RmSE_ENKUlRKS8_NS_3refINS_10DerivationEEESG_NSH_INS_8SyncBaseIZNS0_12queryMissingES6_SD_SD_SD_SE_SE_E8DrvStateSt5mutexSt11unique_lockISM_ESO_EEEEE_clESG_SJ_SG_SQ_.lto_priv.0 (libnixstore.so + 0x1d4fbe)
#12 0x00007fd49e1dc712 _ZNSt17_Function_handlerIFvvESt5_BindIFZN3nix5Store12queryMissingERKSt6vectorINS2_11DerivedPathESaIS5_EERSt3setINS2_9StorePathESt4lessISB_ESaISB_EESG_SG_RmSH_EUlRKSB_NS2_3refINS2_10DerivationEEESJ_NSK_INS2_8SyncBaseIZNS3_12queryMissingES9_SG_SG_SG_SH_SH_E8DrvStateSt5mutexSt11unique_lockISP_ESR_EEEEE_SB_SM_SB_ST_EEE9_M_invokeERKSt9_Any_data.lto_priv.0 (libnixstore.so + 0x1dc712)
#13 0x00007fd49e467dfb _ZN3nix10ThreadPool6doWorkEb (libnixutil.so + 0xe5dfb)
#14 0x00007fd49d8e86d3 execute_native_thread_routine (libstdc++.so.6 + 0xe86d3)
#15 0x00007fd49d672d02 start_thread (libc.so.6 + 0x90d02)
#16 0x00007fd49d6f23ac __clone3 (libc.so.6 + 0x1103ac)

Stack Trace of Thread 4064391 (terminate handler):
--------------------------------------------------
#0  0x00007fd49d6e537f __write (libc.so.6 + 0x10337f)
#1  0x00007fd49e474fae _ZN3nix9writeFullEiSt17basic_string_viewIcSt11char_traitsIcEEb (libnixutil.so + 0xf2fae)
#2  0x00007fd49e441006 _ZN3nix13writeToStderrESt17basic_string_viewIcSt11char_traitsIcEE (libnixutil.so + 0xbf006)
#3  0x000000000048a430 _ZN3nix12_GLOBAL__N_18logFatalERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE (/nix/store/xlg2s2hdngc2gjb6kfwac1g7b4q2pcg5-nix-2.28.3/bin/nix + 0x8a430)
#4  0x000000000048f801 _ZN3nix12_GLOBAL__N_111onTerminateEv.lto_priv.0 (/nix/store/xlg2s2hdngc2gjb6kfwac1g7b4q2pcg5-nix-2.28.3/bin/nix + 0x8f801)
#5  0x00007fd49d8bc20a _ZN10__cxxabiv111__terminateEPFvvE (libstdc++.so.6 + 0xbc20a)
#6  0x00007fd49d8bc275 _ZSt9terminatev (libstdc++.so.6 + 0xbc275)
#7  0x00007fd49e092358 _ZN3nix16BinaryCacheStore21queryPathInfoUncachedERKNS_9StorePathENS_8CallbackISt10shared_ptrIKNS_13ValidPathInfoEEEE.cold (libnixstore.so + 0x92358)
#8  0x00007fd49e22f03f _ZN3nix5Store13queryPathInfoERKNS_9StorePathENS_8CallbackINS_3refIKNS_13ValidPathInfoEEEEE (libnixstore.so + 0x22f03f)
#9  0x00007fd49e22bc7d _ZN3nix5Store13queryPathInfoERKNS_9StorePathE (libnixstore.so + 0x22bc7d)
#10 0x00007fd49e22c1db _ZN3nix5Store27querySubstitutablePathInfosERKSt3mapINS_9StorePathESt8optionalINS_14ContentAddressEESt4lessIS2_ESaISt4pairIKS2_S5_EEERS1_IS2_NS_21SubstitutablePathInfoES7_SaIS8_IS9_SF_EEE (libnixstore.so + 0x22c1db)
#11 0x00007fd49e1d4fbe _ZZN3nix5Store12queryMissingERKSt6vectorINS_11DerivedPathESaIS2_EERSt3setINS_9StorePathESt4lessIS8_ESaIS8_EESD_SD_RmSE_ENKUlRKS8_NS_3refINS_10DerivationEEESG_NSH_INS_8SyncBaseIZNS0_12queryMissingES6_SD_SD_SD_SE_SE_E8DrvStateSt5mutexSt11unique_lockISM_ESO_EEEEE_clESG_SJ_SG_SQ_.lto_priv.0 (libnixstore.so + 0x1d4fbe)
#12 0x00007fd49e1dc712 _ZNSt17_Function_handlerIFvvESt5_BindIFZN3nix5Store12queryMissingERKSt6vectorINS2_11DerivedPathESaIS5_EERSt3setINS2_9StorePathESt4lessISB_ESaISB_EESG_SG_RmSH_EUlRKSB_NS2_3refINS2_10DerivationEEESJ_NSK_INS2_8SyncBaseIZNS3_12queryMissingES9_SG_SG_SG_SH_SH_E8DrvStateSt5mutexSt11unique_lockISP_ESR_EEEEE_SB_SM_SB_ST_EEE9_M_invokeERKSt9_Any_data.lto_priv.0 (libnixstore.so + 0x1dc712)
#13 0x00007fd49e467dfb _ZN3nix10ThreadPool6doWorkEb (libnixutil.so + 0xe5dfb)
#14 0x00007fd49d8e86d3 execute_native_thread_routine (libstdc++.so.6 + 0xe86d3)
#15 0x00007fd49d672d02 start_thread (libc.so.6 + 0x90d02)
#16 0x00007fd49d6f23ac __clone3 (libc.so.6 + 0x1103ac)

[siraben]

Opened NixOS/nix#13484