zkevm Prover exits with code 139 and 255

[RohitAudit]

I am running zkevm testnet on holesky and I am getting following error while running prover node:

zkevm-prover  | 20240324_163300_159455 fde636c 03abc40 file2json() loading JSON file src/main_sm/fork_6/scripts/rom.json
zkevm-prover  | 20240324_163300_181507 fde636c 03abc40 ROM size: 5049 lines
zkevm-prover  | 20240324_163300_270934 fde636c 03abc40 <-- ROM_LOAD done: 0.111491 s
zkevm-prover  | 20240324_163300_278999 fde636c 03abc40 --> ROM_LOAD starting...
zkevm-prover  | 20240324_163300_279015 fde636c 03abc40 file2json() loading JSON file src/main_sm/fork_7/scripts/rom.json
zkevm-prover  | 20240324_163300_344598 fde636c 03abc40 ROM size: 13179 lines
zkevm-prover  | 20240324_163300_615755 fde636c 03abc40 <-- ROM_LOAD done: 0.336754 s
zkevm-prover  | 20240324_163300_642603 fde636c 03abc40 --> ROM_LOAD starting...
zkevm-prover  | 20240324_163300_642625 fde636c 03abc40 file2json() loading JSON file src/main_sm/fork_8/scripts/rom.json
zkevm-prover  | 20240324_163300_702681 fde636c 03abc40 ROM size: 13260 lines
zkevm-prover  | 20240324_163300_959654 fde636c 03abc40 <-- ROM_LOAD done: 0.317040 s
zkevm-prover  | 20240324_163300_985816 fde636c 03abc40 file2json() loading JSON file /mnt/prover/config/scripts/storage_sm_rom.json
zkevm-prover  | 20240324_163300_990682 fde636c 03abc40 --> BINARY_EXECUTOR starting...
zkevm-prover  | 20240324_163300_990701 fde636c 03abc40 --> BINARY_BUILD_FACTORS starting...
zkevm-prover  | 20240324_163301_246365 fde636c 03abc40 <-- BINARY_BUILD_FACTORS done: 0.255645 s
zkevm-prover  | 20240324_163301_246452 fde636c 03abc40 --> BINARY_BUILD_RESET starting...
zkevm-prover  | 20240324_163301_349666 fde636c 03abc40 <-- BINARY_BUILD_RESET done: 0.103200 s
zkevm-prover  | 20240324_163301_349691 fde636c 03abc40 <-- BINARY_EXECUTOR done: 0.359009 s
zkevm-prover  | 20240324_163301_349740 fde636c 03abc40 --> KECCAK_F_SM_EXECUTOR_LOAD starting...
zkevm-prover  | 20240324_163301_349746 fde636c 03abc40 file2json() loading JSON file /mnt/prover/config/scripts/keccak_script.json
zkevm-prover  | 20240324_163302_716279 fde636c 03abc40 <-- KECCAK_F_SM_EXECUTOR_LOAD done: 1.366526 s
zkevm-prover  | 20240324_163302_981913 fde636c 03abc40 --> SHA256_F_SM_EXECUTOR_LOAD starting...
zkevm-prover  | 20240324_163302_981940 fde636c 03abc40 file2json() loading JSON file /mnt/prover/config/scripts/sha256_script.json
zkevm-prover  | 20240324_163303_299566 fde636c 03abc40 <-- SHA256_F_SM_EXECUTOR_LOAD done: 0.317644 s
zkevm-prover exited with code 139

I am using version v5.0.8 and with prover files https://storage.googleapis.com/zkevm/zkproverc/v5.0.0-rc.4-fork.8.tgz

[RohitAudit]

I resolved the issue by increasing the ram for the instance. Although I ran into different issue:

zkevm-prover  | 20240326_112737_430097 c8e9f2b 6e48c40 file2json() loading JSON file /mnt/prover/config/zkevm/zkevm.starkinfo.json
zkevm-prover  | 20240326_112739_178837 c8e9f2b 6e48c40 <-- STARK_INFO_LOAD done: 1.752880 s
zkevm-prover  | 20240326_112739_698593 c8e9f2b 6e48c40 zkError: Prover::genBatchProof() failed calling malloc() of size 350039834624
zkevm-prover  | 20240326_112739_744653 c8e9f2b 6e48c40 CALL STACK
zkevm-prover  | 20240326_112739_744666 c8e9f2b 6e48c40 0: call=zkProver(_Z14printCallStackv+0x45) [0x58675f815615]
zkevm-prover  | 20240326_112739_744668 c8e9f2b 6e48c40 1: call=zkProver(_Z11exitProcessv+0x16) [0x58675f838156]
zkevm-prover  | 20240326_112739_744669 c8e9f2b 6e48c40 2: call=zkProver(_ZN6ProverC2ER10GoldilocksR18PoseidonGoldilocksRK6Config+0x1211) [0x58675f7ab361]
zkevm-prover  | 20240326_112739_744670 c8e9f2b 6e48c40 3: call=zkProver(main+0x10d1) [0x586745ad92d1]
zkevm-prover  | 20240326_112739_744671 c8e9f2b 6e48c40 4: call=/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x72c997a43d90]
zkevm-prover  | 20240326_112739_744672 c8e9f2b 6e48c40 5: call=/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x72c997a43e40]
zkevm-prover  | 20240326_112739_744673 c8e9f2b 6e48c40 6: call=zkProver(_start+0x25) [0x586745b1ca95]
zkevm-prover  | 20240326_112739_744853 c8e9f2b 6e48c40 MEMORY INFO 
zkevm-prover  |     MemTotal: 63252.070312 MB
zkevm-prover  |     MemFree: 419.773438 MB
zkevm-prover  |     MemAvailable: 23197.621094 MB
zkevm-prover  |     Buffers: 2.035156 MB
zkevm-prover  |     Cached: 23365.410156 MB
zkevm-prover  |     SwapCached: 0.000000 MB
zkevm-prover  |     SwapTotal: 0.000000 MB
zkevm-prover  |     SwapFree: 0.000000 MB
zkevm-prover  |     VM: 40101.535156 MB
zkevm-prover  |     RSS: 9.490315 MB
zkevm-prover  | 20240326_112739_744870 c8e9f2b 6e48c40 PROCESS INFO
zkevm-prover  |     Pid: 1
zkevm-prover  |     User time: 21.510000 s
zkevm-prover  |     Kernel time: 27.580000 s
zkevm-prover  |     Total time: 49.090000 s
zkevm-prover  |     Num threads: 11
zkevm-prover  |     Virtual mem: 40101 MB
zkevm-prover  | 20240326_112744_768038 c8e9f2b 6e48c40 --> DATABASE_PROGRAM_CACHE_DESTRUCTOR starting...
zkevm-prover  | 20240326_112744_768065 c8e9f2b 6e48c40 <-- DATABASE_PROGRAM_CACHE_DESTRUCTOR done: 0.000030 s
zkevm-prover  | 20240326_112744_768068 c8e9f2b 6e48c40 --> DATABASE_CACHE_DESTRUCTOR starting...
zkevm-prover  | 20240326_112744_768070 c8e9f2b 6e48c40 <-- DATABASE_CACHE_DESTRUCTOR done: 0.000002 s
zkevm-prover  | 20240326_112744_768071 c8e9f2b 6e48c40 --> DATABASE_MT_CACHE_DESTRUCTOR starting...
zkevm-prover  | 20240326_112744_768073 c8e9f2b 6e48c40 <-- DATABASE_MT_CACHE_DESTRUCTOR done: 0.000001 s
zkevm-prover  | 20240326_112744_768074 c8e9f2b 6e48c40 --> DATABASE_CACHE_DESTRUCTOR starting...
zkevm-prover  | 20240326_112744_768076 c8e9f2b 6e48c40 <-- DATABASE_CACHE_DESTRUCTOR done: 0.000001 s
zkevm-prover exited with code 255

[mohsinriaz17]

Hi, i have the exact same issue
i am on a 48vcpu machine with 192G ram

@RohitAudit what are the specs of your machine?

[DNK90]

@mohsinriaz17 try to increase it to more than 512 GB and 64 CPU. It the same as my previous issue
#840