SPX v1.15.2-rc1 - Testing & Call Outs #8484

rjan90 · 2022-04-13T18:26:31Z

rjan90
Apr 13, 2022
Maintainer

Hello SPX Fellows

v1.15.2-rc1 has landed so we are kicking off some new testing. This one is quite light on tasks as most of the big features in this release have been tested earlier.

🚀 This post if you are participating in this testing round.
If you aren't able to join for reasons, please reach out to @jennijuju or me on Slack!

So this assignment is focused on the smaller nice to have features and improvements, as well as some awesome new scheduling logic for you to observe.

Before upgrading:

Since PoSt workers can read challenges from other workers, setting the file descriptor limit high enough has become a requirement for all workers. Make sure that you have raised the ulimit on your workers before upgrading.

The resource manager has also been enabled again, so keep an extra eye on the connection of your Daemon <> Miner <> Markets.

New features and Improvements

Apart from the big features in this version, like the PoSt workers and Snap Deal enhancements, it also includes some nice to have features like:

lotus client list-asks --protocols to see which protocol nodes are using.
Ping your friends with lotus-miner net ping

Scheduler Improvements

Today, when the scheduler is assigning tasks to your workers, it creates a list “task -> worker”. It is sorted by worker utilization at the start of each schedule cycle, but is not re-sorted when assigning a lot of tasks to the production line of workers in one cycle.

The improvement is that Lotus is tracking the change in the worker utilization while processing the acceptableWindows array - this should result in much better scheduling decisions with only a small increase in compute cost when it's scheduling.

You will see some new miner logs when the scheduler is selecting a worker, please give us some feedback on how the new scheduler performs, under your reply card!

For more improvements & bug fixes see here.

Reply card:

If you have successfully sealed some new sectors on this tag.
If you have successfully made some storage deals on this tag.
If you have successfully made some snap deals on this tag.
If you passed your first WindowPoSt on this tag.
Test lotus client list-asks --protocols | grep 'mk/1.2.0' and check that you can see BOOST nodes when using it.
The resource manager has not caused any connection problems.

Reminders and Risk Mitigations

Questions are welcomed in this discussion, the team will be monitoring this discussion and check-in regularly.

Issues can be reported in https://github.com/filecoin-project/lotus/issues. Issues should be submitted with:

Daemon logs
go routine dumps lotus pprof goroutines
the operation/step at which the problem occurred
The stack trace or logs as relevant. As always, for large log files, you may upload it in this Google Disk folder and link it in the issue!

❗ If you think the v1.15.2-rc1 branch is not working well, especially the resource manager, keep all the logs for describing the issue and disable the resource manager with LOTUS_RCMGR=0, if the problem still persists revert back to the stable v1.15.1 release.

Meest-NAM · 2022-04-14T01:28:00Z

Meest-NAM
Apr 14, 2022

#242.u-i261.d), (RF: i264.u-i315.d), (RF: i318.u-i353.u), (RF: i431.u-i469.d), (RF: i478.u-i529.d), (RF: i532.u-i541.d), (RF: i544.u-i579.u), (RF: i657.u-i695.d), (RF: i704.u-i713.d), (RF: i721.u-i762.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   DI alloc to       %r14
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          298
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   considering       vr6:  (VR: v6J, sz=561, tc=5, sc=0.009, [(RF: i6.d-i188.d), (RF: i191.u-i232.u), (RF: i242.u-i261.d), (RF: i264.u-i315.d), (RF: i318.u-i353.u), (RF: i431.u-i469.d), (RF: i478.u-i529.d), (RF: i532.u-i541.d), (RF: i544.u-i579.u), (RF: i657.u-i695.d), (RF: i704.u-i713.d), (RF: i721.u-i762.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO candidates     pri { %rcx }
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   spill
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > editlist entry (other): (ELI: at i4 for vr60 add D->S, slot=S0)
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > editlist entry (other): (ELI: at i113 for vr61 add R->U, slot=S0)
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --     new RangeFrag    (RF: i6.d-i6.s)
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --     new VirtRange    vr303  :=  (VR: v6J, sz=1, tc=4294967295, sc=INFINITY, [(RF: i6.d-i6.s)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --     new ELI other    (ELI: at i6 for vr303 add D->S, slot=S0)
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --     new RangeFrag    (RF: i232.r-i232.u)
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --     new VirtRange    vr304  :=  (VR: v6J, sz=1, tc=4294967295, sc=INFINITY, [(RF: i232.r-i232.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --     new ELI other    (ELI: at i232 for vr304 add R->U, slot=S0)
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          603
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   considering       vr129:  (VR: v194J, sz=8, tc=2, sc=0.250, [(RF: i335.d-i336.d), (RF: i358.d-i359.d), (RF: i361.d-i364.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main >
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- MAIN ALLOCATION LOOP (DI means 'direct', CO means 'coalesced'):
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > alloc_main:   main allocation loop: begin
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   DI alloc to       %r10
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          10
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   DI alloc to       %r9
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          24
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO alloc to       %r9
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          9
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   considering       vr39:  (VR: v65J, sz=2, tc=1, sc=0.500, [(RF: i97.d-i98.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO candidates     pri { %r10 }
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO alloc to       %r10
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          8
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   considering       vr40:  (VR: v68J, sz=2, tc=2, sc=1.000, [(RF: i83.d-i84.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO candidates     pri { %rcx }
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO alloc to       %rcx
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          7
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   considering       vr41:  (VR: v69J, sz=2, tc=2, sc=1.000, [(RF: i85.d-i86.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO candidates     pri { %r8 }
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO alloc to       %r8
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          6
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > ELI other: (ELI: at i725 for vr340 add R->U, slot=S4)
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO alloc to       %r9
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          42
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   considering       vr92:  (VR: v161J, sz=2, tc=2, sc=1.000, [(RF: i79.d-i80.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO candidates     pri { %rcx }
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   considering       vr6:  (VR: v6J, sz=18, tc=2, sc=0.111, [(RF: i6.d-i23.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   DI alloc to       %r11
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   considering       vr19:  (VR: v33J, sz=8, tc=4, sc=0.500, [(RF: i42.d-i46.d), (RF: i76.u-i78.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO candidates     pri { %rax %rdx }
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          453
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   considering       vr1007:  (VR: v367J, sz=1, tc=4294967295, sc=INFINITY, [(RF: i1296.r-i1296.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   DI alloc to       %rsi
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          191
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   considering       vr669:  (VR: v214J, sz=1, tc=4294967295, sc=INFINITY, [(RF: i318.r-i318.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO candidates     pri { %rcx }
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO alloc to       %rcx
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO candidates     pri { %rdx }
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO alloc to       %rdx
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          602
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   considering       vr177:  (VR: v270J, sz=8, tc=20, sc=2.500, [(RF: i491.d-i498.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO candidates     pri { %rcx } sec { %rdx }
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO alloc to       %rcx
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          601
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   considering       vr191:  (VR: v299J, sz=8, tc=4, sc=0.500, [(RF: i581.d-i588.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO candidates     pri { %rax }
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO evict          vr188:  (VR: v291J, -> r18J sz=26, tc=5, sc=0.192, [(RF: i577.d-i583.u), (RF: i747.u-i752.u), (RF: i759.u-i771.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO alloc to       %rax
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > -- still TODO          601
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   considering       vr188:  (VR: v291J, sz=26, tc=5, sc=0.192, [(RF: i577.d-i583.u), (RF: i747.u-i752.u), (RF: i759.u-i771.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --   CO candidates     pri { %rdx }
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --     new VirtRange    vr1600  :=  (VR: v287J, sz=1, tc=4294967295, sc=INFINITY, [(RF: i534.r-i534.u)])
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --     new ELI other    (ELI: at i534 for vr1600 add R->U, slot=S44)
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --     new RangeFrag    (RF: i660.r-i660.u)
%Y-%m-%dT%H:%M:%S%.3f DEBUG regalloc::bt_main > --     new RangeFrag    (RF: i353.r-i353.u)

export LOTUS_USE_FVM_EXPERIMENTAL=1

If you apply FVM as "1", does not become "Chain sync"
What's wrong with you?

Daemon: 1.15.2-rc1+mainnet+git.dcf6f6414+api1.5.0
Local: lotus version 1.15.2-rc1+mainnet+git.dcf6f6414

1 reply

jennijuju Apr 14, 2022
Maintainer

im not seeing this issue on my node - could you please eleaborate more on how you are running it?

tmyuu · 2022-04-14T06:13:01Z

tmyuu
Apr 14, 2022

If you have successfully sealed some new sectors on this tag.
If you have successfully made some storage deals on this tag.
If you have successfully made some snap deals on this tag.
If you passed your first WindowPoSt on this tag.
Test lotus client list-asks --protocols | grep 'mk/1.2.0' and check that you can see BOOST nodes when using it.
The resource manager has not caused any connection problems.

$ lotus client list-asks --protocols | grep 'mk/1.2.0'

f01608291: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:251.058481ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f0104671: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:5.014216ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]

$ lotus-miner net ping /ip4/212.6.53.86/tcp/24001/p2p/12D3KooWEfUQyxb6hkGbDHPf8cWNeSL3Z6bx9mtj21NC7BpghjBs

PING 12D3KooWEfUQyxb6hkGbDHPf8cWNeSL3Z6bx9mtj21NC7BpghjBs
2022-04-15T11:39:42.191+0900	ERROR	cli	cli/net.go:179	Ping failed: error=RPC error (-32601): method 'Filecoin.NetPing' not found
2022-04-15T11:39:42.192+0900	ERROR	cli	cli/net.go:179	Ping failed: error=RPC error (-32601): method 'Filecoin.NetPing' not found
2022-04-15T11:39:42.192+0900	ERROR	cli	cli/net.go:179	Ping failed: error=RPC error (-32601): method 'Filecoin.NetPing' not found
2022-04-15T11:39:42.192+0900	ERROR	cli	cli/net.go:179	Ping failed: error=RPC error (-32601): method 'Filecoin.NetPing' not found
2022-04-15T11:39:42.193+0900	ERROR	cli	cli/net.go:179	Ping failed: error=RPC error (-32601): method 'Filecoin.NetPing' not found
2022-04-15T11:39:42.193+0900	ERROR	cli	cli/net.go:179	Ping failed: error=RPC error (-32601): method 'Filecoin.NetPing' not found
2022-04-15T11:39:42.193+0900	ERROR	cli	cli/net.go:179	Ping failed: error=RPC error (-32601): method 'Filecoin.NetPing' not found
2022-04-15T11:39:42.193+0900	ERROR	cli	cli/net.go:179	Ping failed: error=RPC error (-32601): method 'Filecoin.NetPing' not found
2022-04-15T11:39:42.194+0900	ERROR	cli	cli/net.go:179	Ping failed: error=RPC error (-32601): method 'Filecoin.NetPing' not found
2022-04-15T11:39:42.194+0900	ERROR	cli	cli/net.go:179	Ping failed: error=RPC error (-32601): method 'Filecoin.NetPing' not found

$ lotus-miner proving compute window-post 0

Took 7m26.32847446s
[{"Deadline":0,"Partitions":[{"Index":0,"Skipped":[0]}],"Proofs":[{"PoStProof":8,"ProofBytes":"pbuS0mq127Q3HhfD4Pf+VdSM89mUZuUP+cJLpeU2vqRgWri1UzpiiM8gTWeQTVKgoBqDm/0GGNnxs7A568lb2DrNyH2WX6fZIXvHpwNYB9XmEQMD80g4m8Oq7Ilgs4jPDii+BuTQa3IGt6Av68A//+L+ipNmMAe19024TI9uFavs8KB9/9iiJbEsS06JhF8Orw8eYnPJYGcVK5ksrS77eHEDoZjl+GjziMnSe99JszaGfQnau5zC29jTIlAT8m2N"}],"ChainCommitEpoch":0,"ChainCommitRand":null}]

$ lotus-worker with windowpost

[flexi_logger][ERRCODE::Time] flexi_logger has to work with UTC rather than with local time, caused by IndeterminateOffset
    See https://docs.rs/flexi_logger/latest/flexi_logger/error_info/index.html#time
%Y-%m-%dT%H:%M:%S%.3f INFO filcrypto::proofs::api > generate_single_window_post_with_vanilla: start
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::api::window_post > generate_single_window_post_with_vanilla:start
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::caches > trying parameters memory cache for: Window_POST[34359738368]
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::caches > no params in memory cache for Window_POST[34359738368]
%Y-%m-%dT%H:%M:%S%.3f INFO storage_proofs_core::parameter_cache > parameter set identifier for cache: FallbackPoSt::PublicParams{sector_size: 34359738368, challenge_count: 10, sector_count: 2349}
%Y-%m-%dT%H:%M:%S%.3f INFO storage_proofs_core::parameter_cache > ensuring that all ancestor directories for: "/usr/local/filecoin/.cache/parameter/param_v28/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.params" exist
%Y-%m-%dT%H:%M:%S%.3f INFO storage_proofs_core::parameter_cache > checking cache_path: "/usr/local/filecoin/.cache/parameter/param_v28/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.params" for parameters
%Y-%m-%dT%H:%M:%S%.3f INFO storage_proofs_core::parameter_cache > Verify production parameters is false
%Y-%m-%dT%H:%M:%S%.3f INFO storage_proofs_core::parameter_cache > read parameters from cache "/usr/local/filecoin/.cache/parameter/param_v28/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.params" 
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::api::post_util > single_partition_vanilla_proofs:start
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::api::post_util > single_partition_vanilla_proofs:finish
%Y-%m-%dT%H:%M:%S%.3f INFO storage_proofs_core::compound_proof > snark_proof:start
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::groth16::prover > Bellperson 0.18.2 is being used!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::groth16::prover > synthesis time: 23.366165505s
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::groth16::prover > starting proof timer
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::locks > GPU is available for FFT!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::program > Using kernel on CUDA.
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::fft > FFT: 1 working device(s) selected. 
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::fft > FFT: Device 0: Tesla T4
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::domain > GPU FFT kernel instantiated!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::locks > GPU is available for Multiexp!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::program > Using kernel on CUDA.
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0)
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::multiexp > Multiexp: Device 0: Tesla T4 (Chunk-size: 4405294)
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::groth16::prover > prover time: 308.982421487s
%Y-%m-%dT%H:%M:%S%.3f INFO storage_proofs_core::compound_proof > snark_proof:finish
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::api::window_post > generate_single_window_post_with_vanilla:finish
%Y-%m-%dT%H:%M:%S%.3f INFO filcrypto::proofs::api > generate_single_window_post_with_vanilla: finish
%Y-%m-%dT%H:%M:%S%.3f INFO filcrypto::proofs::api > generate_single_window_post_with_vanilla: start
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::api::window_post > generate_single_window_post_with_vanilla:start
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::caches > trying parameters memory cache for: Window_POST[34359738368]
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::caches > found params in memory cache for Window_POST[34359738368]
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::api::post_util > single_partition_vanilla_proofs:start
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::api::post_util > single_partition_vanilla_proofs:finish
%Y-%m-%dT%H:%M:%S%.3f INFO storage_proofs_core::compound_proof > snark_proof:start
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::groth16::prover > Bellperson 0.18.2 is being used!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::groth16::prover > synthesis time: 25.565876632s
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::groth16::prover > starting proof timer
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::locks > GPU is available for FFT!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::program > Using kernel on CUDA.
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::fft > FFT: 1 working device(s) selected. 
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::fft > FFT: Device 0: Tesla T4
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::domain > GPU FFT kernel instantiated!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::locks > GPU is available for Multiexp!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::program > Using kernel on CUDA.
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0)
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::multiexp > Multiexp: Device 0: Tesla T4 (Chunk-size: 4405294)
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::groth16::prover > prover time: 316.906488471s
%Y-%m-%dT%H:%M:%S%.3f INFO storage_proofs_core::compound_proof > snark_proof:finish
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::api::window_post > generate_single_window_post_with_vanilla:finish
%Y-%m-%dT%H:%M:%S%.3f INFO filcrypto::proofs::api > generate_single_window_post_with_vanilla: finish
%Y-%m-%dT%H:%M:%S%.3f INFO filcrypto::proofs::api > generate_single_window_post_with_vanilla: start
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::api::window_post > generate_single_window_post_with_vanilla:start
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::caches > trying parameters memory cache for: Window_POST[34359738368]
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::caches > found params in memory cache for Window_POST[34359738368]
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::api::post_util > single_partition_vanilla_proofs:start
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::api::post_util > single_partition_vanilla_proofs:finish
%Y-%m-%dT%H:%M:%S%.3f INFO storage_proofs_core::compound_proof > snark_proof:start
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::groth16::prover > Bellperson 0.18.2 is being used!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::groth16::prover > synthesis time: 25.66076126s
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::groth16::prover > starting proof timer
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::locks > GPU is available for FFT!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::program > Using kernel on CUDA.
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::fft > FFT: 1 working device(s) selected. 
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::fft > FFT: Device 0: Tesla T4
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::domain > GPU FFT kernel instantiated!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::locks > GPU is available for Multiexp!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::program > Using kernel on CUDA.
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0)
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::gpu::multiexp > Multiexp: Device 0: Tesla T4 (Chunk-size: 4405294)
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
%Y-%m-%dT%H:%M:%S%.3f INFO bellperson::groth16::prover > prover time: 315.947873761s
%Y-%m-%dT%H:%M:%S%.3f INFO storage_proofs_core::compound_proof > snark_proof:finish
%Y-%m-%dT%H:%M:%S%.3f INFO filecoin_proofs::api::window_post > generate_single_window_post_with_vanilla:finish
%Y-%m-%dT%H:%M:%S%.3f INFO filcrypto::proofs::api > generate_single_window_post_with_vanilla: finish

1 reply

rjan90 Apr 14, 2022
Maintainer Author

Nice! Two Boost-nodes there 🥳

cryptowhizzard · 2022-04-15T19:06:17Z

cryptowhizzard
Apr 15, 2022

Upgraded f01608291.

Reply card:

If you have successfully sealed some new sectors on this tag.
If you have successfully made some storage deals on this tag.
If you have successfully made some snap deals on this tag.
If you passed your first WindowPoSt on this tag.
Test lotus client list-asks --protocols | grep 'mk/1.2.0' and check that you can see BOOST nodes when using it.
The resource manager has not caused any connection problems.

Updating with more info in next days.

0 replies

benjaminh83 · 2022-04-16T19:11:52Z

benjaminh83
Apr 16, 2022

If you have successfully sealed some new sectors on this tag.
If you have successfully made some storage deals on this tag.
If you have successfully made some snap deals on this tag.
If you passed your first WindowPoSt on this tag.
Test lotus client list-asks --protocols | grep 'mk/1.2.0' and check that you can see BOOST nodes when using it.
The resource manager has not caused any connection problems.

1 reply

benjaminh83 Apr 17, 2022

$ lotus client list-asks --protocols | grep 'mk/1.2.0'
f0127896: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:60.247177ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f01608291: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:20.857988ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f0104671: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:410.474683ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]

Meest-NAM · 2022-04-18T02:16:26Z

Meest-NAM
Apr 18, 2022

If you have successfully sealed some new sectors on this tag.
If you have successfully made some storage deals on this tag.
If you have successfully made some snap deals on this tag.
If you passed your first WindowPoSt on this tag.
Test lotus client list-asks --protocols | grep 'mk/1.2.0' and check that you can see BOOST nodes when using it.
The resource manager has not caused any connection problems.

1 reply

Meest-NAM Apr 18, 2022

lotus client list-asks --protocols | grep 'mk/1.2.0'

f0127896: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:288.876081ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f01608291: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:334.587871ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f0104671: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:47.113437ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]

TippyFlitsUK · 2022-04-18T15:59:56Z

TippyFlitsUK
Apr 18, 2022
Maintainer

If you have successfully sealed some new sectors on this tag.
If you have successfully made some storage deals on this tag.
If you have successfully made some snap deals on this tag.
If you passed your first WindowPoSt on this tag.
Test lotus client list-asks --protocols | grep 'mk/1.2.0' and check that you can see BOOST nodes when using it.
The resource manager has not caused any connection problems.

1 reply

TippyFlitsUK Apr 18, 2022
Maintainer

lotus client list-asks --protocols | grep 'mk/1.2.0'

f0127896: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:96.594384ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f01608291: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:16.798272ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f08403: min:256 MiB max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:5.139614ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f0104671: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:249.504004ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]

neondragon · 2022-04-19T19:24:59Z

neondragon
Apr 19, 2022

Reply card:

If you have successfully sealed some new sectors on this tag.
If you have successfully made some storage deals on this tag.
If you have successfully made some snap deals on this tag.
If you passed your first WindowPoSt on this tag.
Test lotus client list-asks --protocols | grep 'mk/1.2.0' and check that you can see BOOST nodes when using it.
The resource manager has not caused any connection problems.

1 reply

neondragon Apr 19, 2022

# lotus client list-asks --protocols | grep 'mk/1.2.0'
f01608291: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:16.606306ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f08403: min:256 MiB max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:21.651212ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f0694396: min:128 MiB max:64 GiB price:0.00000002 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:10.355872ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f02620: min:512 MiB max:32 GiB price:0.0000094 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:49.670983ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]

SBudo · 2022-04-20T09:43:40Z

SBudo
Apr 20, 2022

If you have successfully sealed some new sectors on this tag.
If you have successfully made some storage deals on this tag.
If you have successfully made some snap deals on this tag.
If you passed your first WindowPoSt on this tag.
Test lotus client list-asks --protocols | grep 'mk/1.2.0' and check that you can see BOOST nodes when using it.
The resource manager has not caused any connection problems.

1 reply

SBudo Apr 20, 2022

lotus client list-asks --protocols | grep 'mk/1.2.0'
f08403: min:256 MiB max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:297.894764ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f0104671: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:153.515561ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f01608291: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:272.761807ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]

jamerduhgamer · 2022-04-24T02:00:45Z

jamerduhgamer
Apr 24, 2022

If you have successfully sealed some new sectors on this tag.
If you have successfully made some storage deals on this tag.
If you have successfully made some snap deals on this tag.
If you passed your first WindowPoSt on this tag.
Test lotus client list-asks --protocols | grep 'mk/1.2.0' and check that you can see BOOST nodes when using it.
The resource manager has not caused any connection problems.

3 replies

jamerduhgamer Apr 28, 2022

lotus client list-asks --protocols | grep 'mk/1.2.0

f0104671: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:120.856886ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]

jamerduhgamer May 2, 2022

Miner has been online without any connection problems for the last 9 days now.

jamerduhgamer May 2, 2022

Ran into AddPiece and PC1 issues with new sectors and was not able to test any storage or snap deals on this tag but will do so on the newer release here. #8579

Angelo-gh3990 · 2022-04-24T04:08:07Z

Angelo-gh3990
Apr 24, 2022

If you have successfully sealed some new sectors on this tag.
If you have successfully made some storage deals on this tag.
If you have successfully made some snap deals on this tag.
If you passed your first WindowPoSt on this tag.
Test lotus client list-asks --protocols | grep 'mk/1.2.0' and check that you can see BOOST nodes when using it.
The resource manager has not caused any connection problems.

2 replies

Angelo-gh3990 Apr 24, 2022

Has been running stable for 10 days now, no issues.

Angelo-gh3990 Apr 24, 2022


f02576: min:4 GiB max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:1.079208191s protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f0127896: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:44.136552ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f01608291: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:55.929774ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f08403: min:256 MiB max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:53.916997ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]
f0104671: min:256 B max:32 GiB price:0 FIL/GiB/Epoch verifiedPrice:0 FIL/GiB/Epoch ping:702.27397ms protos:[/fil/storage/mk/1.0.1,/fil/storage/mk/1.1.0,/fil/storage/mk/1.2.0]

SPX v1.15.2-rc1 - Testing & Call Outs #8484

Uh oh!

Uh oh!

rjan90 Apr 13, 2022 Maintainer

Hello SPX Fellows

Before upgrading:

New features and Improvements

Scheduler Improvements

Reply card:

Reminders and Risk Mitigations

Replies: 10 comments · 12 replies

Uh oh!

Uh oh!

jennijuju Apr 14, 2022 Maintainer

Uh oh!

Uh oh!

Uh oh!

rjan90 Apr 14, 2022 Maintainer Author

Uh oh!

Uh oh!

Reply card:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TippyFlitsUK Apr 18, 2022 Maintainer

Uh oh!

Uh oh!

TippyFlitsUK Apr 18, 2022 Maintainer

Uh oh!

Uh oh!

Reply card:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rjan90
Apr 13, 2022
Maintainer

Replies: 10 comments 12 replies

jennijuju Apr 14, 2022
Maintainer

rjan90 Apr 14, 2022
Maintainer Author

TippyFlitsUK
Apr 18, 2022
Maintainer

TippyFlitsUK Apr 18, 2022
Maintainer