1.15.2-rc1, snap deal error (ReplicaUpdateFailed), fetch error, connection refused...

[Shekelme]

Before submitting an issue, I would like to ask the public about my problem, maybe there is a simple and obvious solution.

The error occurred when trying to package a verified deal into an existing sector. In miner log I see:

2022-05-02T18:47:29.056+0300    WARN    sectors storage-sealing/fsm.go:763      sector 23562 got error event sealing.SectorUpdateReplicaFailed: replica update failed: Schedule: storage call error 0: %!w(failed to acquire sector {1222595 23562} from remote (tried [{99f98850-422b-4cc1-94c7-b113425a104c [http://127.0.0.1:2345/remote/sealed/s-t01222595-23562] [http://127.0.0.1:2345/remote] 10 false true true}]): 1 error occurred:
        * fetch error http://127.0.0.1:2345/remote/sealed/s-t01222595-23562 (storage 99f98850-422b-4cc1-94c7-b113425a104c) -> /media/asus/worker1/sealed/fetching/s-t01222595-23562: do request: Get "http://127.0.0.1:2345/remote/sealed/s-t01222595-23562": dial tcp 127.0.0.1:2345: connect: connection refused

 [Hostname: lenovo8])

Here storage 99f98850-422b-4cc1-94c7-b113425a104c is a long-term storage, NFS share on miner server.

192.168.11.8:/mnt/filecoin-storage-18 on /media/filecoin-storage-18 type nfs4 (rw,noatime,nodiratime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.11.92,local_lock=none,addr=192.168.11.8)

For many months before, traditional sealing took place without any problems of this kind at all. It's a 64GiB-miner by the way.

On that worker lenovo8 I see in it's log:

2022-05-02T18:46:27.924+0300    INFO    main    lotus-worker/main.go:425        Opening local storage; connecting to master
2022-05-02T18:46:27.934+0300    INFO    main    lotus-worker/main.go:483        Setting up control endpoint at 192.168.11.18:3456
2022-05-02T18:46:27.935+0300    INFO    main    lotus-worker/main.go:572        Making sure no local tasks are running
2022-05-02T18:46:29.573+0300    INFO    main    lotus-worker/main.go:595        Worker registered successfully, waiting for tasks
2022-05-02T18:46:29.590+0300    INFO    stores  stores/remote.go:254    Fetch http://127.0.0.1:2345/remote/sealed/s-t01222595-23562 -> /media/asus/worker1/sealed/fetching/s-t01222595-23562
2022-05-02T18:47:29.043+0300    INFO    stores  stores/remote.go:254    Fetch http://127.0.0.1:2345/remote/sealed/s-t01222595-23562 -> /media/asus/worker1/sealed/fetching/s-t01222595-23562

In the mentioned folder there is a file corresponding to this sector, but it is of zero size.

admfc@lenovo8:/media/asus/worker1/sealed/fetching$ ls -l
total 0
-rw-rw-r-- 1 admfc admfc 0 May  2 19:07 s-t01222595-23562

Worker info:

Worker version:  1.6.0
CLI version: lotus-worker version 1.15.2-rc1+mainnet+git.dcf6f6414

Session: bcee8947-6aaa-477a-9441-28ec591ed3bd
Enabled: true
Hostname: lenovo8
CPUs: 32; GPUs: [GeForce RTX 3090]
RAM: 5.761 GiB/503.6 GiB; Swap: 0 B/931.5 GiB
Task types: FIN GET FRU C1 C2 PC2 PR1 PR2 RU GSK

e4dbeb94-b6dc-440f-9811-b207bdb7c9cb:
        Weight: 10; Use: Seal
        Local: /media/asus/worker1

Where to dig in search of a solution, please tell me...

[Shekelme]

And nothing has changed in 1.15.2-rc2+mainnet+git.94d9dee0d.

[Shekelme]

Nothing changes when storing the sector locally on the miner. I tried this on purpose to test the assumption that the problem is caused by NFS.

[LexLuthr]

@Shekelme Did you get an answer from the community on this? If not, then perhaps we need to ask on the slack with link to this discussion.

[Shekelme]

Slack thread. I haven't heard anything clear yet.

[LexLuthr]

@Shekelme Any updates or clarity from slack?

[Shekelme]

Absolute silence for 27 days. Apparently, everything works for everyone.

[LexLuthr]

@Shekelme I was reopened this discussion as requested on slack. I won't be closing it out unless explicitly told by you.
This discussion purpose is clearly stated as "You want community opinion on this"

[scaseye]

sorry, i missed this not many of us snapping at the moment. highly recommend going to 1.15.3 asap.

[LexLuthr]

I am archiving this discussion as there has been no new input from the author for more than 7 days.

[scaseye]

both miner and worker are on lenovo8? can you share the relevant config for the miner and worker?

[Shekelme]

No, lenovo8 is worker server. Dedicated PC2/C2 worker.

[scaseye]

why is the worker using the 127.0.0.1 ? can you post the service and env setup for the worker?

[Shekelme]

Good question. I think miner tells to this worker what to fetch. I startthe workers with this script:

export TMPDIR=/media/asus/tmp1
export GOLOG_FILE=/home/admfc/worker1.log
export CUDA_VISIBLE_DEVICES=0
nohup lotus-worker --worker-repo /media/asus/worker1 run --listen=0.0.0.0:3456 --precommit1=false --precommit2=true --commit=true --addpiece=false --unseal=false > ~/worker1_rust.log 2>&1 &

The envars I use are as follows:

export RUST_LOG=debug
export RUST_BACKTRACE=full
export GOLOG_OUTPUT=file
export MINER_API_INFO=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.tkwV7ooNbqrHdozQW2up_t8bbPtr1tmleP0obUWiVSM:/ip4/192.168.11.92/tcp/2345/http
export LOTUS_WORKER_PATH=/media/asus/worker1
export FIL_PROOFS_MAXIMIZE_CACHING=1
export FIL_PROOFS_PARAMETER_CACHE=/home/admfc/filecoin-proofs
export FIL_PROOFS_PARENT_CACHE=/home/admfc/filecoin-proofs/parent
export FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1
export FIL_PROOFS_USE_GPU_TREE_BUILDER=1
export BELLMAN_CUSTOM_GPU="NVIDIA GeForce RTX 3090:10496"
export RUSTFLAGS="-C target-cpu=native -g"
export FFI_BUILD_FROM_SOURCE=1

[scaseye]

storage json file for this worker?

[scaseye]

also what is in the miner log file?

[scaseye]

here is what my RU worker has for that same line. I think it is a config issue of the ip of the miner. the worker is trying to fetch from 127.0.0.1 . 40.0.0.199 is my miner ip.

{"level":"info","ts":"2022-06-09T23:43:30.066-0700","logger":"stores","caller":"stores/remote.go:254","msg":"Fetch http://40.0.0.199:2345/remote/sealed/s-t01611097-3277 -> /seal2/mainnet32GiB/.lotusworker/worker-pc2-2/sealed/fetching/s-t0161109
7-3277"}

[scaseye]

i dont use 127.0.0.1 anywhere in my stack...

[scaseye]

if not, then there is some issue with the miner accessing that sector. miner logs may shed more light on that.

[Shekelme]

Yes, everything seems to work if you get rid of addresses like 0.0.0.0 and 127.0.0.1 in the miner config and worker startup scripts.