Benchmarking Lotus Worker co-location with non-overlapping resources

[scaseye]

In hardware exploration mode here... Beefy servers exist, however after running lotus-bench testing, while the results are great, the over use of the machine is very low. Has anyone done, or has seen documentation for running 1-X Lotus Workers on one server? if yes, please provide all details.

Beefy Machine Specs:
CPU - 2x AMD EPYC 7742 (64 cores, 2.25 GHz)
GPU - 8x Tesla A100 (40GB of ram per gpu)
MEM - 1,536 GB
OSDrive: 15.36 TB NVMe
DataDrive: 15.36 TB NVMe

`Lotus-Bench Results

results (v28) SectorSize:(34359738368), SectorNumber:(1)
seal: addPiece: 1m29.804081029s (364.9 MiB/s)
seal: preCommit phase 1: 2h58m3.738414505s (3.067 MiB/s)
seal: preCommit phase 2: 10m0.029729199s (54.61 MiB/s)
seal: commit phase 1: 346.675309ms (92.31 GiB/s)
seal: commit phase 2: 1h13m10.819592399s (7.463 MiB/s)
seal: verify: 13.717125ms
unseal: 2h56m42.503142208s (3.091 MiB/s)

generate candidates: 519.523µs (60.15 TiB/s)
compute winning post proof (cold): 4.642506891s
compute winning post proof (hot): 4.516981626s
verify winning post proof (cold): 58.785094ms
verify winning post proof (hot): 5.536314ms

compute window post proof (cold): 12m18.403558027s
compute window post proof (hot): 12m6.609227861s
verify window post proof (cold): 7.605627978s
verify window post proof (hot): 15.723662ms`

Per the documentation: https://docs.filecoin.io/mine/lotus/seal-workers/#lotus-worker-co-location
"The only case where running multiple workers per machine may be a good idea is when there are multiple GPUs available, as lotus currently only supports a single GPU."

There for the 8 gpu on this machine just act all at the same time for one task at a time...

based on memory specs here is a high level layout, just have no idea on how to configure this from the documentation.

[Storage Miner + Node]
1 GPU Export NVIDIA_VISIBLE_DEVICES=[0]
Max Ram 256G
lotus-miner run

[PC1Worker-1] x2
No GPU allocation
Max Ram 256g
lotus-worker run --addpiece=true --precommit1=true --unseal=true --precommit2=false --commit=false

[PC2Commitworkers] x2
2 GPU - Export NVIDIA_VISIBLE_DEVICES=[1,2]
Max Ram: 256G
lotus-worker run --addpiece=false --precommit1=false --unseal=false --precommit2=true --commit=true

so this one beefy machine would run all of these.

• miner (1GPU)
• 2x PC1Workers (no GPUs)
• 2x PC2CommitWorkers (2GPUs each)

[dd45e640b42e6da7da96faee3996ef7c]

so this one beefy machine would run all of these.

miner (1GPU)
2x PC1Workers (no GPUs)
2x PC2CommitWorkers (2GPUs each)

running more then one PC1 worker on the same machine makes no sense right now without code modifications

more then one GPU/worker makes no sense either, with 40GB gpu ram thin about 3-4 workers per gpu

you need to see how to best pledge this with that little RAM. in general: way too much gpu, way too little ram, way to little nvme for sealing

[scaseye]

so referring to the protocol lab setup: https://filecoin.io/vintage/mining-hardware-config-testnet-v3.pdf

there are 6 PC1 processes running. is that done by starting 1 lotus-worker with only precommit1=true? or by configuring 6 separate services each starting a lotus-worker?

[scaseye]

so would one lotus-worker create a process for each repo defined?

lotus-worker --worker-repo /seal1/worker --worker-repo /seal2/worker --worker-repo /seal3/worker --worker-repo /seal4/worker --worker-repo /seal5/worker --worker-repo /seal6/worker run --listen 0.0.0.0:3456 --addpiece=false --precommit1=true --unseal=false --precommit2=false --commit=false

would create the 6 processes in the protocol lab diagram? or separate services like I asked prior?

[dd45e640b42e6da7da96faee3996ef7c]

no it won't. you'd have to tun 6 seperate worker processes

lotus-worker --worker-repo /seal1/worker run --listen 0.0.0.0:3456 --addpiece=false --precommit1=true --unseal=false --precommit2=false --commit=fals

lotus-worker --worker-repo /seal2/worker run --listen 0.0.0.0:3457 --addpiece=false --precommit1=true --unseal=false --precommit2=false --commit=fals

but more then 1 pc1 worker makes no sense to run. the testnet v3 design is "vintage" totally out of date