Linux system tuning to achieve maximum performance for lotus

[markwdalton]

I am looking for what others have done to tune their Linux systems.

RAID0 - striping (I saw someone used ZFS - I am guessing ZFS flavor of RAID0).
* Striping allows parallel writes/reads to multiple drives - no redundancy - so it is gone if you loose a drive.
numactl - to control which CPU socket/memory or core a process is running on
hyperthreading - has anyone tested with Hyperthreading on versus off on the BIOS?
* for HPC (efficient computing) we normally turn it off. - not a slam of other just normally not idle threads
Use of ramfs/tmpfs for parts of the job (using memory as a fast temporary storage device).
sysctl tuning - for more fine tuning of the system for thousands of parameters (about 30 would apply here).
Network device tuning

I want to write up a tuning guide and help some miners. (I am not doing the mining currently), I just do Linux, HPC (CPU, parallel and GPU).

Mark

[dd45e640b42e6da7da96faee3996ef7c]

the first problem to consider is that you have 6 major tasks a lotus worker/miner needs/can to perform

• ap
• pc1
• pc2
• c2
• wdPost
• winPost

you can run them separated, mixed in any combination. tuning a system for this will highly depend on what tasks you run. some need high single core performance - others need multicore as much as possible to perform well.

they all have unique traits and needs you need to consider if you really want to start tuning the last seconds out of it. a pc1 needs 120 mins at minimum for example - how many seconds you get if you core control it to the nvme that is connected to that cpu and not soket 2? is it worth the effort if the next update possibly changes the whole dynamic of the software?

we are doing this for a year now and see a common pattern: in the end the FIL are missing - not the hardware.

[edit]
you can "spend" (pledge) 500FIl per month with 5k USD hardware - and the ratio doesn't get "better" if you get more hardware but worse. the cost of hardware in an operation gets insignificant fast the more FIL you have available to pledge

[dd45e640b42e6da7da96faee3996ef7c]

simplest example that shows how complicated it will be to write a general tuning guide:

PC1s, single core or multicore on cores sharing a L3.

now you run them in parallel. offset by X minutes to not have all the writes at the same time. now you have y nvmes to write to. depending on x you are either better off raid0 them or run them as individual drives depending on if writes overlap or not. if they overlap depends on how fast the pc1s are in the end and how many you run in parallel. how fast the pc1s are depends on how many you run, how they are offset and what else is running on the box - how many you run in parallel depends on if you run them multicore or single core, if they run alone on the box, if its a single or dual socket config, how much ram you have etc.

now we can add reads from the nvmes to the calculation - pc2 somehow needs to read the data.

then we add the exact type of nvme to the calculation.

and you end up with a very very individual tuning guide bound to a setup/server configuration.

[dd45e640b42e6da7da96faee3996ef7c]

the longest we had a configuration perfectly balanced, nothing idle, was 3 month this year. then a release changed the timing balance between tasks and in the end stuff went idle or needed to be completely rearranged.

[dd45e640b42e6da7da96faee3996ef7c]

about zfs: people run zfs for the long term storage, HDDs. not on the nvmes (i hope...). for the simple reason that it rebuilds faster than raid5/6's.

[markwdalton]

Thank you for the feed back. I am trying to help another Filecoin miner.
My goal is just to make sure the system (hardware/linux/config) is not slowing down the process.

I will setup a mini-cluster at home and see what bottlenecks may exist. Perhaps it is just having the PC1 worker running more jobs in parallel to keep the PC2/C2 processes busy. The I/O is not saturated, the await is low.
pc1 is one one drive (at 99%), pc2 is another another, and c2 is on another - all were nvme drives

moving pc1 to tmpfs showed a 2x improvement on the p2, but no change on c2. (if I have the terms correct as far as disk writes).

I will need to sit down and go through the pieces and see what dependencies, and resources may be constraining the performance.

Again thank you for the feed back!
Mark

[dd45e640b42e6da7da96faee3996ef7c]

if you do all tasks on the same machine then raid0 the nvmes

they will actively copy from the pc1 nvme to the pc2 nvme to the c2 nvme now

[dd45e640b42e6da7da96faee3996ef7c]

I will setup a mini-cluster at home and see what bottlenecks may exist. Perhaps it is just having the PC1 worker running more jobs in parallel to keep the PC2/C2 processes busy.

pc1 is a task that loops the same operation 11 times, at the end it writes to the disks - 11 times. but in between the disks are idle. ~15-20 minutes between the loops normally

[dd45e640b42e6da7da96faee3996ef7c]

consideration: if it works get it online sealing on mainnet. every 30s you are under 10TiB on mainnet can cost you a block. just keep storage+main miner process online and adjust the sealing on the fly while being online!!!!

[donkabat]

-> hyperthreading (AMD SMT) - PC1, tested on EPYC 7343 - 16 cores + 512 GB RAM, HPC for CPU

1. SMT ON + FIL_PROOFS_USE_MULTICORE_SDR=1,FIL_PROOFS_MULTICORE_SDR_PRODUCERS=3, SWAP OFF

• 7x PC1 ~ 165 minutes/PC1

2. SMT OFF + FIL_PROOFS_USE_MULTICORE_SDR=1,FIL_PROOFS_MULTICORE_SDR_PRODUCERS=3, SWAP OFF

• 4x PC1 ~165 minutes/PC1

3. SMT OFF + FIL_PROOFS_USE_MULTICORE_SDR=1,FIL_PROOFS_MULTICORE_SDR_PRODUCERS=3, SWAP ON

• 4x PC1 ~165 minutes/PC1

4. SMT OFF + FIL_PROOFS_USE_MULTICORE_SDR=0, SWAP ON

• 8x PC1 ~ 260 minutes/PC1

[benjaminh83]

@donkabat so much good stuff in here! Thanks for helping out on the EPYC MILAN stats.

Please see my comments on your testing here:

SMT ON + FIL_PROOFS_USE_MULTICORE_SDR=1,FIL_PROOFS_MULTICORE_SDR_PRODUCERS=3, SWAP OFF
7x PC1 ~ 165 minutes/PC1

Looks fint to me keeping SMT on. Now looking at your CPU, it only has 16 physical cores. I would not let the SMT second thread running a producer interfear with my master thread, and therefore, I would probably limit the producers, so it is not slowing down any of the master threads. That said, your times are good, so it might be okay here. In theory, I would think max 8 PC1 Jobs, and running them with FIL_PROOFS_MULTICORE_SDR_PRODUCERS=1 and off cause SWAP OFF! So each job would use 2 physical cores, and max out your CPU.

SMT OFF + FIL_PROOFS_USE_MULTICORE_SDR=0, SWAP ON
8x PC1 ~ 260 minutes/PC1

This is expected. You cannot do more than 7 PC1 jobs with 512GiB mem. SWAP is slow down everything like CRAZY. So you would need to keep it at 7 and SWAP off. That said, your CPU is not the one causing the slowdown from 7 to 8 jobs. It's the SWAP that kicks in.

[dd45e640b42e6da7da96faee3996ef7c]

doing a single one gave me 155 mins on a dual 7343 system, no swap, 3 producers

just for the record

[benjaminh83]

@f8-ptrk this is fast!
But didn’t you argue that dual systems was a bit slower? I guess you still ran that single PC1 job with binding to one cpu and it’s memory?

[dd45e640b42e6da7da96faee3996ef7c]

nope. single sector PC1 unbound with the daemon running on the same box. dual 7.6TB nvmes in raid0 - the deamon obviously running on another set of nvmes

[dd45e640b42e6da7da96faee3996ef7c]

we have the following setup:

dual 7343
1TiB ram
2x3.2 nvmes raid1 for os, chain, miner / market essentials / params / parents
2x7,6 nvmes raid0 for sealing / markets deal data
A5000
jbod for disks

we do an initial sealing run with an external pipeline right now, ~14 days i think until we are broke and then i will tune this setup itself for sealing

[LexLuthr]

I am archiving this discussion as there has been no new input for more than 7 days.