Reduce memory use - control multiple instances of chia_full_node

[caeci11ius]

Is there some way to reduce the number of instances of chia_full_node that are running? I seem to have four by default and there doesn't seem to be a way to limit this. Could someone advise if it's possible to reduce, or if having four instances of chia_full_node is strictly necessary for farming to work? Ideally, I'd like to reduce this to two, or even one instance.

I'm running it on an older system with only 4GB of RAM, and it basically eats the entire memory capacity of the system. Since I've disabled the wallet and enabled zswap, I'm at least now avoiding very heavily smashing swap, but it would be good to use less RAM still. This would also help people running on 4GB Raspberry Pi 4s (the swap performance there is very poor, so it's probably really damaging overall system performance).

Top output (prior to disabling wallet - though ideally I'd like to get that back so I can get stats from plotnft show):

%CPU %MEM COMMAND
5.3  25.6 chia_full_node
0.0  24.4 chia_full_node
0.0  20.3 chia_wallet
0.0  20.2 chia_full_node
0.0  19.6 chia_full_node
0.0   1.0 chia_farmer
0.0   0.9 chia_harvester

[InvaderDolan]

I am also issuing the same problem. I have from 17 instances of chia_full_node, each 1.1GiB, so chia eats up to 20GiB of RAM which is crazy! Still couldn't find a fix.
@hoffmang9 @Yostra @mariano54 we need your help. I have a lot of system freezes which affect my farming income. Thank you!

[mariano54]

What version of the software are you running and what OS?
In main we have added a config parameter reserved cores, the higher you make this number, the less sub processes will be created:
https://github.com/Chia-Network/chia-blockchain/blob/main/chia/util/initial-config.yaml#L346

Still it is very strange that each process is using so much, that is not expected behavior.

[InvaderDolan]

Thank you for your response.
I am using the latest version on Ubuntu 20.04. I just updated it recently (Successfully installed chia-blockchain-1.2.11), I am updating the chia-blockchain every week (if there are new updates), hoping that RAM problem will be fixed.
I have 10700 (16 threads), so what would be optimal value to set in reserved_cores:? Will be reserved_cores: 15 enough to keep up with full node?

[arvidn]

Since the processes may share pages, looking at PSS might be more relevant than RSS

[InvaderDolan]

Here is my USS, PSS and RSS usages got from smem:

PID     User      Command                    Swap        USS      PSS      RSS
10895 invader  chia_full_node                     0    11404    77653  1106232 
10896 invader  chia_full_node                     0    11468    77677  1106184 
10894 invader  chia_full_node                     0    12108    78108  1106184 
10903 invader  chia_full_node                     0    13152    78931  1106184 
10902 invader  chia_full_node                     0    13452    79165  1106180 
10906 invader  chia_full_node                     0    14768    80291  1106176 
10897 invader  chia_full_node                     0    14828    80384  1106180 
10900 invader  chia_full_node                     0    15108    80615  1106260 
10907 invader  chia_full_node                     0    15600    81056  1106256 
10898 invader  chia_full_node                     0    16408    81792  1106240 
10901 invader  chia_full_node                     0    16692    82036  1106244 
10899 invader  chia_full_node                     0    16756    82096  1106188 
10886 invader  chia_full_node                     0     9696    82302  1102272 
10904 invader  chia_full_node                     0    17060    82377  1106172 
10885 invader  chia_full_node                     0     9968    82549  1102272 
10905 invader  chia_full_node                     0    24624    89456  1105980

Still uses a lot as I understand.

[caeci11ius]

Ubuntu 21.10
Chia (offical Docker container) v1.2.11

Interestingly I have four instances despite having 2 CPU cores and no hyperthreading in this machine...

This install has been running for several months, so the config.yaml is quite old (i'd guess installed around the point 1.2.8 was current?). It's nice to see the new version is so much better documented (and it has a lot of new useful options!). I'm guessing there's no easy way to regenerate the config while retaining the configured options (particularly for payments...I know it's easy enough to move them over but also think it's good to avoid mistakes when that kind of thing is concerned).

  PID User     Command                         Swap      USS      PSS      RSS 
19236 chia     chia_full_node                620152    49744   204360   581216 
20319 chia     chia_full_node                604684    65724   220111   596736 
 5009 chia     chia_full_node                379876   343772   456082   763932 
 4202 chia     chia_full_node                248624   577972   688631   993132

[caeci11ius]

Have just tried adding reserved_cores: 1 to config.yaml and am still spawning four instances. Is this the minimum, or is the setting not working?

full_node:
  database_path: db/blockchain_v1_CHALLENGE.sqlite
  dns_servers:
  - dns-introducer.chia.net
  enable_profiler: false
  enable_upnp: false
  exempt_peer_networks: []
  farmer_peer:
    host: 127.0.0.1
    port: 8447
  introducer_peer:
    host: introducer.chia.net
    port: 8444
  log_sqlite_cmds: false
  logging: *id001
  max_inbound_farmer: 10
  max_inbound_timelord: 5
  max_inbound_wallet: 20
  network_overrides: *id002
  peer_connect_interval: 30
  peer_connect_timeout: 30
  peer_db_path: db/peer_table_node.sqlite
  port: 8444
  recent_peer_threshold: 6000
  reserved_cores: 1
  rpc_port: 8555
  sanitize_weight_proof_only: false
  selected_network: mainnet
  send_uncompact_interval: 0
  short_sync_blocks_behind_threshold: 20
  simulator_database_path: sim_db/simulator_blockchain_v1_CHALLENGE.sqlite
  simulator_peer_db_path: sim_db/peer_table_node.sqlite
  ssl:
    private_crt: config/ssl/full_node/private_full_node.crt
    private_key: config/ssl/full_node/private_full_node.key
    public_crt: config/ssl/full_node/public_full_node.crt
    public_key: config/ssl/full_node/public_full_node.key
  start_rpc_server: true
  sync_blocks_behind_threshold: 300
  target_outbound_peer_count: 8
  target_peer_count: 80
  target_uncompact_proofs: 100
  timelord_peer:
    host: 127.0.0.1
    port: 8446
  wallet_peer:
    host: 127.0.0.1
    port: 8449
  weight_proof_timeout: 360

[arvidn]

"reserved cores" specifies how many of your cores we should not spawn a process for. i.e., how many to reserve for other programs on your computer

[caeci11ius]

Yep, 2 core CPU with no hyperthreading (yep, it's real old). Currently spawns 4 processes. I want it to spawn fewer, so I tried to reserve 1 core. Still got 4 processes.

[InvaderDolan]

@arvidn the reserved cores option didn't change anything for me (I have old config, so I just added the reserved cores: 14 string). The chia start farmer-no-wallet at least doesn't start the chia_wallet which eats 1.1G of PSS RAM, so it's better than usual one.
Here are my sudo smem outputs in 3 ways:

1. chia start farmer-no-wallet + reserved cores: 14

  PID User     Command                         Swap      USS      PSS      RSS
 9996 invader  chia_daemon                        0    46.8M    47.4M    56.8M 
439872 invader  chia_farmer                        0    49.8M    52.2M    66.9M 
439832 invader  chia_harvester                     0    53.8M    56.0M    70.7M 
440929 invader  chia_full_node                     0     3.8M    69.7M     1.1G 
440917 invader  chia_full_node                     0     7.6M    72.5M     1.1G 
440918 invader  chia_full_node                     0     7.7M    72.5M     1.1G 
440919 invader  chia_full_node                     0     7.7M    72.5M     1.1G 
440922 invader  chia_full_node                     0     7.7M    72.6M     1.1G 
440927 invader  chia_full_node                     0     7.9M    72.7M     1.1G 
440926 invader  chia_full_node                     0     9.0M    73.7M     1.1G 
440928 invader  chia_full_node                     0     8.5M    73.8M     1.1G 
440921 invader  chia_full_node                     0     8.6M    73.9M     1.1G 
440924 invader  chia_full_node                     0    12.2M    76.5M     1.1G 
440923 invader  chia_full_node                     0    12.6M    76.9M     1.1G 
440925 invader  chia_full_node                     0    12.7M    77.0M     1.1G 
440920 invader  chia_full_node                     0    15.8M    79.8M     1.1G 
440916 invader  chia_full_node                     0    19.8M    83.5M     1.1G 
440899 invader  chia_full_node                     0     6.1M    89.4M     1.1G 
440900 invader  chia_full_node                     0     6.2M    89.4M     1.1G 
440142 invader  chia_full_node                     0   385.0M   433.1M     1.2G 

2. chia start farmer-no-wallet

  PID User     Command                         Swap      USS      PSS      RSS
442911 invader  chia_farmer                        0    43.8M    46.2M    60.9M 
442857 invader  chia_harvester                     0    46.4M    48.7M    63.3M 
 9996 invader  chia_daemon                        0    48.9M    49.6M    59.0M 
444670 invader  chia_full_node                     0     2.2M    66.3M     1.1G 
444671 invader  chia_full_node                     0     2.2M    66.3M     1.1G 
444673 invader  chia_full_node                     0     2.2M    66.3M     1.1G 
444672 invader  chia_full_node                     0     2.2M    66.3M     1.1G 
444676 invader  chia_full_node                     0     2.2M    66.3M     1.1G 
444677 invader  chia_full_node                     0     2.2M    66.3M     1.1G 
444678 invader  chia_full_node                     0     2.2M    66.3M     1.1G 
444679 invader  chia_full_node                     0     2.2M    66.3M     1.1G 
444674 invader  chia_full_node                     0     2.2M    66.3M     1.1G 
444675 invader  chia_full_node                     0     2.2M    66.3M     1.1G 
444669 invader  chia_full_node                     0     2.3M    66.3M     1.1G 
444667 invader  chia_full_node                     0     5.4M    69.6M     1.1G 
444668 invader  chia_full_node                     0     5.5M    69.6M     1.1G 
444666 invader  chia_full_node                     0    10.7M    74.6M     1.1G 
444660 invader  chia_full_node                     0     7.9M    77.5M     1.1G 
444661 invader  chia_full_node                     0     9.9M    79.4M     1.1G 
443164 invader  chia_full_node                     0   153.0M   210.1M     1.1G 

3. chia start farmer

  PID User     Command                         Swap      USS      PSS      RSS
445005 invader  chia_daemon                        0    47.2M    48.8M    63.8M 
445426 invader  chia_farmer                        0    47.9M    49.5M    64.9M 
445425 invader  chia_harvester                     0    52.4M    54.0M    69.2M 
446300 invader  chia_full_node                     0     3.9M    73.2M     1.1G 
446301 invader  chia_full_node                     0     3.9M    73.3M     1.1G 
446299 invader  chia_full_node                     0     3.9M    73.3M     1.1G 
446294 invader  chia_full_node                     0     7.5M    76.1M     1.1G 
446295 invader  chia_full_node                     0     7.6M    76.1M     1.1G 
446296 invader  chia_full_node                     0     7.6M    76.2M     1.1G 
446289 invader  chia_full_node                     0     8.4M    76.9M     1.1G 
446290 invader  chia_full_node                     0     8.5M    77.0M     1.1G 
446292 invader  chia_full_node                     0     8.6M    77.0M     1.1G 
446297 invader  chia_full_node                     0     8.5M    77.2M     1.1G 
446288 invader  chia_full_node                     0     9.8M    78.2M     1.1G 
446298 invader  chia_full_node                     0    11.1M    79.6M     1.1G 
446293 invader  chia_full_node                     0    12.0M    80.2M     1.1G 
446291 invader  chia_full_node                     0    15.7M    83.7M     1.1G 
446270 invader  chia_full_node                     0     5.7M    98.0M     1.1G 
446271 invader  chia_full_node                     0     5.8M    98.1M     1.1G 
445427 invader  chia_full_node                     0   198.0M   255.8M     1.1G 
445428 invader  chia_wallet                        0     1.1G     1.1G     1.1G 

All of the 3 snapshots I did right after all of the plots of my farm were appeared in chia farm summary with status Farming.
But yes, Chia don't eat that much of memory in terms of PSS RAM usage.

[mariano54]

You need to make sure you are using the main branch instead of latest (1.2.11)

[mariano54]

So based on this, have we concluded that each process is NOT taking up 1.1GiB but instead 90 MiB? 1.1GiB is the total right?

[InvaderDolan]

Yes, but over the time it takes 2x much more. Since my last message maybe (I don't remember exactly) I reboot the machine once. And recently (approx. from yesterday) I started to get more freezes which cause me 10% decrease of daily plot output.

Here is smem output before the restart (chia start farmer-no-wallet -r):

  PID  User      Command                         Swap      USS      PSS      RSS
 3925 invader  chia_daemon                        0    47.0M    47.4M    52.9M 
 5429 invader  chia_farmer                        0    59.5M    60.3M    68.2M 
 5428 invader  chia_harvester                     0   155.6M   156.4M   164.4M  
 8296 invader  chia_full_node                     0   156.6M   214.9M     1.1G 
 8298 invader  chia_full_node                     0   158.1M   216.5M     1.1G 
 8308 invader  chia_full_node                     0   158.2M   216.5M     1.1G 
 8300 invader  chia_full_node                     0   158.2M   216.6M     1.1G 
 8305 invader  chia_full_node                     0   158.3M   216.6M     1.1G 
 8302 invader  chia_full_node                     0   158.3M   216.6M     1.1G 
 8303 invader  chia_full_node                     0   158.3M   216.7M     1.1G 
 8307 invader  chia_full_node                     0   158.5M   216.8M     1.1G 
 8304 invader  chia_full_node                     0   158.5M   216.8M     1.1G 
 8297 invader  chia_full_node                     0   158.6M   216.9M     1.1G 
 8306 invader  chia_full_node                     0   158.9M   217.2M     1.1G 
 8299 invader  chia_full_node                     0   159.3M   217.7M     1.1G 
 8309 invader  chia_full_node                     0   159.3M   217.7M     1.1G 
 8301 invader  chia_full_node                     0   159.7M   217.9M     1.1G 
 8471 invader  chia_full_node                     0   210.9M   364.2M     1.2G 
 8470 invader  chia_full_node                     0   211.0M   364.2M     1.2G 
 5430 invader  chia_full_node                     0     1.5G     1.5G     1.8G 

Total PSS usage is approx. 5527.9M!

Here is smem output after the restart (chia start farmer-no-wallet -r):

  PID User     Command                         Swap      USS      PSS      RSS
2170550 invader  chia_farmer                        0    44.4M    46.8M    60.5M 
 3925 invader  chia_daemon                        0    49.4M    50.2M    58.2M 
2170456 invader  chia_harvester                     0    48.5M    50.8M    64.3M 
2171789 invader  chia_full_node                     0     2.2M    67.9M     1.1G 
2171788 invader  chia_full_node                     0     2.2M    67.9M     1.1G 
2171790 invader  chia_full_node                     0     2.2M    67.9M     1.1G 
2171787 invader  chia_full_node                     0     2.2M    67.9M     1.1G 
2171791 invader  chia_full_node                     0     2.2M    67.9M     1.1G 
2171792 invader  chia_full_node                     0     2.2M    67.9M     1.1G 
2171786 invader  chia_full_node                     0     2.2M    67.9M     1.1G 
2171793 invader  chia_full_node                     0     2.5M    68.1M     1.1G 
2171781 invader  chia_full_node                     0     4.8M    70.4M     1.1G 
2171783 invader  chia_full_node                     0     4.8M    70.4M     1.1G 
2171785 invader  chia_full_node                     0     4.8M    70.4M     1.1G 
2171780 invader  chia_full_node                     0     9.0M    74.4M     1.1G 
2171784 invader  chia_full_node                     0    10.1M    75.4M     1.1G 
2171782 invader  chia_full_node                     0    19.4M    84.1M     1.1G 
2171773 invader  chia_full_node                     0     9.1M    86.0M     1.1G 
2171774 invader  chia_full_node                     0    18.2M    94.4M     1.1G 
2170811 invader  chia_full_node                     0   175.5M   232.7M     1.1G 

After restart, Total PSS usage is approx. 1549.4M! The total difference is 3978.5M or x3.56!

So, I find the solution just run the restart (chia start farmer-no-wallet -r) via cronjob once a day. I hope in further updated chia won't inflate like a balloon over the time.