Heap growth in LSM-trees + 10.6.1 node

[jasagredo]

The version of the node that uses LSM trees reports a 40% growth in heap size. Without profiling analyzing this issue is very difficult.

As the issue only seems to manifest in the workbench, this issue is here to request instructions to run a workload that exhibits the problem with info-table profiling (-hi) and ideally with a IPE-enabled compiler in the P&T workbench.

[jasagredo]

It seems something is broken in my system as I try to start the cluster and I get this:

[nix-shell:~/code/cardano/cardano-node](js/try-leaks-lmetr)$ start-cluster

workbench:  manifest:  component versions:
   cardano-node        10.6.1   3ebaaafaf377a566f385ca4ce770c3c4b41f32b9 try-leaks-lmetr modified
   ouroboros-consensus 0.28.0.0 minimal                                                                              
   ouroboros-network   0.22.4.0 minimal                                                                              
   cardano-ledger-core 1.19.0.0 minimal                                                                              
   plutus-core         1.53.1.0 babbed700851262045d8b411116640b90685779f                                             
   cardano-crypto      1.3.0    null                                                                                 
   cardano-prelude     0.2.1.0  68e015f6e7f17e67d8dbf16bc1590b926d064962                                             

workbench:  top-level:  profile forge-stress-pre-coay, iteration 1 of 1, identified as:  UNIDENTIFIED
workbench:  run | genesis:  cache entry:  preparing a new one..
workbench:  genesis | cache:  preparing entry k3-1000kD-4000kU-fd4ea08:  hit (/home/javier/.cache/cardano-workbench/genesis/k3-1000kD-4000kU-fd4ea08)

workbench:  run | tag:  allocated run identifier (tag):  2025-12-23-14-39-3ebaa-1061-forge-stress-pre-coay-sup
workbench:  run | profile:  pre-supplied (forge-stress-pre-coay):  /nix/store/i6i2b8plxw2kjjqzqrisvcns84bqqwvz-workbench-profile-data-forge-stress-pre-coay
workbench:  profile | overlay:  {}
workbench:  run | topology:  uni-circle
workbench:  run | time:  allocating time:
  - future offset:         100 (of which 0 is extra)
  - start time:            2025-12-23T14:40:52Z
  - shutdown time:         2025-12-23T15:20:52
  - workload end time:     2025-12-23T15:20:52
  - earliest end:          2025-12-23T15:20:52

workbench:  run | genesis:  deriving from cache
workbench:  genesis:  deriving from cache:  /home/javier/.cache/cardano-workbench/genesis/k3-1000kD-4000kU-fd4ea08 -> run/2025-12-23-14-39-3ebaa-1061-forge-stress-pre-coay-sup/genesis
workbench:  genesis:  finalizing retrieved cache entry in:  run/2025-12-23-14-39-3ebaa-1061-forge-stress-pre-coay-sup/genesis

workbench:  run | genesis:  deploying..
workbench:  lockfile:  waiting to acquire the lock on /tmp/workbench.lock
workbench:  lockfile:  lock acquired
workbench:  run:  allocated 2025-12-23-14-39-3ebaa-1061-forge-stress-pre-coay-sup @ run/2025-12-23-14-39-3ebaa-1061-forge-stress-pre-coay-sup
  - run dir:         run/2025-12-23-14-39-3ebaa-1061-forge-stress-pre-coay-sup
  - profile JSON:    run/2025-12-23-14-39-3ebaa-1061-forge-stress-pre-coay-sup/profile.json
  - node specs:      run/2025-12-23-14-39-3ebaa-1061-forge-stress-pre-coay-sup/node-specs.json
  - topology:        run/2025-12-23-14-39-3ebaa-1061-forge-stress-pre-coay-sup/topology/topology.pdf
  - node base port:  30000
  - EKG URL (node-0):                                  http://localhost:3100/node-0
  - Prometheus URL (node-0):                           http://localhost:3200/node-0
  - RTView URL (depending on cardano-tracer build):    http://localhost:3300
  - era:                conway
  - epoch slots:        600
  - slot duration:      1
  - k:                  3
  - active slots coeff: 0.050
  - hosts:              3
  - pools:              3
    - normal:             3
    - dense:              0
  - UTxO:               4000000, of which:
    - delegated:          1000000
    - generated:          72000
    - stuffed:            4000000
  - delegators:         1000000
  - generator duration: 2400s
    - requested epochs:   3ep
    - effective epochs:   4ep
    - transaction count:  36kTx
    - full blocks:        241

  - terminate at slot:  2400

workbench:  run:  starting 2025-12-23-14-39-3ebaa-1061-forge-stress-pre-coay-sup
workbench:  run | scenario:  starting fixed-loaded
error: <class 'PermissionError'>, [Errno 13] Permission denied: file: /nix/store/6y8czfkdv78vcd8h72ghk22xsk4gihyz-python3.12-supervisor-4.2.5/lib/python3.12/site-packages/supervisor/xmlrpc.py line: 557
workbench:  supervisor:  fatal: failed to start cardano-tracer
config.json -------------------------------------
{
  "ekgRequestFreq": 1,
  "hasEKG": {
    "epHost": "127.0.0.1",
    "epPort": 3100
  },
  "hasPrometheus": {
    "epHost": "127.0.0.1",
    "epPort": 3200
  },
  "logging": [
    {
      "logFormat": "ForMachine",
      "logMode": "FileMode",
      "logRoot": "."
    }
  ],
  "metricsHelp": "../../../cardano-tracer/configuration/metrics_help.json",
  "network": {
    "contents": "tracer.socket",
    "tag": "AcceptAt"
  },
  "networkMagic": 42,
  "resourceFreq": null,
  "rotation": {
    "rpFrequencySecs": 15,
    "rpKeepFilesNum": 10,
    "rpLogLimitBytes": 1000000000,
    "rpMaxAgeHours": 24
  }
}
tracer stdout -----------------------------------
cat: run/2025-12-23-14-39-3ebaa-1061-forge-stress-pre-coay-sup/tracer/stdout: No such file or directory