Extremly high RAM usage after hours of inactivity

[motmot80]

Description

After 4-5 h of inactivity a tiny amount of published messages leads to a doubling the RAM usage.

Reproduction

1. Fill up a cluster with 150 million messages to 50k queues (local non-lazy queues - 16 k queues each node)
   a) each node allocates about 14 GB of 32 GB RAM
   b) publish rates > 10 000 msg/sec fan out exchange are working fine and stable
   c) Filled within 3 hours
2. 4-5 h of total inactivity (no RAM churn visible)
3. Publish some messages f.e. about 50 (no fan out messages - only routed to a single queue)

Expected

The cluster performs like 4 hours before using about the same amount of RAM (14 GB).

Actual

Right after the first messages are published each cluster node allocates another 15 GB RAM!! within 5 minutes hitting the high-watermark.

Log (Info level)

No log entries till the memory high watermark is hit.

Rabbit MQ setup

Rabbit MQ-Version: 3.8.1 (Erlang 22.1.6)
Nodes count: 3
Node hardware: 6 CPUs - 32 GB RAM each

Reproduction setup

Type	Count	Description
Queues (lazy or non-lazy local )	50 000	16 666 per node
Messages ready	150 000 000	~ 14 GB RAM

System load before error

CPU(user): < 10 %
CPU(kernel): < 10 %
RAM: < 15 GB (~50 %)

System load after "resuming"

CPU(user): < 70 %
CPU(kernel): < 15 %
RAM: < 30 GB (~95 %)