[Bug] Ledger Log files not getting removed

[naahk37]

Search before reporting

• I searched in the issues and found nothing similar.

Read release policy

• I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

User environment

Pulsar 4.0.2
openjdk version "17.0.14"
Ubuntu 24.04
3 brokers, 3 bookies, 3 zookeepers

Issue Description

Hi,

I'm running a Pulsar cluster with the specs mentioned above and partitioned topics and I have two problems:

• The filesystem usage on the bookies doesn't seem to go down (bk1: 100GB, bk2: 400GB, bk3: 100G). I already set a retention policy on my main namespace (2weeks, 10GB) and the metriks in Grafana report the correct topic sizes (100GB storage size and ~30GB backlog size). I get the usage on bk1 and bk3, but not the 400GB on bk2...

• The bookie service on bk2 stops frequently with an error "io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4194304 byte(s) of direct memory (used: 2147483648, max: 2147483648)". I can't find the setting where I can control the limit for this memory setting. I already increased the jvm allocations in the pulsar_env.sh those don't seem to correlate..

On bookie bk2 there are around 12k ledger log files, whereas on the other two bookies are ~200 and 85 files.
On bk2 I have the following log entries regarding GC:
"Forced garbage collection triggered by thread: LedgerDirsMonitorThread",
"Garbage collector thread forced to perform GC before expiry of wait time",
"Extracting entry log meta from entryLogId: 185",
"GarbageCollectorThread-6-1 Set forceGarbageCollection to false after force GC to make it forceGC-able again"

It looks like GC is happening, because the "deleted ledger" count goes up when the bookie is running.

Error messages

Reproducing the issue

not really applicable - keeps happening when cluster is running

Additional information

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

There's a known issue that ledger index RocksDB files don't get compacted automatically, but that is different from what you are describing. That is apache/bookkeeper#4554.
Ledger entry count metrics will also be wrong when the compaction hasn't run.
More details about workarounds in apache/pulsar-helm-chart#633

[lhotari]

Pulsar 4.0.2

There are several known issues that have been resolved. Please upgrade to latest 4.0.x version which is 4.0.7 at the moment.
Pulsar 4.0.6 upgraded to BookKeeper 4.17.2 with important fixes.

openjdk version "17.0.14"

What distribution are you using? Pulsar's docker files use Java 21. It's recommended to run Pulsar 4.0.x with Java 21. For example, 17.0.14 contains a known bug https://bugs.openjdk.org/browse/JDK-8351933 which has impacted some Pulsar/BookKeeper users. The problem is fixed since Java 21.0.8. This is available since Pulsar 4.0.6 docker images.

[naahk37]

Alright, thank you!
I installed Java 17, since it is recommended here: https://github.com/apache/pulsar/blob/master/README.md#pulsar-runtime-java-version-recommendation

I don't use Docker, I deployed it bare metal on Ubuntu VMs.
I would now update Java to 21 and then upgrade the cluster.
Is it correct that I just need to update the /bin and /conf directories node after node?

[lhotari]

Is it correct that I just need to update the /bin and /conf directories node after node?

@naahk37 I don't think that it's correct. A better approach would be to track changes that you make in the conf files and apply them to a fresh installation of the new version of Pulsar, although it is possible to copy files in the conf directory to the new version. In the Apache Pulsar project, the actively supported deployment option is by using a Helm deployment with Kubernetes. There's also Docker Compose and (Terraform+)Ansible options available. More details in the deployment docs.

[naahk37]

Yesterday I updated all of the nodes to 4.0.7 (and Java to 21) and the storage usage on my bookie bk2 is now down to 200G and I think the GC is still going.
So thank you! :)

About my 2. question - because of the "OutOfDirectMemoryError" I set dbStorage_writeCacheMaxSizeMb=128 and dbStorage_readAheadCacheMaxSizeMb=256 on bk02, because it was mentioned as a workaround in another thread.
Is it safe to assume that I can use the default values for these two settings again, after the update?

[lhotari]

About my 2. question - because of the "OutOfDirectMemoryError" I set dbStorage_writeCacheMaxSizeMb=128 and dbStorage_readAheadCacheMaxSizeMb=256 on bk02, because it was mentioned as a workaround in another thread.
Is it safe to assume that I can use the default values for these two settings again, after the update?

I'm not aware of changes in this area between 4.0.4 and 4.0.7 . The tuning of various values depends on your environment.

To avoid OutOfDirectMemoryError, you should be setting managedLedgerMaxReadsInFlightSizeInMB.
There are some details in #23466 about tuning the broker cache and it also includes tuning of managedLedgerMaxReadsInFlightSizeInMB.
The broker cache has significantly improved in Pulsar 4.1.x and if you'd upgrade to 4.1.1, you'd get the benefits of PIP-430: Pulsar Broker Cache Improvements.

[naahk37]

Alright, thank you!
I'll think about the upgrade to 4.1 ;)