DBI data limited to 2GB by protobuf limit

[wojas]

The protobuf spec limits individual messages and bytes/string objects to 2GB:
https://protobuf.dev/programming-guides/proto-limits/

This affects our DBI snapshots when one DBI grows larger than this limit. With the PowerDNS Authoritative server, this is most likely to happen with the records DBI in the shard LMDB, at roughly around 20 million records. Lightning Stream will write snapshots with larger DBIs, but it will then fail to load them with an error like:

level=warning msg="[shard         ] Snapshot marked as corrupt and will be ignored" component=receiver db=shard error="invalid length (...) for length-delimited field at byte 103: field length cannot be more than 2GB" 

Workarounds

If you hit this limit, the first workaround you can try is to clean more tombstones by enabling the sweeper section or reducing retention_days, but please do read the warnings in the example config carefully!

Solution

In the short term, we need to find a way to remove this limit for our snapshots. This may affect interoperability with other implementations, where other protobuf implementations may fail to load these large snapshots, but we do not regard these snapshots as a public interface anyway.

In the longer term we may want to change the snapshot format by chunking messages to never exceed this limit.