Stream segment file cleanup command

[smith84]

It's clear from the docs and various forum posts that streams which are not being regularly published to will not have their segment files cleaned up according to the Retention rules.
#4384

This is likely to become an issue for our use case. Users create their own streams in the system, and we promise only 7 days' retention of the messages in their stream. We have defined our streams with 10MiB segment size and 7D max-age retention policy. If one of our users decides to suddenly stop publishing to their stream, the segment files will not be cleaned up, so we'll potentially end up with significant wasted disk space. Ideally I hope to have just one 10MiB file on the disk for each of these dormant streams.

I had thought that perhaps we could set up a background process to periodically drop a small 'null' message into every stream once every 24 hours to trigger a cleanup but on closer reading it looks like the segment files are only cleaned up when the current segment file fills up and Rabbit goes looking to allocate a new file.

In the absence of RabbitMQ server periodically scanning for out of date segment files, what I'm proposing is a command (ideally accessible through the Management API) to trigger the segment file cleanup process, which we could call periodically.

I had a thought that we could potentially monitor for publishing activity and purge the stream from application code via the Management API but I'm not sure that DELETE /api/queues/vhost/name/contents is available for streams. (The Management Web UI doesn't appear to have a Purge function for streams.)

[kjnilsson]

I think periodically re-evaluating streams with a time based retention setting makes sense. We'd always need to keep at least one segment, that cannot change, but we should at least be able to clear any other surplus segments automatically.

Retention evaluation does have a cost, especially for larger streams but doing it hourly would probably be perfectly fine.

[smith84]

Thank you very much for your fast reply.
We will probably end up in the order of 10k-50k streams in our cluster, so I was a bit wary of pushing the idea of an automatic background task, especially if it forces a check on all streams at once.
If the automatic process can be efficient enough to operate on a large number of streams, that sounds perfect and is obviously the easiest approach for us.
And it's clear that in any case at least one segment will be kept - we deliberately configure a low segment size for this reason.

[mkuratczyk]

Hi. There is a pretty good chance you'd be the first to use so many Streams. :) Would you mind sharing some details?

1. what is the use case? why do you need so many streams?
2. if I understand correctly, you are using it already but not yet at that scale. How many do you have right now? any other issues that you see?
3. what's the current/expected message throughput per stream?
4. what's the message size?

[smith84]

1. The system is a central hub for handling IoT messages - customers can control message routing into different streams for authentication/security purposes. We aren't (currently) constraining customers on the number of streams so we need to account for users who may choose to do crazy things...
2. Our staging instance is being used for load testing and has ~350 streams on a relatively small RabbitMQ host, with no problems seen so far. Early on we found issues with high CPU loading on freshly created streams. This was reported through our hosting company and corrected in a recent update (something to do with metrics collection?)
3. Throughput per stream is bursty, peaking at ~100-1000 messages / sec, but averaging at ~1-10 messages / sec.
4. Messages are ~50-100 bytes in size. (We're using binary encoding to keep disk space under control.)

[mkuratczyk]

Thanks a lot for the details! One more question if I may - what protocol do you use for this? Given you mentioned routing, I guess that's over AMQP, but I'd like to double check. Thanks

[smith84]

At this stage we're using AMQP 0-9-1 for both publishing and consuming since we moved from queues to streams fairly late in our development program. We might look to enable STREAM in the next year or so but there are other priorities to tackle first.

[kjnilsson]

rabbitmq/osiris#107

I've created this issue to implement this functionality in the core streams component.