This is an ongoing project for integrating RabbitMQ streams with S3 storage.
rabbitmq-stream-s3 is not stable, with frequent changes in design and functionality.
This project currently requires specific development branches of the rabbitmq-server and osiris repositories:
Branch: aws/streams-tiered-storage-dev
Contains changes needed for S3 integration.
Branch: md/log-reader-and-manifest
Contains the abstraction layer in Osiris. See Tiered Storage Support for RabbitMQ Streams
- Clone the RabbitMQ server repository
git clone https://github.com/rabbitmq/rabbitmq-server.git
cd rabbitmq-server
- Switch to the required branch
git checkout aws/streams-tiered-storage-dev
- Build with the stream-s3 plugin with the correct osiris branch
ADDITIONAL_PLUGINS=rabbitmq_stream_s3 \
dep_rabbitmq_stream_s3="git [email protected]:amazon-mq/rabbitmq-stream-s3.git main" \
dep_osiris="git https://github.com/rabbitmq/osiris md/log-reader-and-manifest" \
make
For more information on how to build and develop plugins in RabbitMQ, see plugin-development
Osiris needs to be configured to use our s3 log_reader and log_manifest:
streams.log_reader = rabbitmq_stream_s3_log_reader
streams.log_manifest = rabbitmq_stream_s3_log_manifest
See rabbitmq_aws
See CONTRIBUTING for more information.
This project is licensed under the Apache-2.0 License.