Add fs backend for tests

[LoisSotoLopez]

Issue #36

• Adds a new backend for the S3 API having the local filesystem as storage, for testing purposes.
• Provides a simple end-to-end test-suite exemplifying the generation of data in the tiered storage:
  • Proper configuration is set to enable tiered storage.
  • A stream is declared and data is published to it, trying to experiment/visualize the logic behind when data is replicated to the tiered storage.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[LoisSotoLopez]

A few comments:

• Not sure about the test suite name.
• Could get some suggestions on what other test cases would be useful to add.
• Not sure about adding a check for "the stream data is cleared out from tiered storage when the stream is locally deleted". I understand that feature is desired but probably not implemented yet. Edit: Ah, just saw the drafted PR about this, nice!.

Also, I'm getting the following error when running the tests after rebasing latest main:

make[1]: Leaving directory '/home/lois/Files/repos/tiered_storage/rabbitmq-server/deps/rabbit'
 GEN    test-build
 ERLC   rabbitmq_stream_s3_api_aws_SUITE.erl rabbitmq_stream_s3_log_manifest_machine_SUITE.erl s3_streams_SUITE.erl
resource.hrl:8:2: record resource already defined
make[1]: *** [../../erlang.mk:2100: /home/lois/Files/repos/tiered_storage/rabbitmq-server/.erlang.mk/rabbitmq_stream_s3.last-testdir-build] Error 1
make: *** [../../erlang.mk:2112: test-build] Error 2

👀

[LoisSotoLopez]

Re-drafting because test don't pass after rebase :)

Edit after figuring out why:
Ah, I was assuming the manifest would always be uploaded and took that as a condition to decide whether the stream existed or not on the tiered storage. Will adapt the code, but I'm wondering why the manifest might not exist there sometimes.

❯ tree .../rmq-ct-s3_streams_SUITE-1-21000@localhost/mnesia/rmq-ct-s3_streams_SUITE-1-21000@localhost/rabbitmq_stream_s3_api_fs/
.../rmq-ct-s3_streams_SUITE-1-21000@localhost/mnesia/rmq-ct-s3_streams_SUITE-1-21000@localhost/rabbitmq_stream_s3_api_fs/
└── rabbitmq
    └── stream
        └── __stream_1_1769682112367012824
        <missing metadata/manifest> here
            └── data
                └── 00000000000000000000.fragment

[the-mikedavis]

Ah yeah the "uploading the manifest" part probably changed in the rebase because of 7c8fbab. There's some logic to debounce changes so we're not uploading manifests for every little change and there was a bug that made the debounce never happen.

You can set the env application:set_env(rabbitmq_stream_s3, manifest_debounce_modifications, 1) so that the manifest uploads for every change.

[the-mikedavis]

For the root directory I was thinking we could pass in a directory we create under the priv_dir from Common Test via application env. So we could have functions like

% rabbitmq_stream_s3_api_fs.erl

-spec data_dir() -> file:filename_all().
data_dir() ->
    {ok, Dir} = application:get_env(rabbitmq_stream_s3, api_fs_data_dir),
    Dir.

-spec key_to_path(rabbitmq_stream_s3_api:key()) -> file:filename_all().
key_to_path(Key) ->
    filename:join(data_dir(), Key).

If we share the same directory between all Rabbit nodes in a suite/group it would match the 'write once read many' interaction we have with an object store: a replica node could read data from the shared "remote tier" directory.

So if we had a three node cluster, the CommonTest log_private directory might look like

% tree path/to/log_private/
path/to/log_private/
├── ct_rabbit@mango22-1
│   └── ...
├── ct_SUITE_NAME_remote_data/rabbitmq/stream
│   ├── __sq_1769635311360384578
│   │   ├── data
│   │   │   ├── 00000000000001030220.fragment
│   │   │   ├── 00000000000001035552.fragment
│   │   │   └── ...
│   │   ├── metadata
│   │   │   └── efe4b29116e5a364.manifest
│   │   └── ...
│   └── ...
├── rmq-ct-SUITE_NAME-1-21162@localhost
│   ├── mnesia
│   │   └── rmq-ct-SUITE_NAME-1-21162@localhost
│   │       ├── stream
│   │       │   └── __sq_1769635311360384578
│   │       │       ├── 00000000000005438715.index
│   │       │       ├── 00000000000005438715.segment
│   │       │       ├── 00000000000005697381.index
│   │       │       └── 00000000000005697381.segment
│   │       └── ...
│   └── ...
├── rmq-ct-SUITE_NAME-2-21216@localhost
│   ├── mnesia
│   │   └── rmq-ct-SUITE_NAME-2-21216@localhost
│   │       ├── stream
│   │       │   └── __sq_1769635311360384578
│   │       │       ├── 00000000000005438715.index
│   │       │       ├── 00000000000005438715.segment
│   │       │       ├── 00000000000005697381.index
│   │       │       └── 00000000000005697381.segment
│   │       └── ...
│   └── ...
└── rmq-ct-SUITE_NAME-3-21270@localhost
    ├── mnesia
    │   └── rmq-ct-SUITE_NAME-3-21270@localhost
    │       ├── stream
    │       │   └── __sq_1769635311360384578
    │       │       ├── 00000000000005438715.index
    │       │       ├── 00000000000005438715.segment
    │       │       ├── 00000000000005697381.index
    │       │       └── 00000000000005697381.segment
    │       └── ...
    └── ...

[LoisSotoLopez]

Not undrafting yet as I would like to make the test multinode and probably force some new writer election testcase. Sounds good?