Galexie: add support for S3 and S3-compatible services.

[overcat]

What problem does your feature solve?

Adding S3 as a storage option will give users more choices, including the wonderfully affordable Cloudflare R2. Frankly, I'm trying to avoid a scenario where I go bankrupt just from using Google Cloud.

I've even cooked up some code locally that I can PR if you're interested!

What would you like to see?

Add support for S3 and S3-compatible services.

What alternatives are there?

N/A

[johncanneto]

related to https://github.com/stellar/go/pull/5708

[sydneynotthecity]

👋 @overcat, glad to see you're using Galexie! we agree that enabling pluggable data stores is a great idea.

If you are using galexie to ingest full ledger history, there is a major feature shipping in Protocol 23, and you may want to hold off full history ingestion until the protocol upgrade later this quarter.

Starting in P23, CAP-67 will give infra operators the option to allow built-in “classic” operations to emit asset events in the same format as smart contract transaction events, unifying the event stream between smart contract invocations and built-in operations. This makes tracking asset balances, circulating supply and token movements much simpler because developers can read from one events stream.

This CAP is unique in the fact that operators will be able to retroactively emit these events for classic operations beginning with the genesis ledger onwards. We are recommending that full archive nodes enable support for the full, unified event stream.

This means you have two options:

1. Reingest full history once Protocol 23 is live, replacing existing S3 ledger meta files with the newly generated meta that includes retroactive events, or
2. Backfill once, after Protocol 23 is live.

We have some blogs that will be published later this month that i can link to with more details

[overcat]

Hi @sydneynotthecity, thank you for this information. I have completed syncing all historical data to GCS 😂, and I will reingest it on Cloudflare R2 when P23 is released.

[urvisavla]

@overcat, thanks for creating the issue! We're planning to prioritize the S3 backend implementation for Galexie. That said, if you’ve already built an S3 backend that supports both reading and writing and you’re open to contributing it, we’d really appreciate it. Feel free to open a PR!

The existing GCS implementation is a good reference for what a datastore implementation should look like and we’ve also put together some guidelines for adding new storage. I’d be happy to collaborate on a PR if that's helpful.

I had a couple quick questions about the GCS datalake you set up, if you don’t mind sharing:

1. Did you use Galexie to generate it? If so, which version were you running?
2. Approx. how long did it take to generate the datalake for pubnet?

[overcat]

Hi @urvisavla,

Feel free to open a PR!

I'll submit this after I've added the full tests.

Did you use Galexie to generate it? If so, which version were you running?

I'm running this via Docker:

REPOSITORY                TAG              IMAGE ID       CREATED       SIZE
stellar/stellar-galexie   latest           1148dc66477c   7 weeks ago   256MB

Approx. how long did it take to generate the datalake for pubnet?

It took about two days in total. I used 97 instances of Linode Dedicated 8 GB (4 CPUs), with each instance processing 600,000 ledgers. Most of the instances finished processing these ledgers in 1.5 days, while a few took 2 days.

[overcat]

I added a new PR #5748, but have not yet added integration tests. I tried to sync the data to R2 with it, and it seems to be working well.