Add galexie service for local ledger meta store #870
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request adds galexie service support to quickstart for local network development, enabling developers to generate and test against local meta lakes. Galexie publishes ledger metadata to a filesystem datastore and serves it via nginx at /meta-archive.
Key changes:
- Adds galexie binary built from source in Dockerfile
- Implements galexie service configuration with filesystem datastore
- Enables service with
--enable galexieflag (local network only)
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| start | Adds galexie environment variable, initialization function, startup logic, validation, and argument processing |
| Dockerfile | Adds galexie builder stage using golang:1.24-trixie and copies binary to quickstart image |
| images.json | Adds galexie dependency from stellar/stellar-galexie repository across all image configurations |
| local/galexie/etc/galexie.toml | Configures galexie with filesystem datastore, admin port, and stellar-core connection |
| common/galexie/bin/start | Provides startup script that executes galexie append command with config |
| local/supervisor/etc/supervisord.conf.d/galexie.conf | Supervisor configuration for galexie service with autostart=false |
| local/supervisor/etc/supervisord.conf.d/meta-archive.conf | Supervisor configuration for meta-archive HTTP server with autostart=true |
| local/nginx/etc/conf.d/meta-archive.conf | Nginx location block to proxy /meta-archive requests to the meta-archive service |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
README.md
Outdated
| - RPC: `http://localhost:8000/rpc` | ||
| - Lab: `http://localhost:8000/lab` | ||
| - Friendbot: `http://localhost:8000/friendbot` | ||
| - Meta Archive: `http://localhost:8000/meta-archive` (available with `--local` and `--enable galexie`) |
There was a problem hiding this comment.
@stellar/data-eng @stellar/platform-eng @stellar/devx Does anyone have opinions on what the meta should be called in quickstart and what path it should be served on? I named it meta-archive because it's an archive of ledger close meta, but lifting my head from keyboard realise we don't ever call it that anywhere else.
The terminology I've seen about:
- "Ledger Metadata Storage" in SEP-54
- "meta lake" I've heard people say out loud a number of times
- "data lake" at https://developers.stellar.org/docs/data/indexers/build-your-own/galexie
- "galexie data lake" at https://quasar.lightsail.network/
- "" at AWS Public Dataset
I'd like to include the word 'meta' in the quickstart path because it is ledger close meta.
/data-lake?/meta-lake?/meta?/ledger-close-metas?/galexie?/meta-archive?
There was a problem hiding this comment.
I think i'd prefer /ledger-meta or /ledger-meta-store. Seems more in line with SEP-54, and is immediately identifiable.
But am fine with pretty much any of them besides *-lake, given this won't store the full history.
There was a problem hiding this comment.
+1 I also like /ledger-meta-store. it denotes that the directory is intended as a file store and what it's storing.
To @mootz12's point the *-lake terminology is more broadly used in big data architectures to describe a centralized place to hold all raw, untransformed data. Since the intent isn't to store full history, or other data objects, I think it's best to stick with *-store instead.
There was a problem hiding this comment.
I've updated this change so that the store is now referred to as the ledger-meta-store everywhere in the code. There's no mention now of archive or lake. For the URL I made it the shorter /ledger-meta.
One question that's outstanding for is: @sydneynotthecity Is there a plan in the future for the data that Galexie generates to be more than ledgers?
If yes, we might want to put this data under a different name here that would encompass whatever data that Galexie is generating rather than making it about ledger meta only.
leighmcculloch
changed the title
What
Add galexie as an optional service in quickstart for local networks. Build galexie from source in the Dockerfile, configure it to publish ledger metadata to a filesystem datastore, and serve the meta archive via nginx at /meta-archive. Enable with --enable galexie flag.
Why
Developers building solutions against mainnet meta lakes generated by galexie have no way to test locally with quickstart. This enables local meta lake generation for development and testing workflows.
Close #835
Dependent on:
Todo