feat: Add MBTilesReloader (Tile Reload Phase 2) by Auspicus · Pull Request #2717 · maplibre/martin

Auspicus · 2026-04-20T11:42:38Z

todo:

cleanup
tests/test.sh end to end test of moving files in and out
update docs/...

CommanderStorm

Very clean. Good job.

I have a few comments (a bit more nit-picky than usual, but we are going to model the rest of this so I hope this is OK with you).

The biggest ask that I have:

Please add a test in tests/test.sh that auto-discovery works as intended with add, edit, move.
Please document this behaviour in our docs

Also, I don't nessarily think that we need an unstable-reloader feature..
LGTM if it works properly..

CommanderStorm · 2026-04-20T19:56:39Z

+        let init: BTreeMap<String, (PathBuf, u64)> = match &config.mbtiles {
+            FileConfigEnum::Config(cfg) => {
+                let mut m = BTreeMap::new();
+                if let Some(s) = &cfg.sources {
+                    for (id, src) in s {
+                        let path = src.get_path();
+                        let Ok(metadata) = path.metadata() else {
+                            continue;
+                        };
+                        let Ok(modified) = metadata.modified() else {
+                            continue;
+                        };
+                        let Ok(unix_epoch_delta) = modified.duration_since(UNIX_EPOCH) else {
+                            continue;
+                        };
+
+                        m.insert(
+                            id.clone(),
+                            (path.clone(), unix_epoch_delta.as_millis() as u64),
+                        );
+                    }
+                }
+                m
+            }
+            _ => BTreeMap::new(),
+        };


I think this collecting from the FS will be a relatively common pattern, at least for COG and PMTiles we will need to use something like this.

Lets extract the usefull part (checking one path? Very open to oppinions) to a method.

Auspicus · 2026-04-20T20:51:17Z

Thanks for the swift feedback @CommanderStorm! I'll get on this again some time this week.

Split Catalog into StaticCatalog so the /catalog endpoint reads tile sources live from TileSourceManager, allowing hot-reloaded sources to appear without a restart. Simplify NewSource/DeletedSource by merging name + unique_name into a single pre-resolved id field, pushing ID resolution upstream. Improve MBTiles reloader bootstrap to compute initial source state with modification timestamps and handle all FileConfigEnum variants. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

`MartinError` was not `Send` because it contained `WebError(actix_web::Error)`, where `actix_web::Error` wraps a non-Send `dyn ResponseError`. This prevented `ReloadAdvisory` (which stores `MartinResult<BoxedSource>`) from being used across `.await` points inside `tokio::spawn`, as required by the MBTiles reloader. The `WebError` variant was introduced in f88fd10 but never used — no code converts an `actix_web::Error` into a `MartinError` via `?` or `.into()`. The srv layer returns `ActixResult` directly, not `MartinResult`. Also adds `Send + Sync` to the `MetricsIntialisationError` box (the only other non-Send variant), and updates the MBTiles reloader to use `MartinResult` and the now-async `apply_changes`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ad E2E tests Move watch-path and initial-source extraction from martin.rs into MBTilesReloader::new(), so the call site only hands over the raw config. Directories are now stored on the struct, removing the need to thread them through to watch() and discover_sources(). Extract the event-loop body into process_event() and update self.sources only after apply_changes() succeeds, avoiding stale state on error. Switch discover_sources() to async fs::read_dir and widen the timestamp field to u128 to preserve full millisecond precision. Add an integration test covering the full add/update/remove lifecycle for a watched MBTiles directory, including polling helpers for catalog and log assertions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…cursive directory watching

…and tighten API Move TileSourceManager into the struct so callers don't need to thread it through to watch/start separately. Rename watch -> start to better reflect that it returns immediately after spawning a background task. Canonicalize watched directories eagerly in new() with explicit warnings on failure, so both start() and discover_sources() operate on the same resolved paths without redundant per-call canonicalization. Sort and dedup after canonicalization to avoid registering the same directory twice (which would produce duplicate reload events for a single file change). Remove the now-redundant is_absolute() filter in start() and the path.is_absolute() guard in discover_sources() — canonicalize() already guarantees absolute paths on success. Make discover_sources and process_event private and add doc comments to the two public methods. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…n up discover_sources Adds `path_modified_ms` and `resolve_dir_entry` to `reload/mod.rs` so the per-entry warn-and-skip logic is reusable across future reloaders (cog, pmtiles, postgres). The initial source snapshot in `new()` and the hot-reload path in `discover_sources` both use these helpers, eliminating the duplicated metadata/timestamp chain. Also flattens the mbtiles extension guard to an early-continue and adds doc comments to the two private methods. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

This reverts commit fe582728fd27235071d9dffd0fe5e6ff4bac8fd0.

Remove the stale warning that catalog refresh at runtime is unsupported, and add a new section explaining the MBTiles hot reload feature introduced in this phase: watched directories automatically reflect file additions, modifications, and removals without a server restart. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… unused warning Use #[cfg(feature = "_tiles")] directly on the idr parameter of Config::resolve() and on the corresponding argument at each call site, so the parameter simply does not exist in feature combinations where it would be unused. Also restores the IdResolver import gates to _tiles across all affected files. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

CommanderStorm

Impressive work.
I have a few passing remarks, but let's merge

CommanderStorm · 2026-04-27T07:58:25Z

+- **File removed** — the source is removed from the catalog.
+
+!!! note
+    Hot reload applies to directories configured under `mbtiles.paths` (or passed on the CLI). Named sources listed under `mbtiles.sources` are snapshotted at startup and are not watched for changes.


🤔 likely we can get rid of this by reshuffling some code.
Lets see if we can get rid of this.

Alas, I am fine with pushing this to a future PR or until someone has time to do this redhuffeling

Yeah, my main hesitation to doing this now was that I don't know what the right way to determine the directories for watching should be. Right now, it's reasonable to assume if you configure a directory for Martin to gather sources from that you probably want Martin to also notice when new sources become available in that directory. With named sources, I still have the following questions about the right directories to watch:

should we watch directories of the named sources?

should we only watch the immediate parent of the source?

if we watch more than the immediate parent, how far above the source should we watch?

CommanderStorm · 2026-04-27T13:09:25Z

+    /// Tile Source Manager to which we should send `ReloadAdvisory` messages.
+    tile_source_manager: TileSourceManager,
+    /// Map of Source ID => (path, modified timestamp, cache policy).
+    sources: BTreeMap<String, (PathBuf, u128, CachePolicy)>,


nit: I usually prefer a struct for these things. Makes it clearer what the u128 is refering to for others, but I am fine as-is ^^

Maybe can be addressed in the follow up to add support for the other local file tile sources.

CommanderStorm · 2026-04-27T13:16:55Z

CI needs to be made happy though..

…should_ be impossible as SQLite does not add FILE_SHARE_DELETE capability when opening the file

Auspicus · 2026-04-27T23:17:02Z

@CommanderStorm Windows is passing now (using a dirty hack). I wish we could simulate a real delete but it looks like SQLite has no way to add a FILE_SHARE_DELETE permission so faking the behaviour feels wrong but accurate since it shouldn't be possible on Windows.

## 🤖 New release * `martin-tile-utils`: 0.7.0 -> 0.7.1 (✓ API compatible changes) * `mbtiles`: 0.17.0 -> 0.17.1 (✓ API compatible changes) * `martin-core`: 0.5.0 -> 0.5.1 (✓ API compatible changes) * `martin`: 1.7.0 -> 1.8.0 <details><summary>Changelog</summary> ## `mbtiles` <blockquote> ## [0.17.1](mbtiles-v0.17.0...mbtiles-v0.17.1) - 2026-04-28 ### Added - support `content_type` in PostgreSQL function source SQL comments for raster tiles ([#2671](#2671)) </blockquote> ## `martin-core` <blockquote> ## [0.5.1](martin-core-v0.5.0...martin-core-v0.5.1) - 2026-04-28 ### Other - update Cargo.toml dependencies </blockquote> ## `martin` <blockquote> ## [1.8.0](martin-v1.7.0...martin-v1.8.0) - 2026-04-28 ### Added - Add MBTilesReloader (Tile Reload Phase 2) ([#2717](#2717)) - support `content_type` in PostgreSQL function source SQL comments for raster tiles ([#2671](#2671)) ### Other - remove a few unused deps from Cargo.toml to not waste time building them ([#2738](#2738)) - update Cargo.toml dependencies </blockquote> </details> --- This PR was generated with [release-plz](https://github.com/release-plz/release-plz/). --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Auspicus mentioned this pull request Apr 20, 2026

Reload PMTiles sources if their underlying PMTiles file data changes #2713

Open

CommanderStorm requested changes Apr 20, 2026

View reviewed changes

Auspicus force-pushed the feat/tile-source-manager/phase-2 branch from 063bb6a to 3db3683 Compare April 24, 2026 07:51

Auspicus commented Apr 26, 2026

View reviewed changes

Comment thread martin/src/config/file/tiles/reload/mbtiles.rs

Auspicus commented Apr 26, 2026

View reviewed changes

Comment thread martin/src/config/file/tiles/reload/mbtiles.rs Outdated

Auspicus commented Apr 26, 2026

View reviewed changes

Comment thread martin/src/error.rs

Auspicus and others added 12 commits April 27, 2026 10:43

feat: add MBTilesReloader

8fec7ca

refactor: use initializer

174a990

refactor: cleanup

ab2748a

refactor: move IdResolver creation out to root

89cf6aa

chore: sort Cargo.toml

df0154e

chore(fmt): apply pre-commit formatting fixes

e0d81c4

chore: add FIXME noting that we should switch from NonRecursive to Re…

70128b3

…cursive directory watching

Auspicus force-pushed the feat/tile-source-manager/phase-2 branch from c0611aa to a578728 Compare April 27, 2026 00:44

pre-commit-ci Bot and others added 5 commits April 27, 2026 00:46

chore(fmt): apply pre-commit formatting fixes

3db7ddf

chore: remove unused RESERVED_KEYWORDS import from main.rs

fe09526

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chore: remove unused Actix error variant from MartinCpError

9bf2409

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Revert "chore: remove unused Actix error variant from MartinCpError"

bd0674c

This reverts commit fe582728fd27235071d9dffd0fe5e6ff4bac8fd0.

chore: cargo clippy + fmt

c416b67

Auspicus commented Apr 27, 2026

View reviewed changes

Comment thread martin/src/config/file/tiles/reload/mbtiles.rs Outdated

Auspicus and others added 4 commits April 27, 2026 11:35

fix: add back feature gate for arguments passed to Catalog::new()

ab858f8

chore: remove extraneous dependencies

d2b9e38

chore: fix lint issue with docs .md file

f2a50cf

Auspicus and others added 2 commits April 27, 2026 17:39

chore(fmt): apply pre-commit formatting fixes

19790ba

Auspicus marked this pull request as ready for review April 27, 2026 07:42

CommanderStorm approved these changes Apr 27, 2026

View reviewed changes

CommanderStorm added 2 commits April 27, 2026 15:16

Update martin/src/bin/martin.rs

a1c51b2

Update martin/src/bin/martin.rs

16dd79f

chore(fmt): apply pre-commit formatting fixes

94b897b

CommanderStorm added the bless label Apr 27, 2026

CommanderStorm enabled auto-merge (squash) April 27, 2026 14:30

github-actions Bot removed the bless label Apr 27, 2026

chore: update blessed test snapshots across all components

6fa969a

CommanderStorm added the pre-commit.ci autofix label Apr 27, 2026

pre-commit-ci Bot removed the pre-commit.ci autofix label Apr 27, 2026

chore(fmt): apply pre-commit formatting fixes

ac4fade

CommanderStorm reviewed Apr 27, 2026

View reviewed changes

Comment thread tests/test.sh Outdated

Apply suggestion from @CommanderStorm

85c8b9e

CommanderStorm mentioned this pull request Apr 27, 2026

Automatic (catalog, cached tiles, ..) update after data change (WATCH_MODE=info) #288

Open

ci: fix typo and reference to MARTIN_BIN

1e2d97b

auto-merge was automatically disabled April 27, 2026 22:25
Head branch was pushed to by a user without write access

test: fake the 'Removed source' test output on Windows since delete _…

670a408

…should_ be impossible as SQLite does not add FILE_SHARE_DELETE capability when opening the file

CommanderStorm reviewed Apr 28, 2026

View reviewed changes

Comment thread docs/content/sources-files.md Outdated

Apply suggestion from @CommanderStorm

00c9477

CommanderStorm reviewed Apr 28, 2026

View reviewed changes

Comment thread docs/content/sources-files.md Outdated

Apply suggestion from @CommanderStorm

5bc9c16

CommanderStorm enabled auto-merge (squash) April 28, 2026 04:21

CommanderStorm merged commit c96a3be into maplibre:main Apr 28, 2026
43 checks passed

CommanderStorm mentioned this pull request Apr 28, 2026

chore: release #2741

Merged

Uh oh!

Conversation

Auspicus commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CommanderStorm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CommanderStorm Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Auspicus commented Apr 20, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CommanderStorm left a comment

Choose a reason for hiding this comment

Uh oh!

CommanderStorm Apr 27, 2026

Choose a reason for hiding this comment

Uh oh!

Auspicus Apr 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

CommanderStorm Apr 27, 2026

Choose a reason for hiding this comment

Uh oh!

Auspicus Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CommanderStorm commented Apr 27, 2026

Uh oh!

Uh oh!

Auspicus commented Apr 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Auspicus commented Apr 20, 2026 •

edited

Loading

Auspicus Apr 27, 2026 •

edited

Loading