What is needed for the v1.0 Release

[CommanderStorm]

Important

please use threads where appropriate, rather than creating a new top level answer (unless its a new topic)

Summary of the blockers

Reworking the admin UI

As noted by @nyurik on slack:

I have been thinking about metrics, and I have realized that we might not want to publish metrics on the same endpoint at all. Instead, it might be better to listen to another port, and only if that port happens to be the same as the primary, we publish both the data and admin together... Using this thinking, perhaps all of UI might want to move to a separate port too? This way users will not be as worried about exposing internals/maputnik/... - as they will only be available via the admin port

Note

This feature was pulled from v1.0
Rationale:
After a bit of digging, I don't think the spitting of the admin items to a new port will be quite as simple:

• the watch mode would be made more complicated to realise => now 2x the different states to communicate, interaction with the lamda code is hard to do without duplicating a lot.
• metrics subsystem would need to be mostly re-implemented (we currently wrap the App, if the metrics endpoint is supposed to be on a different port, this does not work)
• testing effort is 2x ed and larger parts of this need to be touched

This is a larger scale refactoring than I currently want to do, especially since this would make the watch mode so much more difficult.

We can still do this, but

• opt in => --admin-listen-addresses defaults to --listen-addresses, i.e. NOT split
• after we have reworked how we do integration tests

---

Note

NOT blockers as can be done without breakage

• Add multiple projections support (Add multiple projections support #1750)
  Would enable us to add the COG things back though.

• Style Rendering (feat: server-side rendering of styled tiles #1754)

• web ui (Add web-based UI for Martin #1120)

• watch mode (Automatic (catalog, cached tiles, ..) update after data change (WATCH_MODE=info) #288)

• OGC standards

  This is clearly important to some users, so we should at least have a well understood path to do it. Likely simple to do, but it might be a breaking change - so better do it before v1.
  Volunteers needed to research this…

  • feat: OGC API support #2197

• Gracefully handle pmtiles data changes

  Also note that pmtiles should be auto-invalidated when the remote (http) pmtiles returns a different etag with the response (i looked at it a while ago as part of the pmtiles crate, but i don't recall if
  it was ever resolved

  • Handle backend data changes stadiamaps/pmtiles-rs#40
    Current implementation is buggy, but it is so buggy that fixing this is not breaking. changing pmtiles with a cache won't result in working MVTs

Resoved blockers

misc maintenance things

• drop the two deprecated things
• mark COG as doc-hidden and rename the feature to unstable-cog

ETAG cache control

• Add support for etag header #580

Might subtly change the public api, so perhaps we should include it, as it is fairly trivial to add.
=> maybe?

Changing the martin bin vs lib target split

Context #1805 (comment)

Progress:

• Resources
  • Styles
  • Fonts
  • Sprites
• Tiles
  • COG
  • Postgres
  • mbtiles
  • pmtiles
• Configuration

rework cache control

• at least separate the tile cache from other cache usages (pmtiles directory caching, sprites, fonts) - all of these caches should be cleanly separated, as they caused more confusion than really helped.
  Especially relevant to the tile cache because we currently have no way to invalidate it.
• Or worst case - mark cache as experimental, and disable tile cache by default.
• feat(cache): implement sprite caching #2295

[CommanderStorm]

The library question
The reason why most of our releases have been breaking is that we also have a library target.

There are possiblities to fix this that I see.
They all boil down to "lets not worry about the lib too much":

• Let versions differ between crates.io's martin lib and the martin binary
  • leave it at 0.X naming
  • bump major on cargo-semver-checks detecting a breakage
• Separate martin and martins' cargo lib into two separate crates with different versioning

  "one other thought... perhaps we may want to refactor everything before 1x - split martin into martin + core and move all the versioned reusable components into core" (by @nyurik)

• document this fatly on crates.io that martin lib does not follow semver, only the
  martin binary does. => remove cargo-semver-checks from the martin library.

[nyurik]

this one is tricky... so things to consider:

• cargo (b) install martin -- this should work as before, installing the martin binary
• cargo (b) install martin-cp and mbtiles -- we may want to have these as either separate crates, or as independent binary targets of the main martin crate. Keeping them as separate crates is a bit cleaner and easier to advertise though.
• martin-core crate may be easier to reuse - e.g. martin-cp and martin would use the core crate, while keeping the versioning separate.
• mbtiles tool has the same issues as martin binary - and actually mbtiles crate is more reusable (I even recall @pka found some bug in it that I couldn't find later, but I think its still there and should be addressed). I am less concerned with the versioning aspect of this tool though, so itt might be ok to keep the tool and the crate version in sync for now.

[CommanderStorm]

I am quite fond of the "remove the lib from martin and add martin-core instead" approach.
This would allow us to really tidy this up.

Currently this is hard to reason about.

=> martin would be the routing and arg-parsing and pass off to martin-core for the internal catalog, tile/font/style access, ...

The breakage would be that cargo add martin would no longer work.

[nyurik]

sure, but do note that some Clap arg parsing is shared between martin and martin-cp - so the martin crate might become a nearly-empty shim for a stable version

[CommanderStorm]

Yes, but we could bump martin-core independently of martin, solving this breakage...

Should I invest time into this split or do you two think the other options (/ an option I did not think of) is a better fit for us?

[CommanderStorm]

After looking at what this would take and in what order this should be done I think the following is best (if we decide to do this as I proposed)

• do larger changes to the codebases like feat: migrate logging log -> tracing #1720
• create lib and move utils, file_config and source
• (ordering not relevant) move tile-sources/resources + their config one by one
• move rest of the configuration

[CommanderStorm]

In terms of features, these are the features that are nessesary for 1.0

Mentioned by		comments
@nyurik	ETAG cache control #580	maybe?
@CommanderStorm	Style Rendering #1754	likely unrealistic, static MLN is looking like it needs more work => push to future minor release?
@nyurik	web ui #1120	lots of work to do => push to future minor release?
@nyurik	watch mode #288	can be done after 1.0
@CommanderStorm	drop the two deprecated things
@CommanderStorm	mark COG as doc-hidden and rename the feature to unstable-cog or experimental-cog

[CommanderStorm]

include etag header in the response might subtly change the public api, so perhaps we should include it, as it is fairly trivial to add

Could you explain why you think this?
I am missing a bit of context/the idea how this could break anybody.

Would this not just make things a bit more efficient in some usecases?
=> not really breaking

(I am mostly askting how to call this out in the relase notes)

[sharkAndshark]

I don't think any of these are "nessesary" for v1

Agree.

COG - i don't have an opinion on it - @sharkAndshark has been leading that effort, but from my limited understanding it is still a bit under-cooked - so I may need a bit more info to make an opinion

As it's about stability. My opinion is that COG is not a v1.0 thing. A lot work need to be done there. The support is still very basic. And I believe in no big wrong there. The code got heavy inspiration from openlayers, GeoServer, and COG/GeoTIFF/TIFF spec.

[sharkAndshark]

And let's back to the each feature's issue to discuss. It's still early to talk about v1.0? We are still 0.16 loll. Let the v1.0 happens naturally.

[nyurik]

0.16 = we had 26 versions - https://crates.io/crates/martin/versions - and I yanked one of them -- version 1.0.0-alpha.0 that was published over 3 years ago by @stepankuzmin - in other words Martin was thought to be stable even before I began a full rewrite of it ;)

[nyurik]

etag offers a way for the client to validate that the tile is still the same. If we do not return an etag, in theory there could be a difference in behavior with regards to tile refreshes. It's a minor change of course, but if it is simple to do before v1, we might as well.

[CommanderStorm]

Before we move to release 1.0, we should gain a consensus on how to configure the following to avoid this needing backwards-incompat change

• Add multiple projections support #1750

[sharkAndshark]

Actually I think multiple tile projections support is more like a v1.0 thing. User ask it (see #343 ) many times and it's not that diffcult to be done.

[nyurik]

I don't think we must have it implemented for v1.0 -- that version is just to indicate that whatever we have now is stable (-ish). We don't have to have any special features on top of it, only the things that we don't want to break right away. So if adding projection configuration can be done without changing any existing cli/config, we are ok. If on the other hand we need to totally change how we do config or CLI params or aliases or anything else - now's the time

[sharkAndshark]

I see what you mean and agree. Then it's not v1.0 as a lot changes would be done to configuration.

[CommanderStorm]

Potentially, we might want to change the defaults for CORS.

IMO, the current default is fine.

• Enabling CORS via Martin #931