`pmtiles extract` extended functionality - write directly to S3-style bucket, and expose `/extract` endpoint

[jtbaker]

Two feature requests for applications building on PMTiles:

1. Direct S3 output for pmtiles extract
   pmtiles extract s3://bucket/input.pmtiles s3://bucket/output.pmtiles --bbox=... --maxzoom=...
   Write output directly to S3 instead of local disk (useful in disk-constrained environments)
   Eliminates the extra pmtiles upload step

2. HTTP extract endpoint in pmtiles serve
   Expose /extract as a GET (or POST?) route that accepts source file, bbox, maxzoom, etc. as parameters, backed by a ExtractToWriter method, that writes back into the HTTP response stream.
   This would enable stateless, on-the-fly extracts useful for cases like pmtiles-offline, where small regional extracts need to be generated dynamically.

I've got some working code implementing these because I wanted to make sure it was possible to reuse the core extract algorithm - and it is, but wanted to raise an issue to discuss first to check your thoughts before opening a PR.

Thanks!

[bdon]

On 2) for large extracts it will need to be asynchronous and designing an API around creating and managing asynchronous tasks is too opinionated for this tool, so this is out of scope. you can build a separate HTTP service that embeds go-pmtiles as a library.

For 1) this is not too difficult because the current way extract works does not require rewinding the file, so it can be streamed directly to an S3 sink. What is the intended use case here, because most developer machines are not disk-constrained, are you say trying to create a 20GB extract from a 120gb planet file on an ec2 instance with 20gb of ebs storage?

[jtbaker]

What is the intended use case here, because most developer machines are not disk-constrained, are you say trying to create a 20GB extract from a 120gb planet file on an ec2 instance with 20gb of ebs storage?

yeah, I'm thinking of a case such as an ETL pipeline running as a pod on a small K8s node, that may not have a large disk attached to it, and would potentially require a CSI driver setup along with a networked volume attached to be able to handle some extracts on the larger size of things. More generally, since it seems like object storage is the common end-destination for these extracts, would be simpler to be able to do it in one step, and not need to remember to go rm $FILENAME after the pmtiles upload step.

[bdon]

The issue we ran into implementing pmtiles copy from one cloud storage to another is that the credentials now are passed through environment variables. This is not sufficient for two remotes, we would need to change the authentication to have AWS_ACCESS_KEY_ID_1, AWS_SECRET_ACCESS_KEY_1 AWS_ACCESS_KEY_ID_2, etc.

In the case where pmtiles extract s3://planet.pmtiles s3://other.pmtiles are in separate private buckets we would also need to find a way to pass credentials that works. Separating extract and upload does not have this issue but does require the time and storage to keep the file locally.

[jtbaker]

Yeah, definitely would be unfortunate to push a breaking API change with the s3 auth system on users.

OTOH, it could just be a design constraint of the system - dealing with managing multiple sets of s3 env vars for this case feels extra fiddly, and maybe a little too heavy of coupling. My understanding is that the HMAC keys are scoped by user and not constrained to a bucket - so if the same users creds had access to both separate buckets, I think it could still work with the existing auth pattern?

I feel like the bigger part of the bell curve of PMtiles users is use cases like getting the latest protomaps planet files via http, extracting a region, and uploading that to a bucket for internal usage, which I think this would work well for.

How common do you think the workflow of "extract from one private bucket to another private bucket in another account/cloud" actually is? Those cases could always fall back to the current extract + upload combination (switching the env vars in between, I guess?).