Support for appends in S3 Express

[ashtuchkin]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

About a year ago, AWS added ability to append data to objects in S3 Express / Directory Buckets: https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-objects-append.html

Append is atomic, but has the limit that up to 10000 'parts' can be appended before the file needs to be rewritten. Low-level, appending is implemented by using x-amz-write-offset-bytes header, which users need to set to the current size of the file.

We've built pretty efficient system using this functionality with official AWS S3 client, but it would be nice if we could do this using object_store somehow. Is it possible?

Describe the solution you'd like

I can see two options:

1. Ideally a top-level "append" operation could be re-introduced (I found it was removed in 0.8). The reasoning for this was that only in-memory and filesystem supported it at the time. Currently there's a bunch of object stores that support it: S3 (directory buckets), Azure, Ceph, Tencent, Alibaba, etc.
2. Use Attributes / Extensions to provide the header directly. Looking at the code I haven't found a good way to do that - would be grateful if you could point me to it!

Describe alternatives you've considered

Using a different client is the only thing that comes to mind.

Additional context

--

[tustvold]

I think the challenge here is working out an API that can generalise to multiple stores.

I suspect it won't be possible to support LocalFilesystem but it would be good to be able to support the cloud providers.

GCS, and IIRC Azure, have a notion of composing objects that's also somewhat similar
#121

[rtyler]

I have given some consideration to appendable objects. Similar to @tustvold I think this might be challenging to support nicely in a cross-object store fashion, partially since all the implementations are somewhat different, and I don't know if any of them are supported in tools like azurite or localstack.

S3 supports this only for a special type of bucket, Azure appendable blobs must be created as such and then appended (😬), I don't know what the others do however.

As I mentioned in the blog post I linked above, while I think this is technically interesting, I do believe that an append API is more likely to help somebody corrupt data than not 😭

@ashtuchkin are you interested in this support for an upstream project that might be better suited to handle the responsibility or are you using object-store directly?

[ashtuchkin]

I'm using object_store directly right now but open for suggestions! My needs are pretty basic though, just reads/lists/writes and we only need S3.

What do you think about the second option I listed, is it feasible to add a custom header to put operation?

[tustvold]

Probably the best way to model this would be as a PutMode, but I'm not entirely sure what that would look like.

[timvisee]

I'm also interested in this. Appending to objects is a key building block for Qdrant serverless. We currently use a custom HTTP client for this which isn't very nice. We exclusively use S3 Express One Zone for now.

I'd be happy to add support for this. Though, then we'd need to agree on the API first. Any fresh insights on that front? I'm not familiar with the other object stores.

Also, would it be good enough to only implement S3 support first. Or is it expected to roll this out to all object stores (that support it) from the start?