How to resume potentially noncontiguous upload

[TheDocTrier]

Adapted from Zulip chat since I am not sure where these sorts of questions are supposed to be directed:

I am not sure how the server communicates sufficient information in response to a HEAD request that the client can resume their upload. Specifically steps 2 and 3 of "Segmented document creation with PATCH" does not provide very much detail. Presumably, the server will have a collection of byteranges uploaded so far. In response to an ordinary GET request, the client can receive a multipart/byteranges that allows the client to know exactly which portions of the upload are currently missing. But if they only send a HEAD request, the headers they receive in response would only give metadata about the total multipart/byteranges and would not include the nested headers of the multipart/byteranges which would be needed to know what data is missing. Perhaps I have not been able to put together the other parts of the draft to fill in the gaps here.

[awwright]

Hi @TheDocTrier, the mailing list is the primary discussion venue, but this issue tracker is also used.

Indeed, this is not very well specified, and worthy of clarification. The idea is that the HEAD response still contains a Content-Length field, and that tells you what size that the response would have been, had it been a GET request. So if you read a Content-Length in a HEAD response, you can assume that many bytes have been stored, and can be skipped with a subsequent PATCH. However this only applies if your initial PUT request is what created the resource, and you saw a 201 Created response. This should also be clarified.

Note Byte Range PATCH does not intend to cover all cases of resumable uploads, and any cases that are more complicated than this (e.g. resuming a POST request) cannot be handled purely with this document, but I would point you to https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumable-upload/ for these other cases.

How does that work for you?

Would you link me to the Zulip chat please? I may have missed the meeting where this came up.

[TheDocTrier]

Thanks for the reply @awwright . Here is the link to the Zulip chat https://zulip.ietf.org/#narrow/channel/223-httpapi/topic/How.20to.20resume.20upload.20with.20Byte.20Range.20PATCH . I'm not sure what meeting you're referring to; I created a new topic for my question because I thought that's what the Zulip intro video suggested I do. I think I prefer the GitHub issue tracker since it appears easier to follow the progress of multiple conversations (but I have little to no experience with Zulip specifically, so take my opinion with much salt).

Regarding the draft, thank you for linking https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumable-upload/ . I was not aware of that draft and although it handles one usage pattern for uploading files in parts (as a kind of serial stream), it explicitly mentions upfront that it does not handle parallel upload. Byte range patches on the other hand could easily support parallel upload since there is no requirement that byte ranges be contiguous. I also feel that byte range patches offer a more direct method of upload that is compatible with treating a resource's URL as the canonical location for interacting with it, whereas the resumable upload protocol provides a separate resource sink where the upload stream should be directed before it obtains a public facing URL. This also means that byte range patches could theoretically support multiple clients all modifying portions of a common resource independently, something that seems quite incompatible with the serial nature of the resumable uploads.

Regarding your comments on my original question, I have a couple concerns and maybe points of confusion.

1. It would be misleading behavior for the HEAD request directed at an incomplete resource to return a Content-Length field which only provides the number of bytes currently stored. This seems to imply that the Content-Type field would reflect the content type provided in the interrupted PUT request and by extension would likely reflect the content type given within a multipart/byteranges body within a PATCH request. However, if a GET request were subsequently sent, the server might be unable to provide a resource with the corresponding Content-Type since not all of the bytes have been uploaded yet!

   One could imagine that a JSON array was sent in a PUT request using transaction=persist and was interrupted. A GET request should not return the partially uploaded JSON file with the JSON content type because a partial JSON file is likely not a valid JSON file.

2. A key part of my original question was about "noncontiguous" uploads specifically. One can imagine that two noncontiguous byte ranges, each length 200, are included in a patch, for example [1-200] and [401-600]. A HEAD request returning a Content-Length of either 400 or 600 is likely confusing it either case. The section Applying a patch says that a server accepting sparse writes "SHOULD fill in undefined regions with zeros." I think this is a potentially dangerous suggestion since many file formats would be corrupted if sections of them are filled with zeros.

   It would be great if the server could provide a response header that indicates whether sparse writes are supported (perhaps a client could indicate its preference with a Prefer header). If sparse writes are supported by a server, that server should probably not fulfill a GET request with a content range covering a portion of the resource which does not exist. Just as returning uninitialized bytes would be invalid, returning zeros would also likely be invalid. When I read this draft, I thought it was implied or would make the most sense for my server to respond to a GET request with a multipart/byteranges that combines all of the currently uploaded byte ranges. This would prevent a client from thinking the resource was valid when it is not, and it would provide a client with the necessary information to finish uploading the resource.

   Lastly, since a HEAD request would only provide the Content-Length of the multipart/byteranges representation in my case, this would make reliably determining which bytes ranges are left to be uploaded impossible. Perhaps there should be another content type like multipart/byteranges which only includes the multipart headers? Essentially, it would be like a HEAD request for multipart representations. Or, maybe an optional parameter can be added to multipart/byteranges which allows the server to send only nested headers without their corresponding bodies.

Thank you again for taking the time to read my questions and thoughts.

[awwright]

Hi @TheDocTrier, I see what you mean now. I’m usually only on Zulip for IETF meetings.

It would be misleading behavior for the HEAD request directed at an incomplete resource to return a Content-Length field which only provides the number of bytes currently stored.

This is a good argument that the result of an HTTP operation should be atomic, so that a third party doesn’t see an incomplete operation with inconsistent data (like a truncated number, or invalid document). However, this doesn’t seem to be an issue in practice. Most Web servers I’ve looked at will happily send the partial contents of an upload-in-progress as the complete resource. This is more of a limitation of the underlying filesystem, and not of the implementations themselves. More specifically, they consider every write to disk to be a new atomic change that changes the ETag (by including the file size as part of the ETag). As you point out, this can potentially cause problems.

However, if you limit yourself to atomic operations, there’s not a good way to pick up where you left off, at least not with PATCH alone.

Regarding Content-Length specifically, it only describes the length of the response, so if the server only has 100 out of the 400 bytes, it can only send a 100-byte response.

There are only a few things you can do about this:

(1) Accept that there are resources which might be truncated and invalid according to their media type, but this is acceptable because the only party that will be interacting with them is the author, who understands this is a possibility whenever the upload is incomplete. 

The client indicates this is acceptable using transaction=persist


(2) Introduce some sort of transaction or lock system, where the server ensures that no third parties can access or modify the resource until it is in a place that the author and server deem acceptable. However, this may interfere with caching if not properly designed.

(3) Send Content-Length with the complete length (e.g. Content-Length: 400), and hold the response open until the complete contents are known. This is exactly what would be expected for a live stream, where the beginning of the media is known and should be readable right away—but not all of it will become known until later when it is recorded.


This has the problem that if the upload is lost, there’s currently no standard way to see which ranges were received by the server, and which are awaiting upload.

A new HTTP extension could fix this. You have to send some non-200, 2xx response with only the defined regions, in a way that also conveys the “complete” length of the resource. Probably identical to a 206 Partial Content response, if not that status code.



And/or, use a 1xx interim response to indicate that some portion of the upload has been committed, and can be skipped should a resumption become necessary. If you send these responses once a second then you’ll lose at most one second of upload time.


The section Applying a patch says that a server accepting sparse writes "SHOULD fill in undefined regions with zeros." I think this is a potentially dangerous suggestion since many file formats would be corrupted if sections of them are filled with zeros.

The idea here is to not disclose uninitialized content in memory or on disk, in the event that the server decides to pre-allocate the full size of the upload on disk.

There aren’t many good options here. Many file formats are also easily corrupted when truncated, as you mention earlier. E.g., truncating a JSON document from 2600 to just 26.

The only other option I see is that the server simply doesn’t respond with a 2xx, or at least not with a 200. It knows the resource exists but doesn’t have a response that can satisfy the client.

Given that both truncation and zero-filling can corrupt certain file formats, do you have a preference for how servers should handle sparse regions, or should servers reject such requests with a non-2xx status?

It would be great if the server could provide a response header that indicates whether sparse writes are supported.

I think the best way would just be for the client to opportunistically make the write, and if it's unsupported, the server can return 4xx with a list of supported operations that the client can pick from.

As for sparse resources, HTTP doesn't currently have a good notion of sparse resources. It is implicitly supported in a minimal way, in that you can start streaming a response before the end is known. Only if a region that was already sent to a client changes, is it considered a change by a cache. Data being sent for the first time is just the data becoming available.

So I see two solutions to the problem: 1xx headers to signal that data has been committed. And

I thought it was implied or would make the most sense for my server to respond to a GET request with a multipart/byteranges that combines all of the currently uploaded byte ranges. This would prevent a client from thinking the resource was valid when it is not, and it would provide a client with the necessary information to finish uploading the resource.

Yes, this would work well with a 2xx status code. There may also need to be a way for the client to signal it supports this response, otherwise, either respond with 4xx, or holding the connection open until the complete resource is filled out.

There would also have to be a way to get only the list of defined ranges, without the content, like the HEAD request I suggested, but 206 Partial Content

Finally, if I may ask, would you say that you're satisfied with the definition of the media type itself, and how it's defined? This is a really great analysis and confirms many things that I suspected, thanks!

[awwright]

Oops I'm not sure what happened to the second to last sentence there... I think I was going to say:

There would also have to be a way to get only the list of defined ranges, without the content, like the HEAD request I suggested. 206 Partial Content doesn't necessarily list the available (defined) ranges in the message fields, so I'm not sure of a solution that works without a new header or other extension to HTTP.

Investigating this a bit further, perhaps the server could return 416 (Range Not Satisfiable) if the resource is sparse, as a way to signal to the client that a complete resource cannot be transferred at this time, and must be interacted with using Range requests. I think we still need to figure out how this response would list the available ranges.

[awwright]

Hi @TheDocTrier the deadline for getting out new Internet-Drafts is in a few hours, I'm going to clarify a little bit of the language to better answer some of your questions. If you have any comments let me know as soon as possible!

I'm also interested in seeing work on noncontiguous uploads or "sparse resources", so I'd like to hear a little bit more about what you would use that for.

[awwright]

I made 0fad253 to address part of your question

[TheDocTrier]

I tried to cover everything you mentioned and my response got a little out of control so I have included it here as a Gist:

https://gist.github.com/TheDocTrier/3b704b9d110542a1271f56382da206e5

Best,
Michael

[awwright]

Hi @TheDocTrier thanks for your reply, I'll start reading through it now. Would you be able to paste as much of that here as you can, so it's on the record, for our purposes?

[TheDocTrier]

Do you mind if I break out some of the topics covered into separate issues? A lot of them feel unrelated to noncontiguous uploads or only very loosely related.