Add HTTP Range Requests to BDASL spec format #81
Conversation
darobin
left a comment
darobin
left a comment
There was a problem hiding this comment.
Thanks for this @b5. It's good stuff. I think that the primary change that it needs is a bit of massaging to make it implementable by people who have less context. If you want to do that please go ahead, but otherwise I'm happy to do it too, I think I have all the info I need to proceed. LMK!
| </section> | ||
| <section> | ||
| <h2>Streaming Verification</h2> | ||
| <p> |
There was a problem hiding this comment.
I think it would be good to have a bit more context here to explain why people would like to do this. I wouldn't expect people to know about trusted and trustless fetching, that kind of thing.
| <p> | ||
| Streaming verification rounds to the nearest kilobyte for | ||
| verification. For more details and a reference implementation, see | ||
| [[iroh-blobs]]. |
There was a problem hiding this comment.
Ideally, this here spec should have all the details it needs for black box reproducibility, other than references to other specs that have the same property. I'm guessing that you don't need Iroh Blobs to understand what to do, it's just a convenient reference?
There was a problem hiding this comment.
.... kinda. The thing that's important about iroh blobs is the spec on how to actually mix data and hashes in the byte stream. I could include that here, but it's extremely low level, which didn't really feel like it kept with the altitude of the other DASL specs. So I see two options:
- we could bring in the iroh-blobs wire format fully: describing how to intersperse verification hashes with principal data.
- we could describe at a high level what a protocol would need to implement, and leave it at that.
I think doing number 1 would be far more useful: different implementations would be able to interop. But want to make sure that level of detail is appropriate.
There was a problem hiding this comment.
To me the way data and hashes mix should live outside of the spec. It's an implementation detail. Though having it well document for interoperability would be great.
I guess there could be implementations of this spec that would do things differently. E.g. you could send the bytes needed for verification first and the data later. Or you connect to a data server and a verification server separately and mux them together in the client on the fly.
| Both <var>start</var> and <var>end</var> values are optional, and when missing indicate | ||
| the first and late byte, respectively. | ||
| </p> | ||
| <p> |
There was a problem hiding this comment.
This paragraph needs a little love to be understandable by people outside the IPFS bubble.
There was a problem hiding this comment.
i would say we merge it as-is and do the translation for normals in an editorial second pass/PR, so that there's something up sooner in case anyone it's blocking anyone that this is currently unknown/theoretical
| <section> | ||
| <h3>Fufilling Requests</h3> | ||
| <p> | ||
| For a given normalized byte range |
There was a problem hiding this comment.
What makes a byte range normalized?
One thing that's not clear here is who is doing the work? What piece of software is implementing this such that we expect it to comply?
It would probably help to expand this to describe the process. This is a client component that is handed a range request with start & end bytes. It modified those ranges according to a named algorithm. Then it makes the HTTP range request with that modified range. Then, etc. It helps to assume less context.
There was a problem hiding this comment.
this part maybe worth expanding on (at least by reference to, no idea, this???) before merging this draft, to contradict myself above.
| </ul> | ||
| </section> | ||
| <section> | ||
| <h3>Example</h3> |
There was a problem hiding this comment.
I think it would help to work the example out more, for people who don't already know what this is about.
There was a problem hiding this comment.
in particular the reference to "the" streaming protocol and 1024. isn't that chunking multiple specific to the streaming protocol? is there only one? how do client and server negotiate which of each they're gonna use?
There was a problem hiding this comment.
That's a matter of taste, so feel free to ignore. I like examples where numbers that represent something different are ideally different. In this case it would mean that both resulting slices have a different size. So no one wonders: why are the slices of both chunks exactly 512 bytes in size?
I would push the end to something like 1600.
|
|
||
| ## Developing | ||
|
|
||
| Be **particularly** careful not to edit the `.html` files but rather the `.src.html` |
There was a problem hiding this comment.
SMDH i should've PRd this in after the first 3 times i did this
| <p> | ||
| [[rfc9110]] defines HTTP range requests for fetching a single | ||
| contiguous set of bytes from a larger source held by a server. | ||
| Range requests use a Range header: <code>Range: bytes={start}-{end}</code> |
There was a problem hiding this comment.
i assume {start} and {end} are just regular old ints that map to byte#s from 0? prolly doesn't need saying but i tend to err on the side of obnoxiously explicit.
There was a problem hiding this comment.
yes exactly, I can call that out more explicitly
4 participants
... and dev setup instructions.
Some of my editor has borked the formatting of a few files, I've tried to keep the damage to a minimum