Support streaming writes against HTTP backends

[whwjiang]

Remaining work:

• Write unit tests
• Figure out if we need to explicitly close the connection ("flush") on close (the file size tells curl to close the connection, but XRootD may not know the file size in which case we do need to close explicitly)

Previously, writing files to HTTP backends didn't really work. With this addition, we should be able to support a multitude of writes, including:

• 0 byte files
• Files that occupy a single XRootD write buffer length (approximately 1 MB)
• Files that occupy more than one write buffer

To achieve the last item, we have a single libcurl operation that we keep alive for the lifetime of the file. That is to say, if XRootD issues something like open() -> write() -> write() -> ... -> write() -> close(), each of the writes in the chain correspond to a single HTTP PUT call, and the connection to the other side is persisted until the file has been completely sent. This is different from our implementation for S3 PUTs, in which case each "chunk" constitutes its own HTTP PUT request.

This work was done as a precursor to supporting Globus backends in Pelican.

[rw2]

Hi @whwjiang, Justin asked me to take a look at this. Which I will soon. Until then, let me know if you need any help with the unit testing part. It's proven very useful, but isn't always super easy or intuitive to get running.

[bbockelm]

Various small nitpicks. The biggest change would be to determine whether you keep the final flag (and, if not, adjust the error conditions in Open appropriately).

[bbockelm]

Please do this as a separate logical commit (and add in the EOL).

[bbockelm]

Unclear if the final is needed since libcurl knows how large the request should be. Please give it a try without now that we've got more experience with the branch.

[bbockelm]

Switch to Log here as well.

[whwjiang]

Done

[bbockelm]

@whwjiang - can you please rebase with the various formatting fixes noted above included?

[bbockelm]

We're good! Let's go...