GH-46193: [Flight][Format] Extend Flight Location URI Semantics (#46194)

### Rationale for this change
Updating the documentation in Flight.proto and Flight.rst to extend the
semantics of the allowed Flight location URIs.

### What changes are included in this PR?
Just documentation changes. Currently, none of the Arrow Flight
implementations actually implement handling of the returned URIs beyond
possibly parsing them and wrapping in a `Location` structure. It is left
to the consumer to implement the logic of whether to re-use the same
client or spin up a new client with the new location etc. to perform the
`DoGet` request against. As such, there wasn't a need to make any
code/library changes to accomodate this as part of this PR.


* GitHub Issue: #46193

---------

Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Ian Cook <[email protected]>
Co-authored-by: Raúl Cumplido <[email protected]>

Author: 4 people

docs/source/format/Flight.rst

@@ -333,6 +333,13 @@ schemes for the given transports:
 +----------------------------+--------------------------------+
 | (reuse connection)         | arrow-flight-reuse-connection: |
 +----------------------------+--------------------------------+
+| HTTP (1)                   | http: or https:                |
++----------------------------+--------------------------------+
+
+Notes:
+
+* \(1) See :ref:`flight-extended-uris` for semantics when using
+   http/https as the transport. It should be accessible via a GET request.
 
 Connection Reuse
 ----------------
@@ -360,6 +367,52 @@ string, so the obvious candidates are not compatible.  The chosen
 representation can be parsed by both implementations, as well as Go's
 ``net/url`` and Python's ``urllib.parse``.
 
+.. _flight-extended-uris:
+
+Extended Location URIs
+----------------------
+
+In addition to alternative transports, a server may also return
+URIs that reference an external service or object storage location.
+This can be useful in cases where intermediate data is cached as
+Apache Parquet files on cloud storage or is otherwise accessible
+via an HTTP service. In these scenarios, it is more efficient to be
+able to provide a URI where the client may simply download the data
+directly, rather than requiring a Flight service to read it back into
+memory and serve it from a ``DoGet`` request.
+
+To avoid the complexities of Flight clients having to implement support
+for multiple different cloud storage vendors (e.g. AWS S3, Google Cloud),
+we extend the URIs to only allow an HTTP/HTTPS URI where the client can
+perform a simple GET request to download the data. Authentication can be
+handled either by negotiating externally to the Flight protocol or by the
+server sending a presigned URL that the client can make a GET request to.
+This should be supported by all current major cloud storage vendors, meaning
+only the server needs to know the semantics of the underlying object store APIs.
+
+When using an extended location URI, the client should ignore any
+value in the ``Ticket`` field of the ``FlightEndpoint``. The
+``Ticket`` is only used for identifying data in the context of a
+Flight service, and is not needed when the client is directly
+downloading data from an external service.
+
+Clients should assume that, unless otherwise specified, the data is
+being returned using the :ref:`format-ipc` just as it would
+via a ``DoGet`` call. If the returned ``Content-Type`` header is a generic
+media type such as ``application/octet-stream``, the client should still assume
+it is an Arrow IPC stream. For other media types, such as Apache Parquet,
+the server should use the appropriate IANA Media Type that a client
+would recognize.
+
+Finally, the server may also allow the client to choose what format the
+data is returned in by respecting the ``Accept`` header in the request.
+If multiple formats are requested and supported, the choice of which to
+use is server-specific. If none of the requested content-types are
+supported, the server may respond with either 406 (Not Acceptable),
+415 (Unsupported Media Type), or successfuly respond with a different
+format that it does support, along with the correct ``Content-Type``
+header.
+
 Error Handling
 ==============
 

format/Flight.proto

@@ -426,8 +426,41 @@ message Ticket {
 }
 
 /*
- * A location where a Flight service will accept retrieval of a particular
- * stream given a ticket.
+ * A location to retrieve a particular stream from. This URI should be one of
+ * the following:
+ *  - An empty string or the string 'arrow-flight-reuse-connection://?':
+ *    indicating that the ticket can be redeemed on the service where the
+ *    ticket was generated via a DoGet request.
+ *  - A valid grpc URI (grpc://, grpc+tls://, grpc+unix://, etc.):
+ *    indicating that the ticket can be redeemed on the service at the given
+ *    URI via a DoGet request.
+ *  - A valid HTTP URI (http://, https://, etc.):
+ *    indicating that the client should perform a GET request against the
+ *    given URI to retrieve the stream. The ticket should be empty
+ *    in this case and should be ignored by the client. Cloud object storage
+ *    can be utilized by presigned URLs or mediating the auth separately and
+ *    returning the full URL (e.g. https://amzn-s3-demo-bucket.s3.us-west-2.amazonaws.com/...).
+ *
+ * We allow non-Flight URIs for the purpose of allowing Flight services to indicate that
+ * results can be downloaded in formats other than Arrow (such as Parquet) or to allow
+ * direct fetching of results from a URI to reduce excess copying and data movement.
+ * In these cases, the following conventions should be followed by servers and clients:
+ *
+ *  - Unless otherwise specified by the 'Content-Type' header of the response,
+ *    a client should assume the response is using the Arrow IPC Streaming format.
+ *    Usage of an IANA media type like 'application/octet-stream' should be assumed to
+ *    be using the Arrow IPC Streaming format.
+ *  - The server may allow the client to choose a specific response format by
+ *    specifying an 'Accept' header in the request, such as 'application/vnd.apache.parquet'
+ *    or 'application/vnd.apache.arrow.stream'. If multiple types are requested and
+ *    supported by the server, the choice of which to use is server-specific. If
+ *    none of the requested content-types are supported, the server may respond with
+ *    either 406 (Not Acceptable) or 415 (Unsupported Media Type), or successfully
+ *    respond with a different format that it does support along with the correct
+ *    'Content-Type' header.
+ *
+ * Note: new schemes may be proposed in the future to allow for more flexibility based
+ * on community requests.
  */
 message Location {
   string uri = 1;