Find a way for users to thread arbitrary metadata across network calls

[gabotechs]

User's might expect certain metadata adjacent to the query to be present in their own physical nodes. With normal DataFusion, this can be done by injecting arbitrary object into the config extensions with config.setExtension, and it will be recursively propagated across nodes.

This project introduces network boundaries between different execution nodes, and therefore, any config extensions set at the head of the plan will not be automatically propagated to further nodes executed in different hosts.

One solution to this would be to allow users access to the gRPC request metadata both in ArrowFlightReadExec and in ArrowFlightEndpoint:

• users could choose to write to the gRPC metadata in ArrowFlightReadExec, duping into it a serialized version of anything they want to propagate
• users could access the gRPC metadata in the SessionBuilder trait passed to the ArrowFlightEndpoint, read anything from it, build their own objects, and inject them into the SessionContext in the normal session building methods from SessionBuilder

gRPC already has a specification for passing through metadata like this, and the API is really nice, so my proposal is to just use that.

[robtandy]

If the user is able to set extensions in the local config (meaning the one where the plans are created), then they have arranged for the SessionConfig to be aware of the code of the extension, meaning the extension was compiled with the executable. Presumably, the head node gets deployed at the same time as the worker nodes. So, couldn't the ArrowFlightEndpoint be made aware of a trait that would provide the customizations of the config?

I'm not understanding the use cases where we would want to do this dynamically.

[gabotechs]

I'm not understanding the use cases where we would want to do this dynamically.

For example, imagine a user wants to propagate a JWT authentication token all the way from the head of the plan to a leaf node that goes to some external API, and there's several network hops in between:

    user SQL request + JWT
              │
              ▼
   ┌──────────────────────┐
   │   Head of the plan   │
   └──────────┬───────────┘  
         network hop
   ┌──────────┴───────────┐
   │      Some work       │
   └──────────────────────┘  
         network hop
   ┌──────────┴───────────┐
   │      Some work       │
   └──────────────────────┘  
         network hop
   ┌──────────┴───────────┐
   │      Leaf node       │
   └──────────────────────┘  
              │
              ▼
      Some API that requires
      the JWT

[gabotechs]

#100