Introduce ConfigExtensionExt, allowing the propagation of arbitrary ConfigExtensions across network boundaries

[gabotechs]

DataFusion allows users to provide their own ConfigExtensions that get carried by the SessionConfig through the execution of a query.

https://github.com/apache/datafusion/blob/b84ddfde0689e8ecb9b71521fd95c7245d68d4ca/datafusion/common/src/config.rs#L1174-L1223

Users can thread their own ConfigExtension in order to provide additional context to their own execution nodes or other custom implementations.

---

In Distributed DataFusion, sessions are not automatically propagated across network boundaries, and any ConfigExtension set in the head stage will be lost when doing a network hop.

This PR allows propagating user-provided ConfigExtensions in a distributed context, using GRPC metadata for propagating these extensions across network boundaries.

For example, if a user has this ConfigExtension:

    extensions_options! {
        pub struct CustomExtension {
            pub foo: String, default = "".to_string()
            pub bar: usize, default = 0
            pub baz: bool, default = false
        }
    }

    impl ConfigExtension for CustomExtension {
        const PREFIX: &'static str = "custom";
    }

It can be:

1. added to the session in the head stage:

        let custom_extension = CustomExtension {
            foo: "foo".to_string(),
            bar: 1,
            baz: true,
        };
-       state.config_mut().options().extensions.insert(custom_extension); // <- this is normal datafusion
+       state.add_distributed_option_extension(custom_extension)?;

2. recursively propagated across network calls in the Arrow Flight Endpoint

        #[async_trait]
        impl SessionBuilder for CustomSessionBuilder {
            async fn session_state(&self, mut state: SessionState) -> Result<SessionState, DataFusionError> {
+               state.propagate_distributed_option_extension::<CustomExtension>()?;
                Ok(state)
            }
        }

With just the two lines above, any user-provided CustomExtension will be able to cross network boundaries and be accessible anywhere in the plan, regardless of the physical node executing the query.

[NGA-TRAN]

I did not review the details but the high level. Intuitively, this makes sense and I support this. What I am trying to figure out is where we use this. I am working on a very specific example of explain analyze and see if we will need this so it will be a use case for it. Otherwise, maybe you will help come up with a use case?

[gabotechs]

I did not review the details but the high level. Intuitively, this makes sense and I support this. What I am trying to figure out is where we use this. I am working on a very specific example of explain analyze and see if we will need this so it will be a use case for it. Otherwise, maybe you will help come up with a use case?

This satisfies the same use-cases as upstream's ConfigExtension, so for the same reason users of normal DataFusion would want ConfigExtensions in their programs, they would also want this if running in distributed mode.

One example: A JWT auth token that needs to make it's way from a end user request all the way to a leaf execution node in a distributed query with 10 stages.

[fmonjalet]

Mostly minor remarks, the design looks good to me!

[robtandy]

I totally understand the need for this now and this design looks good.