chore(csharp/src/Drivers/Apache): Cleanup HiveServer2-based code with shared Thrift request/response interfaces

[birschick-bq]

Refactor API to improve handling of request and responses to simplify number of overloads.
Refactor API to send the IResponse to the Reader (IArrowArrayStream).

• The Stream/Reader is now responsible to close the operation.
• The Statement is no longer responsible for keeping a singleton instance of the (most recent) response.

Replaces #2797

[birschick-bq]

@CurtHagenlocher - Resolved the merge conflicts. Added a check to ensure the HiveServer2Statement doesn't get re-used as that would change the Response object and make it inconsistent for the QueryResult/Stream.

@jadewang-db Can you run the E2E test against a Databrick system to confirm no regressions?

[CurtHagenlocher]

Thanks! I really like the improved state management here. Please consider a few small changes as described.

[toddmeng-db]

small nit, I wonder if this logic should go in DatabricksCompositeReader

[birschick-bq]

@toddmeng-db - When you say "this logic" are you referring to the CloseOperation call in the Dispose method?

Both BaseDatabricksReader and DatabricksCompositeReader inherit from TracingReader. So I've coded the CloseOperation call in both of their Dispose methods.

Am I missing something?

[birschick-bq]

@toddmeng-db - If you could run the E2E tests against a Databricks server to confirm I haven't caused any regressions. If you could check that there are no open operations after correct Dispose of streams - that would be great, too.

[toddmeng-db]

Oh, sorry. DatabricksCompositeReader will end up calling duplicate CloseOperation during BaseDatabricksReader.Dispose() (through activeReader.dispose()), do we want to avoid this?

[birschick-bq]

Ah. Good catch. I didn't see the containment of BaseDatabricksReading inside DatabricksCompositeReader.
Thanks!

[birschick-bq]

Removed the duplicate call to CloseOperation in DatabricksCompositeReader. It should be in the BaseDatabricksReader so that inheritors get the benefit.

[toddmeng-db]

Actually, it might be better if it's in DatabricksCompositeReader? DatabricksCompositeReader doesn't inherit from BaseDatabricksReader (sorry I know that's a bit confusing, probably needs a rename), and better represents the stream's lifecycle. It's possible the BaseDatabricksReader (activeReader) owned by DatabricksCompositeReader is not yet initialized when DatabricksCompositeReader is initialized.

Currently this is how they relate:
DatabricksCompositeReader is created first. If there is directresults, we initialize the BaseDatabricksReader. If there is no directresults, we wait for the first FetchResultsResponse and inspect the result to determine which BaseDatabricksReader we want to use

[birschick-bq]

@toddmeng-db - Added a "stateful" close operation on the BaseDatabricksReader. Let me know if this is too complicated a solution.

[toddmeng-db]

Tests look good to me

[birschick-bq]

Thanks for confirming!

[CurtHagenlocher]

Thanks!