refactor(csharp): make CloudFetch pipeline protocol-agnostic

[jadewang-db]

🥞 Stacked PR

Use this link to review incremental changes.

• stack/PECO-2788-cloudfetch-protocol-agnostic [Files changed]
  • stack/PECO-2790-statement-execution-result-fetcher [Files changed]
    • stack/PECO-2791-connection-statement-rest-execution [Files changed]
      • stack/PECO-2837-session-management [Files changed]
        • stack/PECO-2839-inline-reader [Files changed]
          • stack/PECO-2840-protocol-selection [Files changed]
            • stack/PECO-2838-statement-execution [Files changed]
              • stack/PECO-2792-feature-parity [Files changed]
                • stack/PECO-2849-extend-cloudfetch-e2e-tests [Files changed]

---

Make the CloudFetch pipeline protocol-agnostic to support both Thrift and REST API implementations, enabling code reuse and easier addition of new protocols in the future.

Changes:

• Created BaseResultFetcher abstract class extracting common fetching logic

  • Background task management, error handling, queue management
  • Template method pattern with FetchAllResultsAsync() for protocol-specific logic
  • Reduces code duplication by ~175 lines across implementations

• Refactored CloudFetchResultFetcher to extend BaseResultFetcher

  • Reduced from 401 to 314 lines (22% reduction)
  • Kept Thrift-specific logic: URL caching, expiration, refresh
  • Removed duplicate background task and error handling code

• Made IDownloadResult protocol-agnostic

  • Replaced TSparkArrowResultLink with generic properties
  • Added FileUrl, StartRowOffset, RowCount, ByteCount, ExpirationTime, HttpHeaders
  • DownloadResult.FromThriftLink() factory method for backward compatibility

• Enhanced CloudFetchDownloader to support custom HTTP headers

  • Respects IDownloadResult.HttpHeaders property
  • Enables REST API authentication without code changes

Benefits:

• Enables REST API (Statement Execution API) to reuse CloudFetch pipeline
• No behavior changes to existing Thrift implementation
• Better separation of concerns and testability

[eric-wang-1990]

I am not seeing null as a valid WaitTimeout? https://docs.databricks.com/api/workspace/statementexecution/executestatement

Also this should probably map to disposition? https://docs.databricks.com/api/workspace/statementexecution/executestatement#disposition
In the SEA wiki it says if we assign inline but result is larger than 25MB it will fail. And for EXTERNAL_LINK there is also an upper limit of 100GB.
I think in order for SEA to work like thrift, we may need a 3rd option like AUTO, which just like thrift directResult capability, it will return either inline result or cloudfetch links automatically depending on the result data size.

[eric-wang-1990]

I think overall it looks good to me, can we run this against all the tests to make sure no regression is introduced?
Also it's better if we can hold off this until we migrate all the existing PRs that is include in the arrow-adbc branch but not this driver branch, there are a handful of them like ~5ish

[eric-wang-1990]

This was changed in one of existing PR: #45

[jadewang-db]

int DefaultMemoryBufferSizeMB = 100; need change it to 100

[jadewang-db]

int DefaultMemoryBufferSizeMB = 100; need change it to 100

[eric-wang-1990]

How does this work? Is this for SEA?
For SEA does the activityTrace propogate successfully? If we need a follow up item can we add a TODO