Databricks Driver: Export Bucket On GCS

[qiao-x]

Hi, team
I've tested the create table using statement on Databricks-GCP and it succeeds to export csv to a bucket on GCS, so i'm wondering if it's feasible to implement the GCS export feature based on this statement and the extract files function here

proof

[qiao-x]

Hi @KSDaemon any suggestions?

[KSDaemon]

Hey!
You need to:

• add support for GCS in unload() of Databricks Driver here
• add support for GCS in getCsvFiles of Databricks Driver here
• Note, there is already createExternalTableFromSql() implementation in Databricks Driver which might require some alignment
• You can have a look at Snowflake driver as example.

[qiao-x]

@KSDaemon good news, i've completed the poc in our gcp-databricks env with export bucket feature. I will then create a MR for this and request a review from you guys.

But i met some issues when implementing this solution
When i fork from v1.2.27, and simply build databricks-jdbc driver without any code modification and then put dist/src into the container, everything works fine, but things change when I fork from master(i believe a ODBC driver is introduced there). Even i modify nothing and just replace the dist/src with my local built version. It shows an error when any queries are triggered:

Error: EISDIR: illegal operation on a directory, open '/cube/node_modules/@cubejs-backend/databricks-jdbc-driver/dist/download/META-INF/'\n    at DatabricksDriver.testConnection (/cube/node_modules/@cubejs-backend/jdbc-driver/src/JDBCDriver.ts:221:13)\n    at /cube/node_modules/@cubejs-backend/server-core/src/core/server.ts:621:15\n    at Object.query (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryCache.ts:568:26)\n    at QueryQueue.processQuery (/cube/node_modules/@cubejs-backend/query-orchestrator/src/orchestrator/QueryQueue.js:839:25)

[KSDaemon]

Cool!

Rgd error - interesting... need to look

[qiao-x]

Hi@KSDaemon
I've create a PR #9407 Could you review this and instruct me on how to add UTs?

[KSDaemon]

@qiao-x That's awesome! Sure, I'll have a look!

[igorlukanin]

Thanks for the contribution @qiao-x! This has been released in v1.3.0: https://cube.dev/blog/cube-core-v1-3-performance-improvements-and-upgrades#new-in-data-source-support