File tree Expand file tree Collapse file tree 1 file changed +1
-17
lines changed
kedro-datasets/kedro_datasets/_utils Expand file tree Collapse file tree 1 file changed +1
-17
lines changed Original file line number Diff line number Diff line change @@ -19,23 +19,7 @@ def get_spark() -> Union[SparkSession, "DatabricksSession"]:
1919 from databricks .connect import DatabricksSession
2020
2121 # We can't test this as there's no Databricks test env available
22- try :
23- spark = DatabricksSession .builder .getOrCreate () # pragma: no cover
24- # this can't be narrowed down since databricks-connect throws error of Exception type
25- except Exception as e :
26- error_message = str (e )
27- if (
28- error_message
29- == "Cluster id or serverless are required but were not specified."
30- ):
31- raise type (e )(
32- "DatabricksSession is expected to behave as singleton but it didn't. "
33- "Either set up DATABRICKS_CONFIG_PROFILE or DATABRICKS_PROFILE and DATABRICKS_SERVERLESS_COMPUTE_ID "
34- "env variables in your hooks prior to using the spark session. "
35- "Read more about these variables here: "
36- "https://docs.databricks.com/aws/en/dev-tools/databricks-connect/cluster-config#config-profile-env-var"
37- ) from e
38- pass
22+ spark = DatabricksSession .builder .getOrCreate () # pragma: no cover
3923
4024 except ImportError :
4125 # For "normal" spark sessions that don't use databricks-connect
You can’t perform that action at this time.
0 commit comments