[BUG] metadata-service auth type returns empty token

[ericfeunekes]

Describe the bug
Once I connect to the Databricks server from the VS Code extention, it adds .databricks/databricks.env that includes a variable DATABRICKS_AUTH_TYPE=metadata-service. I think this causes problems with authentication because I get the following error in the VS Code output window:
create_and_register_databricks_globals - ValueError: default auth: metadata-service: Metadata Service returned empty token. Config: host=https://"https://***.azuredatabricks.net/", auth_type=metadata-service, cluster_id=***, metadata_service_url=***. Env: DATABRICKS_HOST, DATABRICKS_AUTH_TYPE, DATABRICKS_CLUSTER_ID, DATABRICKS_METADATA_SERVICE_URL

To Reproduce
Steps to reproduce the behavior:

1. Follow instructions to authenticate.
2. Try to connect using Databricks connect in a jupyter notebook
3. Get the error above

System information:

1. Paste the output ot the Help: About command (CMD-Shift-P).

Version: 1.83.1 (Universal)
Commit: f1b07bd25dfad64b0167beb15359ae573aecd2cc
Date: 2023-10-10T23:46:55.789Z (2 wks ago)
Electron: 25.8.4
ElectronBuildId: 24154031
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: 11.4.183.29-electron.0
OS: Darwin arm64 22.6.0

2. Databricks Extension Version
   1.2.2

[kartikgupta-db]

Hi @ericfeunekes. This error should only be affecting local notebooks. Can you confirm that other parts of the extension are affected as well? (eg running a python file)

[ericfeunekes]

Yes that seems to be true. I don't really have a python file to run that way, but I do know it can connect to and show clusters, sync files, etc.

[Createdd]

I have the same issue. Is there any way to inspect this error in more detail? Otherwise, how to solve this namespace issue?

[ericfeunekes]

Hi @ericfeunekes. This error should only be affecting local notebooks. Can you confirm that other parts of the extension are affected as well? (eg running a python file)

This also affects the cli though. Recently I used the vs code extension and then opened a new terminal outside of vs code (in iterm2). When I tried using the cli it gave me the same error.

It seems like the extension is setting a global environment variable that needs to be unset for the cli to work.

This makes the extension break more than it helps.

[pernilak]

I have a similar problem. The extension takes over my databricks CLI. In stead of using the profile in my .databrickscfg or the configurations in my databricks.yml, it sets thing up with a default cluster (that I specified in the extension) and uses metadata-service as authentication method instead of azure cli.

[cltj]

Experiencing the same. Kindly suggest a workaround for this.

[ianepreston]

I was able to work around this issue by updating the DATABRICKS_AUTH_TYPE environment variable to match my preferred authentication. So in my case that was export DATABRICKS_AUTH_TYPE=azure_cli The terraform provider docs have a list of the possible auth types.

[nilskk-t]

I still have the same issue when trying to use Databricks Connect to run a local python file on my databricks cluster. Does anyone have a solution for this. I use Extension Version 2.5.1

[ilia-db]

@nilskk-t can you provide reproduction steps? In the v2 version of the extension we no longer inject databricks env vars into terminals, so you shouldn't see issues when running CLI manually. It's still possible to encounter issues when you run noteboooks - for example if you don't restart a notebook server after changing extension auth.

[nilskk-t]

@ilia-db when I try to run a Databricks Notebook from VS Code with Databricks Connect I get the following error:

Traceback (most recent call last):
  File "c:\Users\user\.vscode\extensions\databricks.databricks-2.5.1-win32-x64\resources\python\dbconnect-bootstrap.py", line 48, in <module>
    from databricks.sdk.runtime import dbutils  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\user\Documents\repositories\project\.venv\Lib\site-packages\databricks\sdk\runtime\__init__.py", line 172, in <module>
    dbutils = RemoteDbUtils()
              ^^^^^^^^^^^^^^^
  File "c:\Users\user\Documents\repositories\project\.venv\Lib\site-packages\databricks\sdk\dbutils.py", line 194, in __init__
    self._config = Config() if not config else config
                   ^^^^^^^^
  File "c:\Users\user\Documents\repositories\project\.venv\Lib\site-packages\databricks\sdk\config.py", line 131, in __init__
    raise ValueError(message) from e
ValueError: default auth: metadata-service: Expecting value: line 1 column 1 (char 0). Config: host=https://********.azuredatabricks.net, auth_type=metadata-service, cluster_id=********, metadata_service_url=***. Env: DATABRICKS_HOST, DATABRICKS_AUTH_TYPE, DATABRICKS_CLUSTER_ID, DATABRICKS_METADATA_SERVICE_URL

I tried it with OAuth and Azure CLI authentication

[ilia-db]

Hi, this can happen it you've changed an auth in the extension, but haven't restarted your local notebook kernel. Can you try restarting and re-running?

We also show this notification about restarting kernels after you change auth, but it's easy to miss:

Does the problem goes away after you restart the kernel?

[nilskk-t]

I don't use IPython Notebooks, but normal Python files that begin with # Databricks notebook source, but I also created a normal python file that uses Databricks Connect to connect to the Databricks Cluster, that throws the same error.

[nilskk-t]

I also get this error when selecting a cluster in databricks connect:

create_and_register_databricks_globals - ValueError: default auth: metadata-service: Expecting value: line 1 column 1 (char 0). Config: host=https://adb-***.azuredatabricks.net, auth_type=metadata-service, cluster_id=***, metadata_service_url=***. Env: DATABRICKS_HOST, DATABRICKS_AUTH_TYPE, DATABRICKS_CLUSTER_ID, DATABRICKS_METADATA_SERVICE_URL
unknown - KeyError: 'spark'

The cluster that I use is a shared cluster with Unity Catalog enabled and Databricks Runtime 15.4

Also the Configuration I am using is in mode production and not development, can this have an impact?

Update: I switched to a target configuration that is in development mode and it still throws this error