Support Scopes and Authorization Details in Workspace Client

[aravind-segu]

What changes are proposed in this pull request?

This PR introduces two new high level parameters to WorkspaceClient -> Scopes and Authorization Details

• These parameters will be propogated to OAuth Authentication Methods
• The PR Updates current OAuth Authentication Methods to take in scopes and authorization details when retrieving tokens
• Currently the databricks-cli OAuth Credential Strategy does not support scopes and authorization details as this work is still in progress
• Introduces a new runtime-oauth credential strategy
  • If scopes are specified in notebook environments, the pat token is exchanged for oauth token
• If no scopes are mentioned then traditional notebook authentication is performed
• In Default Credentials runtime-oauth now takes precedence so that it is checked first before falliing back to pat token authentication
• Token exchange API does not support all-apis due to security constraints, so runtime-native-auth is still maintained

How is this tested?

• Added Unit tests for Notebook OAuth
• Need some pointers or repro docs to E2E test all OAuth providers

E2E test for Notebook OAuth

Scopes respected correctly

Here since the token has serving.serving-endpoints, clusters.list returns an error

When added compute.clusters both methods run successfully

Fallback to Notebook PAT correctly

Token cache is used correctly and the same token is returned if its not expired

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/sdk-py

Inputs:

• PR number: 1094
• Commit SHA: 1152d5b52e9b53bbc9b8fa9d0733c8e6616af01d

Checks will be approved automatically on success.