Use Google Auth's load_credentials_from_* Methods

[chrisguitarguy]

These methods handle services accounts as well any any other credential method. Got someone using DBT with AWS-based Auth (see #86)? This will work with AWS Auth.

Typical service account auth with a key file and oauth2 jwt assertion grant stuff still works and is covered here: https://github.com/googleapis/google-auth-library-python/blob/v2.14.1/google/auth/_default.py#L408

DBT BigQuery requires google-api-core>=2.11 which requires google-auth>=2.14.1 which supports oauth2 service acounts as well as other things.

Basically this will just let folks auth bigquery however they would auth any gcloud resources without any extra effort. And the previous versions config still works exactly the same.

resolves #86
docs not necessary, IMO? Maybe some additional examples

Problem

Right now DBT auth is limited to service account credentials, which requires a private key for JWT assertion oauth2 grants.

Solution

Instead of using the service account auth directly, use google.auth's load_credentials_from_file and load_credentials_from_dict which support the already existing usage cases as well as any other type of gcloud auth the SDKs support.

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

[chrisguitarguy]

I'd also note that get_bigquery_defaults does all this stuff already, it just happens to require a specific GOOGLE_APPLICATION_CREDENTIALS environment variable + file. But that will handle any external_account auth. This just allows it to be any file or JSON/Dict

[nathaniel-may]

notes from sync: specifically look at impact on profile.

[joewood]

Any update on this PR? External accounts in a key-file currently doesn't work. I suspect this may fix it.

[chrisguitarguy]

For what it's worth @joewood you can just stick your external account creds in GOOGLE_APPLICATION_CREDENTIALS and it'll work with dbt auth's oauth auth type.

[joewood]

Thanks @chrisguitarguy, we actually use WIP and a custom IDP so the OAUTH approach doesn't work. I'm actually running with your fork and I needed to add a new auth type "default", which does a simple:

  credentials, _ = google.auth.default(scopes=profile_credentials.scopes)

This gets around the issue entirely and uses the active WIP service account (for us at least).

[mikealfare]

Thanks for the PR @chrisguitarguy! We're in the process of updating our credential and connection management for dbt-bigquery and this looks like a solid improvement. Do you mind resolving conflicts between your branch our main? We moved a few things around, namely credentials is its own module now instead of being part of connections. But I think we could replace some custom methods in there with the solution you're suggesting. I understand that it's been quite a while since you submitted this, so I also don't mind taking this over. I would pull your commits in so that you get the credit and include you in the changelog. It's up to you. Either way, thanks for the contribution!

[chrisguitarguy]

@mikealfare sorry, I did not see your november message until now. I sorted out the conflicts (basically by re-doing this PR from the current main rather than trying to untangle the conflicts).