Upgrading version 0.15.1 → 0.16.0 cause "Runtime Error: Could not find profile named 'elementary"

[phalanx-hk]

Describe the bug
Yesterday I upgrading elementary-data[bigquery] from 0.15.1 to 0.16.0, but Upgrading caused a Runtime Error.
I include below the contents of the files and logs that are considered necessary for troubleshooting.

To Reproduce
Steps to reproduce the behavior:

1. setting dependencies and relevant yaml file as below
   pyproject.toml

"dbt-core >=1.8,<1.9",
"dbt-bigquery >=1.8,<1.9",
"elementary-data[bigquery] ==0.16.0",

profiles.yml

elementary:
  target: prod
  outputs:
    prod:
      dataset: dbt_elementary
      job_execution_timeout_seconds: 300
      job_retries: 1
      location: asia-northeast1
      method: oauth
      priority: interactive
      project: {project_id}
      threads: 4
      type: bigquery

packages.yml

packages:
  - package: elementary-data/elementary
    version: 0.16.0

dbt_project.yml

models:
  ...
  elementary:
  +schema: "dbt_elementary"
  +docs:
    show: false
flags:
  # Required from dbt 1.8 and above for certain Elementary features
  require_explicit_package_overrides_for_builtin_materializations: false

2. run command

edr report --env prod --project-name {project_id}  --days-back=30

3. see error

    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-10 23:39:29 — INFO — Running with edr=0.16.0
2024-09-10 23:39:29 — INFO — Installing packages for edr internal dbt package...
2024-09-10 23:39:29 — INFO — Running dbt command --log-format json deps --project-dir {WORKING_DIR}/dbt/.venv/lib/python3.12/site-packages/elementary/monitor/dbt_project
{"data": {"exc": "Runtime Error\n  Could not find profile named 'elementary'"}, "info": {"category": "", "code": "Z002", "extra": {}, "invocation_id": "3ef899d9-d564-4170-a149-06641a6a5aac", "level": "error", "msg": "Encountered an error:\nRuntime Error\n  Could not find profile named 'elementary'", "name": "MainEncounteredError", "pid": 33465, "thread": "MainThread", "ts": "2024-09-10T14:39:31.495433Z"}}
2024-09-10 23:39:32 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Runtime Error
  Could not find profile named 'elementary'

Expected behavior
A report of the dbt project will be generated.

Environment (please complete the following information):

• Elementary CLI (edr) version: 0.16.0
• Elementary dbt package version: 0.16.0
• dbt version you're using: 1.8.6
• Data warehouse: bigquery

[geo909]

I have the exact same issue after upgrading from 0.15.1 to 0.16.1.

> dbt --version                                                                                                                                                         BI-811-dbt-sources-Fix-freshness-tests-and-alerting [3059a3f4] modified
Core:
  - installed: 1.8.6
  - latest:    1.8.6 - Up to date!

Plugins:
  - bigquery: 1.8.2 - Up to date!

Note that I also upgraded dbt from 1.7.x (dbt-core was 1.7.10 and dbt-bigquery was 1.7.6) to 1.8 at the same time.

[geo909]

Another tip, in case it helps.

I'm fixing the following:

# requirements.txt
dbt-core==1.8.6
dbt-bigquery==1.8.2

and

# packages.yml
packages
  - package: elementary-data/elementary
      version: 0.16.1

1. With python package elementary-data==0.16.1 (same problem with 0.16.0):

> edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"                                                                                                                                       main [0e491059] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-17 10:19:10 — INFO — Running with edr=0.16.1
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "c48be6d8-6481-490d-890d-0baec2937684", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 24886, "thread": "MainThread", "ts": "2024-09-17T07:19:11.602498Z"}}
2024-09-17 10:19:12 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
2024-09-17 10:19:14 — INFO — Elementary's database and schema: '"bigquery-xxxx.user_gtzanakis_dbt"'
2024-09-17 10:19:14 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "43151044-9715-4608-b6f1-a1fa68f06067", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 24886, "thread": "MainThread", "ts": "2024-09-17T07:19:15.226170Z"}}
2024-09-17 10:19:15 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 177, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 114, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.

2. With python package elementary-data==0.15.1:

> edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"                                                                                                                                       main [0e491059] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.15.1, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-17 10:24:21 — INFO — Running with edr=0.15.1
2024-09-17 10:24:21 — INFO — edr internal dbt package is not up-to-date, updating it...
2024-09-17 10:24:21 — INFO — Running dbt --log-format json deps --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:24:25 — INFO — Running with dbt=1.8.6
2024-09-17 10:24:25 — INFO — Updating lock file in file path: /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project/package-lock.yml
2024-09-17 10:24:25 — INFO — Installing dbt-labs/dbt_utils
2024-09-17 10:24:25 — INFO — Installed from version 0.8.6
2024-09-17 10:24:25 — INFO — Updated version available: 1.3.0
2024-09-17 10:24:25 — INFO — Installing elementary-data/elementary
2024-09-17 10:24:25 — INFO — Installed from version 0.15.1
2024-09-17 10:24:25 — INFO — Updated version available: 0.16.1
2024-09-17 10:24:25 — INFO — 
2024-09-17 10:24:25 — INFO — Updates available for packages: ['dbt-labs/dbt_utils', 'elementary-data/elementary']                 
Update your versions in packages.yml, then run dbt deps
2024-09-17 10:24:44 — INFO — edr (0.15.1) and Elementary's dbt package (0.15.2) are compatible.
2024-09-17 10:24:49 — INFO — Elementary's database and schema: '"bigquery-xxxx.user_gtzanakis_dbt"'
2024-09-17 10:24:49 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:24:57 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_source_freshness_results", "macro_args": {"days_back": 7, "invocations_per_test": 720}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:04 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:11 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:18 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:25 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_singular_tests", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:31 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 7, "exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:38 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_dbt_models_test_coverage", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:45 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_nodes_depends_on_nodes", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:51 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocation", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:58 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocations_data", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper

I don't know if it makes a difference but mentioning it just in case: I'm using pyenv virtualenv in ubuntu to handle my virtual environments, hopefully that's not the issue here.

[ofek1weiss]

Hey,
@phalanx-hk , is your profiles.yml file located in your project directory or in the user directory?

• if it is in the project directory, can you try adding --profiles-dir . to your command.

@geo909 from the logs you provided, i assume that in your profiles.yml file, i assume you define the following:

    keyfile: service-account-dbt.json

can you try setting the keyfile path to the absolute path of the file and see if this helps, something like

   keyfile: "/home/<user>/.../service-account-dbt.json

[geo909]

@ofek1weiss Still doesn't work:

ferryhopper:
  target: dev_user
  outputs:
    dev_user:
      dataset: user_{{ env_var('DBT_USER') }}_dbt
      job_execution_timeout_seconds: 600
      job_retries: 1
      keyfile: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"
      location: EU
      method: service-account
      priority: interactive
      project: bigquery-xxxxxx
      threads: 2
      type: bigquery

Output:

> edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"                                                                                                                                                                                                                                main [3a3a10e9] modified untracked
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-24 17:49:36 — INFO — Running with edr=0.16.0
2024-09-24 17:49:36 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Path 'ferryhopper' does not exist.
2024-09-24 17:49:36 — ERROR — Failed to parse Elementary's database and schema.
2024-09-24 17:49:36 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-24 17:49:36 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Path 'ferryhopper' does not exist.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 175, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 112, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Path 'ferryhopper' does not exist.

Tried this one too, just in case:

tzanakis@FHLT7FD7DL3:~/git/bi/dbt (*)
> edr report --project-dir "/home/tzanakis/git/bi/dbt/ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"                                                                                                                                                                                     main [3a3a10e9] untracked
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-24 17:53:23 — INFO — Running with edr=0.16.0
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 226007, "thread": "MainThread", "ts": "2024-09-24T14:53:25.257806Z"}}
2024-09-24 17:53:25 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
2024-09-24 17:53:28 — INFO — Elementary's database and schema: '"bigquery-xxxxxx.user_gtzanakis_dbt"'
2024-09-24 17:53:28 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 226007, "thread": "MainThread", "ts": "2024-09-24T14:53:28.741599Z"}}
2024-09-24 17:53:29 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 175, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 112, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.

Note that, if I change the virtual environment to switch back to 0.15.1 and change absolutely nothing else, both the above commands run without errors.

[phalanx-hk]

@ofek1weiss
Thanks for your feedback!
I try adding --profiles-dir ., it works for me.

run command:
edr report --env prod --project-name {project_id} --days-back=30 --profiles-dir .

[ofek1weiss]

@geo909 did you try both at the same time? (setting both the keyfile and the command flags to absolute paths)? or was it one at a time?

the reason i am asking is that in the logs for the second command, you can see the log No such file or directory: 'service-account-dbt.json', i would have expected to see No such file or directory: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"

[geo909]

@geo909 did you try both at the same time? (setting both the keyfile and the command flags to absolute paths)? or was it one at a time?

the reason i am asking is that in the logs for the second command, you can see the log No such file or directory: 'service-account-dbt.json', i would have expected to see No such file or directory: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"

Oops.. Apologies, that was my mistake. I was trying the full path in the wrong profile.

So, the following combination finally worked!

• keyfile: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"
• edr report --project-dir "ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"

Played around with different expressions just in case, but relative paths expressed in different ways do not work, as expected probably. For example, the following:

• keyfile: "./ferryhopper/service-account-dbt.json"
• edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"

[ofek1weiss]

@geo909 Firstly, that is great to hear :)
I'll try to figure out what change caused the keyfile issue, and see if it is reversible.
Regarding --profiles-dir, it should work with relative paths, so doing the following should work:

• keyfile: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"
• edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"
  can you please try that and let me know if it works/share logs if it does not? 🙏

[geo909]

@ofek1weiss

No, the last combination does not work. To recap:

Works

• keyfile: "/home/tzanakis/git/bi/dbt/ferryhopper/service-account-dbt.json"
• edr report --project-dir "ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"

Output:

> edr report --project-dir "ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"                                                                                                                                                        main [3a3a10e9] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-25 12:04:17 — INFO — Running with edr=0.16.0
2024-09-25 12:04:24 — INFO — edr (0.16.0) and Elementary's dbt package (0.16.1) are compatible.
2024-09-25 12:04:26 — INFO — Elementary's database and schema: '"bigquery-xxxxxx.user_gtzanakis_dbt"'
2024-09-25 12:04:26 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:32 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_source_freshness_results", "macro_args": {"days_back": 7, "invocations_per_test": 720}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:35 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:39 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:42 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:46 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_singular_tests", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:49 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 7, "exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:54 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_dbt_models_test_coverage", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:58 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_nodes_depends_on_nodes", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:05:01 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocation", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:05:04 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocations_data", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper

Does NOT work

• keyfile: "/home/tzanakis/git/bi/dbt/ferryhopper/service-account-dbt.json"
• edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"

Output:

> edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"                                                                                                                                                                                main [3a3a10e9] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-25 12:06:18 — INFO — Running with edr=0.16.0
2024-09-25 12:06:18 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Path './ferryhopper' does not exist.
2024-09-25 12:06:18 — ERROR — Failed to parse Elementary's database and schema.
2024-09-25 12:06:18 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ./ferryhopper
2024-09-25 12:06:18 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Path './ferryhopper' does not exist.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 175, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 112, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Path './ferryhopper' does not exist.

Let me know if you need to me to try anything else!

[ofek1weiss]

ohhhhhhh, you use elementary 0.16.0, in 0.16.1 it should work (it is the only difference between them)
@geo909

[geo909]

ohhhhhhh, you use elementary 0.16.0, in 0.16.1 it should work (it is the only difference between them) @geo909

Ough.. I had tried with 0.16.1 earlier (see here) with the error again but I must have switched to 0.16.0 when playing around with the versions. Apologies again!

But we're getting there. Indeed now 0.16.1 works with the following combination:

• edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"
• keyfile: "/home/tzanakis/Tmp/bi.bak/dbt/ferryhopper/service-account-dbt.json"

But if I switch the keyfile parameter in profiles.yml to simply keyfile: service-account-dbt.json it is going to fail again.

Note that dbt runs fine with dbt run --project-dir "ferryhopper" --profiles-dir "ferryhopper" with the parameter set as keyfile: service-account-dbt.json.

[geo909]

@ofek1weiss I cloned elementary's repo and played around, the issue is with this commit.

If you comment-out the line cwd=self.project_dir, my edr command works fine with keyfile: service-account-dbt.json, put it back and it breaks again.

Edit: Tried with the latest version of elementary, and it's the same if I comment out this line, this is what brakes it for me.