[bug] edr monitor: failed to update the sent alerts #2042
Description
Describe the bug
After upgrading to version 0.20.0 and dbt package 0.20.1 we started running into issues where the edr monitor command ends with return code 1. We run this command multiple times with a different filter (e.g. --filters tags:accounting), some of these command fail with multiple retries. But not all of them, if it is exactly 1 or no alerts that should be sent, than the command will run without errors. We also do some retries and here we observe that not for all cases it fails to update the alert status.
To Reproduce
Not really sure how to create a minimal reproduce test case, but I'll try to give as much context on the issue as I can. My model test definitions:
models:
- name: some_model
meta:
owner: [Some Owner]
data_tests:
- dbt_utils.expression_is_true:
name: assert_active_contracts_in_correct_state
meta:
description: >-
blabla
expression: state in ('active_lease', 'shortfall')
where: is_active = true and last_payment_on > current_date
tags: [data-quality, debtors]
- dbt_utils.expression_is_true:
name: assert_inactive_contracts_in_correct_state
meta:
description: >-
bladibla
expression: state in ('early_buyout_in_full', 'paid_in_full', 'shortfall')
where: >-
is_active = false and date_trunc('month', last_payment_on) != date_trunc('month', current_date)
tags: [data-quality, debtors]The test assert_active_contracts_in_correct_state is sent succesfully to teams, while assert_inactive_contracts_in_correct_state is failing.
The first try results in following logging. Note that it logs 2 sent alerts, which I think is not true. It succeeds in sending 1 alert to teams, while the other one fails. This is what I also observe in Teams, 1 alert is sent as expected, the other one isn't.
[2025-10-29, 11:33:08 UTC] {logging_mixin.py:190} INFO - Sending alert to team debtors using command: /usr/local/airflow/production_dbt_venv/bin/edr monitor --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --filters tags:debtors --teams-webhook $TEAMS_WEBHOOK --profile-target default --days-back 1 --maximum-columns-in-alert-samples 25 --report-url $REPORT_URL
[2025-10-29, 11:33:08 UTC] {custom_operator.py:87} INFO - Copying /usr/local/airflow/dags/dbt/production to /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306
[2025-10-29, 11:33:09 UTC] {custom_operator.py:108} INFO - Running command in project dir: /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306
[2025-10-29, 11:33:09 UTC] {custom_operator.py:109} INFO - /usr/local/airflow/production_dbt_venv/bin/edr monitor --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --filters tags:debtors --teams-webhook $TEAMS_WEBHOOK --profile-target default --days-back 1 --maximum-columns-in-alert-samples 25 --report-url $REPORT_URL
[2025-10-29, 11:33:12 UTC] {logging_mixin.py:190} WARNING - /usr/local/airflow/.local/lib/python3.11/site-packages/watchtower/__init__.py:430 WatchtowerWarning: Received empty message. Empty messages cannot be sent to CloudWatch Logs
[2025-10-29, 11:33:12 UTC] {custom_operator.py:117} INFO - ________ __
[2025-10-29, 11:33:12 UTC] {custom_operator.py:117} INFO - / ____/ /__ ____ ___ ___ ____ / /_____ ________ __
[2025-10-29, 11:33:12 UTC] {custom_operator.py:117} INFO - / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
[2025-10-29, 11:33:12 UTC] {custom_operator.py:117} INFO - / /___/ / __/ / / / / / __/ / / / /_/ /_/ / / / /_/ /
[2025-10-29, 11:33:12 UTC] {custom_operator.py:117} INFO - /_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/ \__, /
[2025-10-29, 11:33:12 UTC] {custom_operator.py:117} INFO - /____/
[2025-10-29, 11:33:12 UTC] {custom_operator.py:117} INFO - Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary
[2025-10-29, 11:33:12 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:33:12 — INFO — Running with edr=0.20.0
[2025-10-29, 11:33:37 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:33:37 — INFO — edr (0.20.0) and Elementary's dbt package (0.20.1) are compatible.
[2025-10-29, 11:33:38 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:33:38 — INFO — Elementary's database and schema: '"analytics.elementary"'
[2025-10-29, 11:33:38 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:33:38 — INFO — Running internal dbt run to populate alerts
[2025-10-29, 11:33:38 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:33:38 — INFO — Running dbt command run -s elementary_cli.alerts.alerts_v2 --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --target default --vars {"days_back": 1}
[2025-10-29, 11:33:58 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:33:58 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_pending_alerts", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --target default
[2025-10-29, 11:34:09 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:34:09 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_last_alert_sent_times", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --target default
[2025-10-29, 11:34:10 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:34:10 — INFO — Update skipped alerts
[2025-10-29, 11:34:33 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:34:33 — ERROR — Could not send the alert - <class 'elementary.monitor.alerts.test_alert.TestAlertModel'>.
[2025-10-29, 11:34:33 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:34:33 — ERROR — Could not send the alert - 49016812-8833-45e3-9184-428cc8bfcbf7.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54. Full alert: {"id": "49016812-8833-45e3-9184-428cc8bfcbf7.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "alert_class_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54.None.generic", "model_unique_id": "model.beequip_etl_dbt.states_on_contracts", "detected_at": "2025-10-29 11:30:14 UTC", "database_name": "analytics", "schema_name": "validations", "owners": ["Some Owner"], "tags": ["debtors", "data-quality"], "subscribers": [], "status": "fail", "suppression_interval": 0, "test_unique_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "elementary_unique_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54.None.generic", "test_name": "assert_inactive_contracts_in_correct_state", "test_display_name": "Assert Inactive Contracts In Correct State", "severity": "ERROR", "table_name": "states_on_contracts", "table_full_name": "analytics.validations.states_on_contracts", "test_type": "dbt_test", "test_sub_type": "generic", "test_sub_type_display_name": "Generic", "test_results_description": "Got 193 results, configured to fail if != 0", "test_results_query": "select\n *\nfrom (select * from analytics.validations.states_on_contracts where is_active = false and date_trunc('month', last_payment_on) != date_trunc('month', current_date)) dbt_subquery\n\nwhere not(state in ('early_buyout_in_full', 'paid_in_full', 'shortfall'))", "test_short_name": "assert_inactive_contracts_in_correct_state", "test_description": "Indien een contract in Soft4Lease inactief is volgens het amortisatieschema dan moet de status aangepast worden. Pas dit in Soft4Lease aan!", "other": {}, "test_params": {"expression": "state in ('early_buyout_in_full', 'paid_in_full', 'shortfall')", "model": "{{ get_where_subquery(ref('states_on_contracts')) }}"}, "test_rows_sample": [$MY_TEST_RESULTS], "column_name": null, "env": null}
[2025-10-29, 11:34:33 UTC] {custom_operator.py:117} INFO - Sending alerts |████████████████████████████████████████| 2/2 [100%] in 2.0s (0.99/s)
[2025-10-29, 11:34:33 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:34:33 — INFO — Update sent alerts
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_latest_invocation", "macro_args": {}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --target default
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_latest_invocation", "macro_args": {}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306', '--target', 'default']' is 9004
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_adapter_type_and_unique_id", "macro_args": {}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --target default
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_adapter_type_and_unique_id", "macro_args": {}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306', '--target', 'default']' is 468
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.data_monitoring.data_monitoring:edr (0.20.0) and Elementary's dbt package (0.20.1) are compatible.
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_elementary_database_and_schema", "macro_args": {}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --target default
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_elementary_database_and_schema", "macro_args": {}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306', '--target', 'default']' is 444
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.data_monitoring.data_monitoring:Elementary's database and schema: '"analytics.elementary"'
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.data_monitoring.alerts.data_monitoring_alerts:Running internal dbt run to populate alerts
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - INFO:elementary.clients.dbt.command_line_dbt_runner:Running dbt command run -s elementary_cli.alerts.alerts_v2 --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --target default --vars {"days_back": 1}
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - INFO:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_pending_alerts", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --target default
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_pending_alerts", "macro_args": {"days_back": 1, "type": null}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306', '--target', 'default']' is 21024028
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - INFO:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_last_alert_sent_times", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --target default
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_last_alert_sent_times", "macro_args": {"days_back": 1, "type": null}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306', '--target', 'default']' is 876
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.fetchers.alerts.alerts:Update skipped alerts
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Running dbt command run -s elementary_cli.update_alerts.update_skipped_alerts --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --target default --vars {"alert_ids": ["4fa0be26-b747-4a75-962e-cc8eb34b8a02.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "4fa0be26-b747-4a75-962e-cc8eb34b8a02.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "5178472c-0968-421f-bc9c-5ce93ff4f755.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "5178472c-0968-421f-bc9c-5ce93ff4f755.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "0a4086ef-1bca-4a05-afeb-8d238354b884.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "0a4086ef-1bca-4a05-afeb-8d238354b884.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "83b82ea1-a5ee-41e6-9581-0cd23d50f605.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "83b82ea1-a5ee-41e6-9581-0cd23d50f605.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "3166b9eb-98b7-4de3-933e-c31953aeb146.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "3166b9eb-98b7-4de3-933e-c31953aeb146.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "784a4fa2-b892-456e-91dd-3fea15338c52.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "784a4fa2-b892-456e-91dd-3fea15338c52.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "9257993f-3c40-4b53-aa07-8f43235dbb82.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "9257993f-3c40-4b53-aa07-8f43235dbb82.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "acde66c3-32cb-4772-a681-36e463b6ee0e.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "acde66c3-32cb-4772-a681-36e463b6ee0e.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "1ed89e33-9466-4d8b-8ab4-4649b3ed8b5f.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "1ed89e33-9466-4d8b-8ab4-4649b3ed8b5f.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "fd73a7d5-e573-4433-833d-10f71aac26cd.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "fd73a7d5-e573-4433-833d-10f71aac26cd.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "c8416177-f408-4b8d-b16a-c5dc111a2520.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "c8416177-f408-4b8d-b16a-c5dc111a2520.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "39ed6022-8eb2-4277-9021-c7094a0948d4.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "39ed6022-8eb2-4277-9021-c7094a0948d4.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "0219231f-24b3-497d-93b8-cbce9da098e9.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "0219231f-24b3-497d-93b8-cbce9da098e9.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "da3b11c0-8ba5-430a-ab1b-13ed94a6d2bb.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "da3b11c0-8ba5-430a-ab1b-13ed94a6d2bb.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "ebd4d7cc-0453-4b0a-85ae-63560703c4a8.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "ebd4d7cc-0453-4b0a-85ae-63560703c4a8.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "7d4e8c34-7d01-422f-852f-da3f8a287f29.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "7d4e8c34-7d01-422f-852f-da3f8a287f29.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "9ea85eec-e2a8-42bf-9c58-3bd12c285faf.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "9ea85eec-e2a8-42bf-9c58-3bd12c285faf.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "8e609135-78d4-4ebd-a0d1-55af7475c19a.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "8e609135-78d4-4ebd-a0d1-55af7475c19a.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "fa191f3f-6016-44bd-b475-a769b344728e.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "fa191f3f-6016-44bd-b475-a769b344728e.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "7006d05d-c4f8-4680-ba2d-88bddfe9b62f.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "7006d05d-c4f8-4680-ba2d-88bddfe9b62f.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "7c975375-c9a9-44d4-a3f7-24b8dac26538.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "7c975375-c9a9-44d4-a3f7-24b8dac26538.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "d622cc73-c1b7-41c9-9943-46212688c234.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "d622cc73-c1b7-41c9-9943-46212688c234.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "106dd868-174c-4834-82b7-6187566b239c.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "106dd868-174c-4834-82b7-6187566b239c.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "420e295e-6f94-4b89-813d-06b0e71bfdb5.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "420e295e-6f94-4b89-813d-06b0e71bfdb5.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "f5378a87-ff65-4ead-adfa-f71cbba0843b.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "f5378a87-ff65-4ead-adfa-f71cbba0843b.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54"]}
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Running dbt command run -s elementary_cli.update_alerts.update_skipped_alerts --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --target default --vars {"alert_ids": ["822bd8e0-4fe8-492d-9157-0294e192cbf9.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "822bd8e0-4fe8-492d-9157-0294e192cbf9.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "fc60bd74-b51c-4789-bf08-2fa6163a1832.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a", "fc60bd74-b51c-4789-bf08-2fa6163a1832.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "039becd5-a8db-4399-8e39-c21a74a23184.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "039becd5-a8db-4399-8e39-c21a74a23184.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a"]}
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - on 1: ERROR:elementary.monitor.data_monitoring.alerts.data_monitoring_alerts:Could not send the alert - <class 'elementary.monitor.alerts.test_alert.TestAlertModel'>.
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - on 2: ERROR:elementary.monitor.data_monitoring.alerts.data_monitoring_alerts:Could not send the alert - 49016812-8833-45e3-9184-428cc8bfcbf7.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54. Full alert: {"id": "49016812-8833-45e3-9184-428cc8bfcbf7.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "alert_class_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54.None.generic", "model_unique_id": "model.beequip_etl_dbt.states_on_contracts", "detected_at": "2025-10-29 11:30:14 UTC", "database_name": "analytics", "schema_name": "validations", "owners": ["Some Owner"], "tags": ["debtors", "data-quality"], "subscribers": [], "status": "fail", "suppression_interval": 0, "test_unique_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "elementary_unique_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54.None.generic", "test_name": "assert_inactive_contracts_in_correct_state", "test_display_name": "Assert Inactive Contracts In Correct State", "severity": "ERROR", "table_name": "states_on_contracts", "table_full_name": "analytics.validations.states_on_contracts", "test_type": "dbt_test", "test_sub_type": "generic", "test_sub_type_display_name": "Generic", "test_results_description": "Got 193 results, configured to fail if != 0", "test_results_query": "select\n *\nfrom (select * from analytics.validations.states_on_contracts where is_active = false and date_trunc('month', last_payment_on) != date_trunc('month', current_date)) dbt_subquery\n\nwhere not(state in ('early_buyout_in_full', 'paid_in_full', 'shortfall'))", "test_short_name": "assert_inactive_contracts_in_correct_state", "test_description": "Indien een contract in Soft4Lease inactief is volgens het amortisatieschema dan moet de status aangepast worden. Pas dit in Soft4Lease aan!", "other": {}, "test_params": {"expression": "state in ('early_buyout_in_full', 'paid_in_full', 'shortfall')", "model": "{{ get_where_subquery(ref('states_on_contracts')) }}"}, "test_rows_sample": [$MY_TEST_RESULTS], "column_name": null, "env": null}
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.fetchers.alerts.alerts:Update sent alerts
[2025-10-29, 11:34:41 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Running dbt command run -s elementary_cli.update_alerts.update_sent_alerts --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113306 --target default --vars {"alert_ids": ["49016812-8833-45e3-9184-428cc8bfcbf7.test.beequip_etl_dbt.assert_active_contracts_in_correct_state.78b03c798a"], "sent_at": "2025-10-29 11:34:33"}
[2025-10-29, 11:34:41 UTC] {custom_operator.py:133} ERROR - Command failed with exit code 1
In the retry we see that it tries to sent 1 alert, but it fails to do so when trying to update the status
[2025-10-29, 11:37:54 UTC] {logging_mixin.py:190} INFO - Sending alert to team debtors using command: /usr/local/airflow/production_dbt_venv/bin/edr monitor --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --filters tags:debtors --teams-webhook $TEAMS_WEBHOOK --profile-target default --days-back 1 --maximum-columns-in-alert-samples 25 --report-url $REPORT_URL
[2025-10-29, 11:37:54 UTC] {custom_operator.py:87} INFO - Copying /usr/local/airflow/dags/dbt/production to /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752
[2025-10-29, 11:37:55 UTC] {custom_operator.py:108} INFO - Running command in project dir: /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752
[2025-10-29, 11:37:55 UTC] {custom_operator.py:109} INFO - /usr/local/airflow/production_dbt_venv/bin/edr monitor --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --filters tags:debtors --teams-webhook $TEAMS_WEBHOOK --profile-target default --days-back 1 --maximum-columns-in-alert-samples 25 --report-url $REPORT_URL
[2025-10-29, 11:37:58 UTC] {logging_mixin.py:190} WARNING - /usr/local/airflow/.local/lib/python3.11/site-packages/watchtower/__init__.py:430 WatchtowerWarning: Received empty message. Empty messages cannot be sent to CloudWatch Logs
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO - ________ __
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO - / ____/ /__ ____ ___ ___ ____ / /_____ ________ __
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO - / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO - / /___/ / __/ / / / / / __/ / / / /_/ /_/ / / / /_/ /
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO - /_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/ \__, /
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO - /____/
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO - Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:37:58 — INFO — Running with edr=0.20.0
[2025-10-29, 11:38:17 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:17 — INFO — edr (0.20.0) and Elementary's dbt package (0.20.1) are compatible.
[2025-10-29, 11:38:17 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:17 — INFO — Elementary's database and schema: '"analytics.elementary"'
[2025-10-29, 11:38:17 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:17 — INFO — Running internal dbt run to populate alerts
[2025-10-29, 11:38:17 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:17 — INFO — Running dbt command run -s elementary_cli.alerts.alerts_v2 --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default --vars {"days_back": 1}
[2025-10-29, 11:38:27 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:27 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_pending_alerts", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:34 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:34 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_last_alert_sent_times", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:36 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:36 — INFO — Update skipped alerts
[2025-10-29, 11:38:37 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:37 — ERROR — Could not send the alert - <class 'elementary.monitor.alerts.test_alert.TestAlertModel'>.
[2025-10-29, 11:38:37 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:37 — ERROR — Could not send the alert - 49016812-8833-45e3-9184-428cc8bfcbf7.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54. Full alert: {"id": "49016812-8833-45e3-9184-428cc8bfcbf7.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "alert_class_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54.None.generic", "model_unique_id": "model.beequip_etl_dbt.states_on_contracts", "detected_at": "2025-10-29 11:30:14 UTC", "database_name": "analytics", "schema_name": "validations", "owners": ["Some Owner"], "tags": ["data-quality", "debtors"], "subscribers": [], "status": "fail", "suppression_interval": 0, "test_unique_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "elementary_unique_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54.None.generic", "test_name": "assert_inactive_contracts_in_correct_state", "test_display_name": "Assert Inactive Contracts In Correct State", "severity": "ERROR", "table_name": "states_on_contracts", "table_full_name": "analytics.validations.states_on_contracts", "test_type": "dbt_test", "test_sub_type": "generic", "test_sub_type_display_name": "Generic", "test_results_description": "Got 193 results, configured to fail if != 0", "test_results_query": "select\n *\nfrom (select * from analytics.validations.states_on_contracts where is_active = false and date_trunc('month', last_payment_on) != date_trunc('month', current_date)) dbt_subquery\n\nwhere not(state in ('early_buyout_in_full', 'paid_in_full', 'shortfall'))", "test_short_name": "assert_inactive_contracts_in_correct_state", "test_description": "Indien een contract in Soft4Lease inactief is volgens het amortisatieschema dan moet de status aangepast worden. Pas dit in Soft4Lease aan!", "other": {}, "test_params": {"expression": "state in ('early_buyout_in_full', 'paid_in_full', 'shortfall')", "model": "{{ get_where_subquery(ref('states_on_contracts')) }}"}, "test_rows_sample": [$MY_TEST_RESULTS], "column_name": null, "env": null}
[2025-10-29, 11:38:37 UTC] {custom_operator.py:117} INFO - Sending alerts |████████████████████████████████████████| 1/1 [100%] in 0.7s (1.50/s)
[2025-10-29, 11:38:37 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:37 — INFO — Update sent alerts
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_latest_invocation", "macro_args": {}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_latest_invocation", "macro_args": {}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752', '--target', 'default']' is 9005
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_adapter_type_and_unique_id", "macro_args": {}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_adapter_type_and_unique_id", "macro_args": {}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752', '--target', 'default']' is 469
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.data_monitoring.data_monitoring:edr (0.20.0) and Elementary's dbt package (0.20.1) are compatible.
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_elementary_database_and_schema", "macro_args": {}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_elementary_database_and_schema", "macro_args": {}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752', '--target', 'default']' is 445
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.data_monitoring.data_monitoring:Elementary's database and schema: '"analytics.elementary"'
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.data_monitoring.alerts.data_monitoring_alerts:Running internal dbt run to populate alerts
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.clients.dbt.command_line_dbt_runner:Running dbt command run -s elementary_cli.alerts.alerts_v2 --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default --vars {"days_back": 1}
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_pending_alerts", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_pending_alerts", "macro_args": {"days_back": 1, "type": null}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752', '--target', 'default']' is 688397
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_last_alert_sent_times", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_last_alert_sent_times", "macro_args": {"days_back": 1, "type": null}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752', '--target', 'default']' is 3189
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.fetchers.alerts.alerts:Update skipped alerts
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - on 0: ERROR:elementary.monitor.data_monitoring.alerts.data_monitoring_alerts:Could not send the alert - <class 'elementary.monitor.alerts.test_alert.TestAlertModel'>.
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - on 1: ERROR:elementary.monitor.data_monitoring.alerts.data_monitoring_alerts:Could not send the alert - 49016812-8833-45e3-9184-428cc8bfcbf7.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54. Full alert: {"id": "49016812-8833-45e3-9184-428cc8bfcbf7.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "alert_class_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54.None.generic", "model_unique_id": "model.beequip_etl_dbt.states_on_contracts", "detected_at": "2025-10-29 11:30:14 UTC", "database_name": "analytics", "schema_name": "validations", "owners": ["Some Owner"], "tags": ["data-quality", "debtors"], "subscribers": [], "status": "fail", "suppression_interval": 0, "test_unique_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "elementary_unique_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54.None.generic", "test_name": "assert_inactive_contracts_in_correct_state", "test_display_name": "Assert Inactive Contracts In Correct State", "severity": "ERROR", "table_name": "states_on_contracts", "table_full_name": "analytics.validations.states_on_contracts", "test_type": "dbt_test", "test_sub_type": "generic", "test_sub_type_display_name": "Generic", "test_results_description": "Got 193 results, configured to fail if != 0", "test_results_query": "select\n *\nfrom (select * from analytics.validations.states_on_contracts where is_active = false and date_trunc('month', last_payment_on) != date_trunc('month', current_date)) dbt_subquery\n\nwhere not(state in ('early_buyout_in_full', 'paid_in_full', 'shortfall'))", "test_short_name": "assert_inactive_contracts_in_correct_state", "test_description": "Indien een contract in Soft4Lease inactief is volgens het amortisatieschema dan moet de status aangepast worden. Pas dit in Soft4Lease aan!", "other": {}, "test_params": {"expression": "state in ('early_buyout_in_full', 'paid_in_full', 'shortfall')", "model": "{{ get_where_subquery(ref('states_on_contracts')) }}"}, "test_rows_sample": [$MY_TEST_RESULTS], "column_name": null, "env": null}
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.fetchers.alerts.alerts:Update sent alerts
[2025-10-29, 11:38:38 UTC] {custom_operator.py:133} ERROR - Command failed with exit code 1
Expected behavior
All alerts are sent to teams without any issues. Currently it looks like when we use --filter and it results in more than 1 alert to be sent, the command fails. If it is exactly 1 or no alerts that are sent, than the command will run as expected.
Screenshots
The successfully uploaded alert:
Environment (please complete the following information):
- Elementary CLI (edr) version: 0.20.0
- Elementary dbt package version: 0.20.1
- dbt version you're using: 1.9.4
- Data warehouse: Snowflake
- Infrastructure details: Production using Airflow
Would you be willing to contribute a fix for this issue?
Don't have context to fix this