You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This dbt package is part of Elementary, the dbt-native data observability solution for data and analytics engineers.
17
-
Set up in minutes, gain immediate visibility, detect data issues, send actionable alerts, and understand impact and root cause.
18
-
Available as self-hosted or Cloud service with premium features.
16
+
This dbt-native package powers **Elementary**, helping data and analytics engineers **detect data anomalies** and build **rich metadata tables** from their dbt runs and tests. Gain immediate visibility into data quality trend and uncover potential issues, all within dbt.
19
17
20
-
#### Table of Contents
18
+
Choose the observability tool that fits your needs:
-[Get more out of Elementary](#get-more-out-of-elementary-dbt-package)
24
-
-[Run results and dbt artifacts](#run-results-and-dbt-artifacts)
25
-
-[Data anomaly detection as dbt tests](#data-anomaly-detection-as-dbt-tests)
20
+
✅ [**Elementary Open Source**](https://docs.elementary-data.com/oss/oss-introduction) – A powerful, self-hosted tool for teams that want full control.
21
+
22
+
✅ [**Elementary Cloud Platform**](https://docs.elementary-data.com/cloud/introduction) – A fully managed, enterprise-ready solution with **automated ML-powered anomaly detection, flexible data discovery, integrated incident management, and collaboration tools**—all with minimal setup and infrastructure maintenance.
23
+
24
+
### Table of Contents
25
+
26
+
-[What's Inside the Elementary dbt Package?](#whats-inside-the-elementary-dbt-package)
27
+
-[Get more out of Elementary dbt package](#get-more-out-of-elementary-dbt-package)
28
+
-[Data Anomaly Detection & Schema changes as dbt Tests](#data-anomaly-detection--schema-changes-as-dbt-tests)
29
+
-[Elementary Tables - Run Results and dbt Artifacts](#elementary-tables---run-results-and-dbt-artifacts)
26
30
-[AI-powered data validation and unstructured data tests](#ai-powered-data-validation-and-unstructured-data-tests)
The **Elementary dbt package** is designed to enhance data observability within your dbt workflows. It includes two core components:
34
38
35
-
```yml packages.yml
36
-
packages:
37
-
- package: elementary-data/elementary
38
-
version: 0.18.0
39
-
## Docs: https://docs.elementary-data.com
40
-
```
39
+
-**Elementary Tests** – A collection of **anomaly detection tests** and other data quality checks that help identify unexpected trends, missing data, or schema changes directly within your dbt runs.
40
+
-**Metadata & Test Results Tables** – The package automatically generates and updates **metadata tables** in your data warehouse, capturing valuable information from your dbt runs and test results. These tables act as the backbone of your **observability setup**, enabling **alerts and reports** when connected to an Elementary observability platform.
41
41
42
-
2. Run `dbt deps`
42
+
## Get more out of Elementary dbt package
43
43
44
-
3. Add to your `dbt_project.yml`:
44
+
The **Elementary dbt package** helps you find anomalies in your data and build metadata tables from your dbt runs and tests—but there's even more you can do.
45
45
46
-
```yml
47
-
models:
48
-
## elementary models will be created in the schema '<your_schema>_elementary'
49
-
## for details, see docs: https://docs.elementary-data.com/
50
-
elementary:
51
-
+schema: "elementary"
52
-
```
46
+
To generate observability reports, send alerts, and govern your data quality effectively, connect your dbt package to one of the following options:
53
47
54
-
4. Run `dbt run --select elementary`
48
+
-**Elementary OSS**
49
+
-**A self-hosted, open-source CLI** that integrates seamlessly with your dbt project and the Elementary dbt package. It **enables alerting and provides the basic Elementary data observability report**, offering a comprehensive view of your dbt runs, all dbt test results, data lineage, and test coverage. It’s ideal for small teams of data and/or analytics engineers seeking a straightforward, non-collaborative setup for data observability. Quickstart [here](https://docs.elementary-data.com/oss/quickstart/quickstart-cli), and our team and community can provide great support on [Slack](https://www.elementary-data.com/community) if needed.
50
+
-**Elementary Cloud**
51
+
- A **fully managed, enterprise-ready** solution designed for **scalability and automation**. It offers automated **ML-powered anomaly detection**, flexible **data discovery**, an integrated **incident management system**, and **collaboration features.** Delivering **high value with minimal setup and infrastructure maintenance**, it's ideal for teams looking to enhance data reliability without operational overhead. To learn more, [book a demo](https://cal.com/maayansa/elementary-intro-github-package) or [start a trial](https://www.elementary-data.com/signup).
55
52
56
-
Check out the [full documentation](https://docs.elementary-data.com/).
## Data Anomaly Detection & Schema changes as dbt Tests
59
58
60
-
Elementary has 3 offerings: This dbt package, Elementary Community (OSS) and Elementary (cloud service).
59
+
**Elementary tests are configured and executed like native tests in your project!**
61
60
62
-
- **dbt package**
63
-
- For basic data monitoring and dbt artifacts collection, Elementary offers a dbt package. The package adds logging, artifacts uploading, and Elementary tests (anomaly detection and schema) to your project.
64
-
- **Elementary Community**
65
-
- An open-source CLI tool you can deploy and orchestrate to send alerts and self-host the Elementary report. Best for data and analytics engineers that require basic observability capabilities or for evaluating features without vendor approval. Our community can provide great support on [Slack](https://www.elementary-data.com/community) if needed.
66
-
- **Elementary Cloud**
67
-
- Ideal for teams monitoring mission-critical data pipelines, requiring guaranteed uptime and reliability, short-time-to-value, advanced features, collaboration, and professional support. The solution is secure by design, and requires no access to your data from cloud. To learn more, [book a demo](https://cal.com/maayansa/elementary-intro-github-package) or [start a trial](https://www.elementary-data.com/signup).
61
+
Elementary dbt tests help track and alert on schema changes as well as key metrics and metadata over time, including freshness, volume, distribution, cardinality, and more.
68
62
69
-
## Run Results and dbt artifacts
63
+
**Seamlessly configured and run like native dbt tests,** Elementary tests detect anomalies and outliers, helping you catch data issues early.
70
64
71
-
The package automatically uploads dbt artifacts and run results to tables in your data warehouse:
65
+
Example of an Elementary test config in `schema.yml`:
72
66
73
-
Run results tables:
67
+
```
74
68
75
-
- dbt_run_results
76
-
- model_run_results
77
-
- snapshot_run_results
78
-
- dbt_invocations
79
-
- elementary_test_results (all dbt test results)
69
+
models:
70
+
- name: all_events
71
+
config:
72
+
elementary:
73
+
timestamp_column: 'loaded_at'
74
+
columns:
75
+
- name: event_count
76
+
tests:
77
+
- elementary.column_anomalies:
78
+
column_anomalies:
79
+
- average
80
+
where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
81
+
anomaly_sensitivity: 2
82
+
time_bucket:
83
+
period: day
84
+
count:1
80
85
81
-
Metadata tables:
86
+
```
82
87
83
-
- dbt_models
84
-
- dbt_tests
85
-
- dbt_sources
86
-
- dbt_exposures
87
-
- dbt_metrics
88
-
- dbt_snapshots
88
+
Elementary tests include:
89
89
90
-
Here you can find [additional details about the tables](https://docs.elementary-data.com/guides/modules-overview/dbt-package).
90
+
### **Anomaly Detection Tests**
91
91
92
-
## Data anomaly detection as dbt tests
92
+
-**Volume anomalies -** Monitors the row count of your table over time per time bucket.
93
+
-**Freshness anomalies -** Monitors the freshness of your table over time, as the expected time between data updates.
94
+
-**Event freshness anomalies -** Monitors the freshness of event data over time, as the expected time it takes each event to load - that is, the time between when the event actually occurs (the **`event timestamp`**), and when it is loaded to the database (the **`update timestamp`**).
95
+
-**Dimension anomalies -** Monitors the count of rows grouped by given **`dimensions`** (columns/expressions).
96
+
-**Column anomalies -** Executes column level monitors on a certain column, with a chosen metric.
97
+
-**All columns anomalies** - Executes column level monitors and anomaly detection on all the columns of the table.
93
98
94
-
Elementary dbt tests collect metrics and metadata over time, such as freshness, volume, schema changes, distribution, cardinality, etc.
95
-
Executed as any other dbt tests, the Elementary tests alert on anomalies and outliers.
99
+
### **Schema Tests**
96
100
97
-
**Elementary tests are configured and executed like native tests in your project!**
101
+
-**Schema changes -** Alerts on a deleted table, deleted or added columns, or change of data type of a column.
102
+
-**Schema changes from baseline** - Checks for schema changes against baseline columns defined in a source’s or model’s configuration.
103
+
-**JSON schema** - Allows validating that a string column matches a given JSON schema.
104
+
-**Exposure validation test -** Detects changes in your models’ columns that break downstream exposure.
98
105
99
-
Example of Elementary test config in `properties.yml`:
106
+
Read more about the available [Elementary tests and configuration](https://docs.elementary-data.com/data-tests/introduction).
100
107
101
-
```yml
102
-
models:
103
-
- name: your_model_name
104
-
config:
105
-
elementary:
106
-
timestamp_column: updated_at
107
-
tests:
108
-
- elementary.table_anomalies
109
-
- elementary.all_columns_anomalies
110
-
```
108
+
## Elementary Tables - Run Results and dbt Artifacts
109
+
110
+
The **Elementary dbt package** automatically stores **dbt artifacts and run results** in your data warehouse, creating structured tables that provide visibility into your dbt runs and metadata.
111
+
112
+
### **Metadata Tables - dbt Artifacts**
113
+
114
+
These tables provide a comprehensive view of your dbt project structure and configurations:
115
+
116
+
-**dbt_models** – Details on all dbt models.
117
+
-**dbt_tests** – Stores information about dbt tests.
118
+
-**dbt_sources** – Tracks source tables and freshness checks.
## elementary models will be created in the schema '<your_schema>_elementary'
179
+
## for details, see docs: <https://docs.elementary-data.com/>
180
+
elementary:
181
+
+schema: "elementary"
182
+
183
+
```
184
+
185
+
4. Run `dbt run --select elementary`
186
+
187
+
Check out the [full documentation](https://docs.elementary-data.com/).
147
188
148
189
## Community & Support
149
190
@@ -154,4 +195,4 @@ The cloud service or the CLI tool read the data from these tables, send alerts a
154
195
155
196
Thank you :orange_heart: Whether it's a bug fix, new feature, or additional documentation - we greatly appreciate contributions!
156
197
157
-
Check out the [contributions guide](https://docs.elementary-data.com/general/contributions) and [open issues](https://github.com/elementary-data/elementary/issues) in the main repo.
198
+
Check out the [contributions guide](https://docs.elementary-data.com/oss/general/contributions) and [open issues](https://github.com/elementary-data/elementary/issues) in the main repo.
0 commit comments