-
+
-# [dbt native data observability](https://www.elementary-data.com/)
+# [dbt-native data observability](https://www.elementary-data.com/)
+
-## Get more out of Elementary dbt package
+## Data Anomaly Detection & Schema changes as dbt Tests
-Elementary has 3 offerings: This dbt package, Elementary Community (OSS) and Elementary (cloud service).
+**Elementary tests are configured and executed like native tests in your project!**
-- **dbt package**
- - For basic data monitoring and dbt artifacts collection, Elementary offers a dbt package. The package adds logging, artifacts uploading, and Elementary tests (anomaly detection and schema) to your project.
-- **Elementary Community**
- - An open-source CLI tool you can deploy and orchestrate to send alerts and self-host the Elementary report. Best for data and analytics engineers that require basic observability capabilities or for evaluating features without vendor approval. Our community can provide great support on [Slack](https://www.elementary-data.com/community) if needed.
-- **Elementary Cloud**
- - Ideal for teams monitoring mission-critical data pipelines, requiring guaranteed uptime and reliability, short-time-to-value, advanced features, collaboration, and professional support. The solution is secure by design, and requires no access to your data from cloud. To learn more, [book a demo](https://cal.com/maayansa/elementary-intro-github-package) or [start a trial](https://www.elementary-data.com/signup).
+Elementary dbt tests help track and alert on schema changes as well as key metrics and metadata over time, including freshness, volume, distribution, cardinality, and more.
-## Run Results and dbt artifacts
+**Seamlessly configured and run like native dbt tests,** Elementary tests detect anomalies and outliers, helping you catch data issues early.
-The package automatically uploads dbt artifacts and run results to tables in your data warehouse:
+Example of an Elementary test config in `schema.yml`:
-Run results tables:
+```
-- dbt_run_results
-- model_run_results
-- snapshot_run_results
-- dbt_invocations
-- elementary_test_results (all dbt test results)
+models:
+ - name: all_events
+ config:
+ elementary:
+ timestamp_column: 'loaded_at'
+ columns:
+ - name: event_count
+ tests:
+ - elementary.column_anomalies:
+ column_anomalies:
+ - average
+ where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
+ anomaly_sensitivity: 2
+ time_bucket:
+ period: day
+ count:1
-Metadata tables:
+```
-- dbt_models
-- dbt_tests
-- dbt_sources
-- dbt_exposures
-- dbt_metrics
-- dbt_snapshots
+Elementary tests include:
-Here you can find [additional details about the tables](https://docs.elementary-data.com/guides/modules-overview/dbt-package).
+### **Anomaly Detection Tests**
-## Data anomaly detection as dbt tests
+- **Volume anomalies -** Monitors the row count of your table over time per time bucket.
+- **Freshness anomalies -** Monitors the freshness of your table over time, as the expected time between data updates.
+- **Event freshness anomalies -** Monitors the freshness of event data over time, as the expected time it takes each event to load - that is, the time between when the event actually occurs (the **`event timestamp`**), and when it is loaded to the database (the **`update timestamp`**).
+- **Dimension anomalies -** Monitors the count of rows grouped by given **`dimensions`** (columns/expressions).
+- **Column anomalies -** Executes column level monitors on a certain column, with a chosen metric.
+- **All columns anomalies** - Executes column level monitors and anomaly detection on all the columns of the table.
-Elementary dbt tests collect metrics and metadata over time, such as freshness, volume, schema changes, distribution, cardinality, etc.
-Executed as any other dbt tests, the Elementary tests alert on anomalies and outliers.
+### **Schema Tests**
-**Elementary tests are configured and executed like native tests in your project!**
+- **Schema changes -** Alerts on a deleted table, deleted or added columns, or change of data type of a column.
+- **Schema changes from baseline** - Checks for schema changes against baseline columns defined in a source’s or model’s configuration.
+- **JSON schema** - Allows validating that a string column matches a given JSON schema.
+- **Exposure validation test -** Detects changes in your models’ columns that break downstream exposure.
-Example of Elementary test config in `properties.yml`:
+Read more about the available [Elementary tests and configuration](https://docs.elementary-data.com/data-tests/introduction).
-```yml
-models:
- - name: your_model_name
- config:
- elementary:
- timestamp_column: updated_at
- tests:
- - elementary.table_anomalies
- - elementary.all_columns_anomalies
-```
+## Elementary Tables - Run Results and dbt Artifacts
+
+The **Elementary dbt package** automatically stores **dbt artifacts and run results** in your data warehouse, creating structured tables that provide visibility into your dbt runs and metadata.
+
+### **Metadata Tables - dbt Artifacts**
+
+These tables provide a comprehensive view of your dbt project structure and configurations:
+
+- **dbt_models** – Details on all dbt models.
+- **dbt_tests** – Stores information about dbt tests.
+- **dbt_sources** – Tracks source tables and freshness checks.
+- **dbt_exposures** – Logs downstream data usage.
+- **dbt_metrics** – Captures dbt-defined metrics.
+- **dbt_snapshots** – Stores historical snapshot data.
+- **dbt_seeds -** Stores current metadata about seed files in the dbt project.
+- **dbt_columns** - Stores detailed information about columns across the dbt project.
+
+### **Run Results Tables**
-Read about the available [Elementary tests and configuration](https://docs.elementary-data.com/data-tests/introduction).
+These tables track execution details, test outcomes, and performance metrics from your dbt runs:
+
+- **dbt_run_results** – Captures high-level details of each dbt run.
+- **model_run_results** – Stores execution data for dbt models.
+- **snapshot_run_results** – Logs results from dbt snapshots.
+- **dbt_invocations** – Tracks each instance of dbt being run.
+- **elementary_test_results** – Consolidates all dbt test results, including Elementary anomaly tests.
+
+For a full breakdown of these tables, see the [documentation](https://docs.elementary-data.com/dbt/package-models).
## AI-powered data validation and unstructured data tests
@@ -135,15 +158,33 @@ models:
Learn more in our [AI data validations documentation](https://docs.elementary-data.com/data-tests/ai-data-tests/ai_data_validations).
-## How Elementary works?
+## Quickstart - dbt Package
-Elementary dbt package creates tables of metadata and test results in your data warehouse, as part of your dbt runs.
+1. Add to your `packages.yml`:
-The cloud service or the CLI tool read the data from these tables, send alerts and present the results in the UI.
+```
+packages:
+ - package: elementary-data/elementary
+ version: 0.18.0
+ ## Docs:
-
+```
+
+2. Run `dbt deps`
+3. Add to your `dbt_project.yml`:
+
+```
+models:
+ ## elementary models will be created in the schema '