elementary-data · arbiv · May 18, 2025 · May 18, 2025 · May 18, 2025 · May 18, 2025
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -43,3 +43,9 @@ repos:
         entry: mypy --no-error-summary
         language: system
         files: ^elementary/.*\.py$
+
+  - repo: https://github.com/CoderJoshDK/precommit-mintlify-validate/
+    rev: v0.2.0
+    hooks:
+      - id: mintlify-validate
+        args: [docs]
diff --git a/docs/Dockerfile b/docs/Dockerfile
@@ -1,8 +1,7 @@
-FROM node:19
+FROM node:20.3.0
 
 WORKDIR /app
 RUN npm i -g mintlify
-RUN mintlify install
 
 EXPOSE 3000
 CMD ["mintlify", "dev"]
diff --git a/docs/_snippets/ai-generate-test.mdx b/docs/_snippets/ai-generate-test.mdx
@@ -0,0 +1,3 @@
+<Card title="Generate your anomaly test with Elementary AI" icon="wand-magic-sparkles" horizontal="true" href="https://elementary-data.com/community">
+  Let our Slack chatbot create the anomaly test you need.
+</Card>
diff --git a/docs/_snippets/alerts/alerts-configuration.mdx b/docs/_snippets/alerts/alerts-configuration.mdx
@@ -1,5 +1,5 @@
 <CardGroup cols={2}>
-  <Card title="Alerts rules" icon="cloud" iconType="solid" href="/cloud/guides/alert-rules">
+  <Card title="Alerts rules" icon="cloud" iconType="solid" href="/cloud/features/alerts-and-incidents/alert-rules">
     Use Alert Rules to distribute your alerts to the right channels.
   </Card>
   <Card title="Alerts configuration in code" icon="code" iconType="solid" href="/cloud/guides/alerts-configuration">

diff --git a/docs/_snippets/alerts/alerts-introduction.mdx b/docs/_snippets/alerts/alerts-introduction.mdx
@@ -14,6 +14,8 @@ Custom channel, suppression interval, alert filters, etc.
 <Frame>
   <div className="dark:bg-white rounded-md p-4">
     <img
+      height ="400"
+      width="400"
       src="https://res.cloudinary.com/dvmtceykh/image/upload/v1707742279/od5lminwcmktkjfh0lao.png"
       alt="Slack alert format"
     />

diff --git a/docs/_snippets/alerts/description.mdx b/docs/_snippets/alerts/description.mdx
@@ -0,0 +1,30 @@
+Elementary supports configuring description for tests that are included in alerts.
+It's recommended to add an explanation of what does it mean if this test fails, so alert will include this context.
+
+<CodeGroup>
+
+```yml test
+tests:
+  - not_null:
+    meta:
+      description: "This is the test description"
+```
+
+```yml test config block
+{{ config(
+    tags=["Tag1","Tag2"]
+    meta={
+        description: "This is the test description"
+    }
+) }}
+```
+
+```yml dbt_project.yml
+tests:
+  path:
+    subfolder:
+      +meta:
+        description: "This is the test description"
+```
+
+</CodeGroup>
diff --git a/docs/_snippets/alerts/owner.mdx b/docs/_snippets/alerts/owner.mdx
@@ -0,0 +1,59 @@
+Elementary enriches alerts with [owners for models or tests](https://docs.getdbt.com/reference/resource-configs/meta#designate-a-model-owner)).
+
+- If you want the owner to be tagged on slack use '@' and the email prefix of the slack user (@jessica.jones to tag [email protected]).
+- You can configure a single owner or a list of owners (`["@jessica.jones", "@joe.joseph"]`).
+
+<CodeGroup>
+
+```yml model
+models:
+  - name: my_model_name
+    meta:
+      owner: "@jessica.jones"
+```
+
+```yml test
+tests:
+  - not_null:
+    meta:
+      owner: ["@jessica.jones", "@joe.joseph"]
+```
+
+```yml test/model config block
+{{ config(
+    tags=["Tag1","Tag2"]
+    meta={
+        "description": "This is a description",
+        "owner": "@jessica.jones"
+    }
+) }}
+```
+
+```yml dbt_project.yml
+models/sources:
+  path:
+    subfolder:
+      +meta:
+        owner: "@jessica.jones"
+
+tests:
+  path:
+    subfolder:
+      +meta:
+        owner: "@jessica.jones"
+
+# table level:
+
+sources:
+  - name: source_name
+    database: db
+    schema: schema
+    tables:
+      - name: orders
+        meta:
+          owner: "@jessica.jones"
+
+
+```
+
+</CodeGroup>
diff --git a/docs/_snippets/alerts/subscribers.mdx b/docs/_snippets/alerts/subscribers.mdx
@@ -0,0 +1,44 @@
+If you want additional users besides the owner to be tagged on an alert, add them as subscribers.
+
+- If you want the subscriber to be tagged on slack use '@' and the email prefix of the slack user (@jessica.jones to tag [email protected]).
+- You can configure a single subscriber or a list (`["@jessica.jones", "@joe.joseph"]`).
+
+<CodeGroup>
+
+```yml model
+models:
+  - name: my_model_name
+    meta:
+      subscribers: "@jessica.jones"
+```
+
+```yml test
+tests:
+  - not_null:
+    meta:
+      subscribers: ["@jessica.jones", "@joe.joseph"]
+```
+
+```yml test/model config block
+{{ config(
+    meta={
+        "subscribers": "@jessica.jones"
+    }
+) }}
+```
+
+```yml dbt_project.yml
+models:
+  path:
+    subfolder:
+      +meta:
+        subscribers: "@jessica.jones"
+
+tests:
+  path:
+    subfolder:
+      +meta:
+        subscribers: "@jessica.jones"
+```
+
+</CodeGroup>
diff --git a/docs/_snippets/alerts/tags.mdx b/docs/_snippets/alerts/tags.mdx
@@ -0,0 +1,39 @@
+You can use [tags](https://docs.getdbt.com/reference/resource-configs/tags) to provide context to your alerts.
+
+- You can tag a group or a channel in a slack alert by adding `#channel_name` as a tag.
+- Tags are aggregated,so a test alert will include both the test and the parent model tags.
+
+<CodeGroup>
+
+```yml model
+models:
+  - name: my_model_name
+    tags: ["#marketing", "#data_ops"]
+```
+
+```yml test
+tests:
+  - not_null:
+    tags: ["#marketing", "#data_ops"]
+```
+
+```yml test/model config block
+{{ config(
+    tags=["#marketing", "#data_ops"]
+    }
+) }}
+```
+
+```yml dbt_project.yml
+models:
+  path:
+    subfolder:
+      tags: ["#marketing", "#data_ops"]
+
+tests:
+  path:
+    subfolder:
+      tags: ["#marketing", "#data_ops"]
+```
+
+</CodeGroup>
diff --git a/docs/_snippets/cli/bigquery-cli.mdx b/docs/_snippets/cli/bigquery-cli.mdx
@@ -41,3 +41,5 @@ of [BigQuery](https://docs.getdbt.com/reference/warehouse-setups/bigquery-setup)
 <Snippet file="dwh/bigquery/cli_service_account.mdx" />
 
 Add the full path of this JSON file to your connection profile under 'keyfile'.
+
+
diff --git a/docs/_snippets/cli/clickhouse-cli.mdx b/docs/_snippets/cli/clickhouse-cli.mdx
@@ -0,0 +1,35 @@
+### ClickHouse connection profile
+
+After installing Elementary's dbt package upon running `dbt deps`,
+you can generate Elementary's profile for usage with `edr` by running the following command within your project:
+
+```shell
+dbt run-operation elementary.generate_elementary_cli_profile
+```
+
+The command will print to the terminal a partially filled template of the profile that's needed for `edr` to work.
+
+```yml Clickhouse
+## CLICKHOUSE ##
+## By default, edr expects the profile name 'elementary'.      ##
+## Configure the database and schema of elementary models.     ##
+## Check where 'elementary_test_results' is to find it.        ##
+
+elementary:
+  outputs:
+    default:
+      type: clickhouse
+      host: [hostname]
+      user: [username]
+      password: [password]
+      port: [port]
+      schema: [schema name] # elementary schema, usually [schema name]_elementary
+      threads: [1 or more]
+      # sslmode: [optional, set the sslmode used to connect to the database]
+```
+
+Note: Anomaly detection is not supported for Clickhouse.
+
+We support the same format and connection methods (user password, key pair authentication, SSO) as dbt. Please refer to
+dbt's documentation of [Clickhouse profile](https://docs.getdbt.com/reference/warehouse-setups/clickhouse-setup) for
+further details.
diff --git a/docs/_snippets/cloud/ai-privacy-policy-short.mdx b/docs/_snippets/cloud/ai-privacy-policy-short.mdx
@@ -0,0 +1,20 @@
+
+### Privacy & Data Use
+
+Elementary’s AI agents are designed for secure, transparent operation, with full user control at every step. All processing runs through self-hosted models on **Amazon Bedrock**, ensuring that **no data is shared with external services**, and nothing is stored or used for model training.
+
+Everything is **opt-in** and **disabled by default**. You control which AI features are enabled, and every action an agent takes—such as opening a pull request—**requires your explicit approval**.
+
+Each agent may use the following data to provide recommendations:
+
+- **Metadata** – asset names, folders, column names, types, tags, descriptions, owners, and custom fields
+- **SQL queries** – to analyze logic, infer lineage, or suggest optimizations
+- **Test results and incidents** – including failed assertions and alert metadata
+- **Historical execution data** – query runtimes, update frequency, freshness, and resource usage
+- **User-defined policies** *(if configured)* – for testing, governance, or asset configuration
+- **Commit and pull request history** *(when applicable)* – to connect changes with incidents
+- **Chat input** *(if applicable)* – prompts used during the session only, not stored or logged
+
+Elementary does **not collect or share any sensitive data**, and you can review and align our AI use with your organization’s governance policies at any time.
+
+For full details, see our [AI Privacy Policy](/cloud/ai-privacy).
diff --git a/docs/_snippets/cloud/features.mdx b/docs/_snippets/cloud/features.mdx
@@ -0,0 +1,81 @@
+
+### Detection & Coverage
+
+Elementary integrates powerful anomaly detection and dbt tests into a unified detection strategy.
+Effective detection of data issues requires a comprehensive approach,
+including both pipeline and data monitoring, validation tests,
+anomaly detection for unexpected behavior, and a single interface to manage it all at scale.
+
+<CardGroup cols={2}>
+  <Card title="Anomaly Detection" href="/cloud/features/anomaly-detection/monitors-overview">
+    ML-powered monitors automatically detect data quality issues.
+    Out-of-the-box for volume and freshness, and opt-in for data quality metrics.
+  </Card>
+  <Card title="Data Tests" href="/cloud/features/data-tests/data-tests-overview">
+    Validate data and track the results of dbt tests, dbt packages tests (dbt-utils, dbt-expectations, elementary) and custom SQL tests.  
+  </Card>
+  <Card title="Schema Validation" href="/cloud/features/data-tests/schema-validation-test">
+    Validate there are no breaking changes in tables schema, JSON schema, and downstream exposures such as dashboards.  
+  </Card>
+  <Card title="Runs & Performance" href="/cloud/features/performance-monitoring/performance-monitoring">
+    Track failures and runs of jobs, models, and tests overtime.
+    Pipeline failures and performance issues can cause data incidents, and create unneceserry costs.
+  </Card>
+  <Card title="Code & UI Configuration" href="/cloud/features/data-tests/data-tests-overview#test-configuration">
+    Configure Elementary in code, or via the UI for non-technical users or for adding tests in bulk.
+    The platform opens PRs to your repo, saving hours of tedious YAML edits.
+  </Card>
+  <Card title="Test Recommendations">
+    Coming soon!
+  </Card>
+</CardGroup>
+
+### Triage & Response
+
+Detecting issues is just the first step to achieve data reliability.
+Elementary offers tools to create an effective response plan, for faster recovery.
+This includes investigating the root cause and impact of issues, communicating issues to the relevant people, assigning owners to fix issues, keeping track of open incidents and more.
+
+<CardGroup cols={2}>
+  <Card title="Data Lineage" href="/cloud/features/data-lineage/lineage">
+    Column-level lineage that spans through sources, models and BI tools, enriched with monitoring results. Enables granular root cause and impact analysis. 
+  </Card>
+  <Card title="Ownership" href="/cloud/best-practices/governance-for-observability">
+    Define clear ownership of data assets and enable owners to be informed and accountable for the health and status of their data.   
+  </Card>
+  <Card title="Actionable Alerts" href="/cloud/features/alerts-and-incidents/alerts-and-incidents-overview">
+    Distribute highly configurable alerts to different channels and integrations. 
+    Automatically tag owners, and enable setting status and assigns at the alert level.  
+  </Card>
+  <Card title="Automated Grouping" href="/cloud/features/alerts-and-incidents/incidents">
+    Different failures related to the same issue are grouped automatically to a single incident.
+    This accelerates triage and response, and reduces alerts fautigue.  
+  </Card>
+  <Card title="Incident Management" href="/cloud/features/alerts-and-incidents/incident-management">
+    Manage all open incidents in a single interface, with a clear view of status and assignees.
+    Track historical incidents and high-level incidents metrics. 
+  </Card>
+</CardGroup>
+
+### Collaboration & Communication
+
+The data team doesn’t live in a silo - you have many stakeholders.
+The only way to improve data trust is by bringing in more team members, users and stakeholders to the data health process.
+Elementary fosters collaboration by allowing you to easily share and communicate the status of issues,
+the overall health of the data platform and progress made to improve it with the broader organization.
+
+<CardGroup cols={2}>
+  <Card title="Overview Dashboard" href="/cloud/features/collaboration-and-communication/data-observability-dashboard">
+    Up to date dashboard with current status and trends of data issues. 
+    Share the dashboard with others, enable them to slice results and stay informed.
+  </Card>
+  <Card title="Assets Tags" href="/cloud/best-practices/governance-for-observability">
+    Enable effective collaboration and communication by grouping related data assets and tests by business domains, data products, priority, etc.   
+  </Card>
+  <Card title="Data Catalog" href="/cloud/features/collaboration-and-communication/catalog">
+    Search and explore your datasets information - descriptions, columns, column descriptions, compiled code, datasets health and more.
+  </Card>
+  <Card title="Data Health Score" href="/cloud/features/collaboration-and-communication/data-health">
+    See the Data Health scores of all your datasets by domain and share with stakeholders.
+  </Card>
+</CardGroup>
diff --git a/docs/_snippets/cloud/features/alerts-and-incidents/alert-types.mdx b/docs/_snippets/cloud/features/alerts-and-incidents/alert-types.mdx
@@ -0,0 +1,7 @@
+Elementary can be configured to send alerts on:
+
+- Model run failures
+- Failures and/or warnings of dbt tests (including Elementary dbt package and other packages)
+- Failures and/or warnings Elementary Anomaly Detection monitors
+- Failures and/or warning of custom SQL tests
+- dbt source freshness failures
diff --git a/docs/_snippets/cloud/features/anomaly-detection/all-anomalies-configuration.mdx b/docs/_snippets/cloud/features/anomaly-detection/all-anomalies-configuration.mdx
@@ -0,0 +1,2 @@
+- **Severity** - Should a failure be considered a warning or a failure. Default is warning. 
+- **Test metadata** - Add metadata such as tags and owner to the test.
diff --git a/docs/_snippets/cloud/features/anomaly-detection/automated-monitors-cards.mdx b/docs/_snippets/cloud/features/anomaly-detection/automated-monitors-cards.mdx
@@ -0,0 +1,10 @@
+<CardGroup cols={2}>
+<Card title="Automated Freshness" href="/cloud/features/anomaly-detection/automated-freshness">
+  Monitors updates to tables and how frequently a table is updated, 
+  and fails if there is an unexpected delay. 
+</Card>
+<Card title="Automated Volume" href="/cloud/features/anomaly-detection/automated-volume">
+  Monitors how many rows were added or removed to a table on each update,
+  and fails if there is an unexpected drop or spike in rows. 
+</Card>
+</CardGroup>
diff --git a/docs/_snippets/cloud/features/anomaly-detection/automated-monitors-intro.mdx b/docs/_snippets/cloud/features/anomaly-detection/automated-monitors-intro.mdx
@@ -0,0 +1,5 @@
+Out-of-the-box ML-powered monitoring for freshness and volume issues on all production tables.
+The automated monitors feature provides broad coverage and detection of critical pipeline issues, without any configuration effort.
+
+These monitors track updates to tables, and will detect data delays, incomplete updates, and significant volume changes.
+Additionally, there will be no increase in compute costs as the monitors leverage only warehouse metadata (e.g. information schema, query history).
Original file line number	Diff line number	Diff line change
Expand Up		@@ -41,3 +41,5 @@ of [BigQuery](https://docs.getdbt.com/reference/warehouse-setups/bigquery-setup)
		<Snippet file="dwh/bigquery/cli_service_account.mdx" />

		Add the full path of this JSON file to your connection profile under 'keyfile'.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		- Severity - Should a failure be considered a warning or a failure. Default is warning.
		- Test metadata - Add metadata such as tags and owner to the test.