elementary-data · Maayan-s · Mar 20, 2024 · Mar 20, 2024 · Mar 20, 2024 · Mar 21, 2024
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -43,3 +43,9 @@ repos:
         entry: mypy --no-error-summary
         language: system
         files: ^elementary/.*\.py$
+
+  - repo: https://github.com/CoderJoshDK/precommit-mintlify-validate/
+    rev: v0.2.0
+    hooks:
+      - id: mintlify-validate
+        args: [docs]
diff --git a/docs/Dockerfile b/docs/Dockerfile
@@ -1,8 +1,7 @@
-FROM node:19
+FROM node:20.3.0
 
 WORKDIR /app
 RUN npm i -g mintlify
-RUN mintlify install
 
 EXPOSE 3000
 CMD ["mintlify", "dev"]
diff --git a/docs/_snippets/alerts/alerts-configuration.mdx b/docs/_snippets/alerts/alerts-configuration.mdx
@@ -1,5 +1,5 @@
 <CardGroup cols={2}>
-  <Card title="Alerts rules" icon="cloud" iconType="solid" href="/cloud/guides/alert-rules">
+  <Card title="Alerts rules" icon="cloud" iconType="solid" href="/features/alerts-and-incidents/alert-rules">
     Use Alert Rules to distribute your alerts to the right channels.
   </Card>
   <Card title="Alerts configuration in code" icon="code" iconType="solid" href="/cloud/guides/alerts-configuration">

diff --git a/docs/_snippets/alerts/description.mdx b/docs/_snippets/alerts/description.mdx
@@ -0,0 +1,30 @@
+Elementary supports configuring description for tests that are included in alerts.
+It's recommended to add an explanation of what does it mean if this test fails, so alert will include this context.
+
+<CodeGroup>
+
+```yml test
+tests:
+  - not_null:
+    meta:
+      description: "This is the test description"
+```
+
+```yml test config block
+{{ config(
+    tags=["Tag1","Tag2"]
+    meta={
+        description: "This is the test description"
+    }
+) }}
+```
+
+```yml dbt_project.yml
+tests:
+  path:
+    subfolder:
+      +meta:
+        description: "This is the test description"
+```
+
+</CodeGroup>
diff --git a/docs/_snippets/alerts/owner.mdx b/docs/_snippets/alerts/owner.mdx
@@ -0,0 +1,46 @@
+Elementary enriches alerts with [owners for models or tests](https://docs.getdbt.com/reference/resource-configs/meta#designate-a-model-owner)).
+
+- If you want the owner to be tagged on slack use '@' and the email prefix of the slack user (@jessica.jones to tag [email protected]).
+- You can configure a single owner or a list of owners (`["@jessica.jones", "@joe.joseph"]`).
+
+<CodeGroup>
+
+```yml model
+models:
+  - name: my_model_name
+    meta:
+      owner: "@jessica.jones"
+```
+
+```yml test
+tests:
+  - not_null:
+    meta:
+      owner: ["@jessica.jones", "@joe.joseph"]
+```
+
+```yml test/model config block
+{{ config(
+    tags=["Tag1","Tag2"]
+    meta={
+        "description": "This is a description",
+        "owner": "@jessica.jones"
+    }
+) }}
+```
+
+```yml dbt_project.yml
+models:
+  path:
+    subfolder:
+      +meta:
+        owner: "@jessica.jones"
+
+tests:
+  path:
+    subfolder:
+      +meta:
+        owner: "@jessica.jones"
+```
+
+</CodeGroup>
diff --git a/docs/_snippets/alerts/subscribers.mdx b/docs/_snippets/alerts/subscribers.mdx
@@ -0,0 +1,44 @@
+If you want additional users besides the owner to be tagged on an alert, add them as subscribers.
+
+- If you want the subscriber to be tagged on slack use '@' and the email prefix of the slack user (@jessica.jones to tag [email protected]).
+- You can configure a single subscriber or a list (`["@jessica.jones", "@joe.joseph"]`).
+
+<CodeGroup>
+
+```yml model
+models:
+  - name: my_model_name
+    meta:
+      subscribers: "@jessica.jones"
+```
+
+```yml test
+tests:
+  - not_null:
+    meta:
+      subscribers: ["@jessica.jones", "@joe.joseph"]
+```
+
+```yml test/model config block
+{{ config(
+    meta={
+        "subscribers": "@jessica.jones"
+    }
+) }}
+```
+
+```yml dbt_project.yml
+models:
+  path:
+    subfolder:
+      +meta:
+        subscribers: "@jessica.jones"
+
+tests:
+  path:
+    subfolder:
+      +meta:
+        subscribers: "@jessica.jones"
+```
+
+</CodeGroup>
diff --git a/docs/_snippets/alerts/tags.mdx b/docs/_snippets/alerts/tags.mdx
@@ -0,0 +1,39 @@
+You can use [tags](https://docs.getdbt.com/reference/resource-configs/tags) to provide context to your alerts.
+
+- You can tag a group or a channel in a slack alert by adding `#channel_name` as a tag.
+- Tags are aggregated,so a test alert will include both the test and the parent model tags.
+
+<CodeGroup>
+
+```yml model
+models:
+  - name: my_model_name
+    tags: ["#marketing", "#data_ops"]
+```
+
+```yml test
+tests:
+  - not_null:
+    tags: ["#marketing", "#data_ops"]
+```
+
+```yml test/model config block
+{{ config(
+    tags=["#marketing", "#data_ops"]
+    }
+) }}
+```
+
+```yml dbt_project.yml
+models:
+  path:
+    subfolder:
+      tags: ["#marketing", "#data_ops"]
+
+tests:
+  path:
+    subfolder:
+      tags: ["#marketing", "#data_ops"]
+```
+
+</CodeGroup>
diff --git a/docs/_snippets/cloud/features.mdx b/docs/_snippets/cloud/features.mdx
@@ -0,0 +1,81 @@
+
+### Detection & Coverage
+
+Elementary integrates powerful anomaly detection and dbt tests into a unified detection strategy.
+Effective detection of data issues requires a comprehensive approach,
+including both pipeline and data monitoring, validation tests,
+anomaly detection for unexpected behavior, and a single interface to manage it all at scale.
+
+<CardGroup cols={2}>
+  <Card title="Anomaly Detection" href="/features/anomaly-detection/monitors-overview">
+    ML-powered monitors automatically detect data quality issues.
+    Out-of-the-box for volume and freshness, and opt-in for data quality metrics.
+  </Card>
+  <Card title="Data Tests" href="/features/data-tests/data-tests-overview">
+    Validate data and track the results of dbt tests, dbt packages tests (dbt-utils, dbt-expectations, elementary) and custom SQL tests.  
+  </Card>
+  <Card title="Schema Validation" href="/features/data-tests/schema-validation-test">
+    Validate there are no breaking changes in tables schema, JSON schema, and downstream exposures such as dashboards.  
+  </Card>
+  <Card title="Runs & Performance" href="/features/performance-monitoring/performance-monitoring">
+    Track failures and runs of jobs, models, and tests overtime.
+    Pipeline failures and performance issues can cause data incidents, and create unneceserry costs.
+  </Card>
+  <Card title="Code & UI Configuration" href="/features/data-tests/data-tests-overview#test-configuration">
+    Configure Elementary in code, or via the UI for non-technical users or for adding tests in bulk.
+    The platform opens PRs to your repo, saving hours of tedious YAML edits.
+  </Card>
+  <Card title="Test Recommendations">
+    Coming soon!
+  </Card>
+</CardGroup>
+
+### Triage & Response
+
+Detecting issues is just the first step to achieve data reliability.
+Elementary offers tools to create an effective response plan, for faster recovery.
+This includes investigating the root cause and impact of issues, communicating issues to the relevant people, assigning owners to fix issues, keeping track of open incidents and more.
+
+<CardGroup cols={2}>
+  <Card title="Data Lineage" href="/features/data-lineage/lineage">
+    Column-level lineage that spans through sources, models and BI tools, enriched with monitoring results. Enables granular root cause and impact analysis. 
+  </Card>
+  <Card title="Ownership" href="/best-practices/governance-for-observability">
+    Define clear ownership of data assets and enable owners to be informed and accountable for the health and status of their data.   
+  </Card>
+  <Card title="Actionable Alerts" href="/features/alerts-and-incidents/alerts-and-incidents-overview">
+    Distribute highly configurable alerts to different channels and integrations. 
+    Automatically tag owners, and enable setting status and assigns at the alert level.  
+  </Card>
+  <Card title="Automated Grouping" href="/features/alerts-and-incidents/incidents">
+    Different failures related to the same issue are grouped automatically to a single incident.
+    This accelerates triage and response, and reduces alerts fautigue.  
+  </Card>
+  <Card title="Incident Management" href="/features/alerts-and-incidents/incident-management">
+    Manage all open incidents in a single interface, with a clear view of status and assignees.
+    Track historical incidents and high-level incidents metrics. 
+  </Card>
+</CardGroup>
+
+### Collaboration & Communication
+
+The data team doesn’t live in a silo - you have many stakeholders.
+The only way to improve data trust is by bringing in more team members, users and stakeholders to the data health process.
+Elementary fosters collaboration by allowing you to easily share and communicate the status of issues,
+the overall health of the data platform and progress made to improve it with the broader organization.
+
+<CardGroup cols={2}>
+  <Card title="Overview Dashboard" href="/features/collaboration-and-communication/data-observability-dashboard">
+    Up to date dashboard with current status and trends of data issues. 
+    Share the dashboard with others, enable them to slice results and stay informed.
+  </Card>
+  <Card title="Assets Tags" href="/best-practices/governance-for-observability">
+    Enable effective collaboration and communication by grouping related data assets and tests by business domains, data products, priority, etc.   
+  </Card>
+  <Card title="Data Catalog" href="/features/collaboration-and-communication/catalog">
+    Search and explore your datasets information - descriptions, columns, column descriptions, compiled code, datasets health and more.
+  </Card>
+  <Card title="Data Health Score" href="/features/collaboration-and-communication/data-health">
+    See the Data Health scores of all your datasets by domain and share with stakeholders.
+  </Card>
+</CardGroup>
diff --git a/docs/_snippets/cloud/features/alerts-and-incidents/alert-types.mdx b/docs/_snippets/cloud/features/alerts-and-incidents/alert-types.mdx
@@ -0,0 +1,7 @@
+Elementary can be configured to send alerts on:
+
+- Model run failures
+- Failures and/or warnings of dbt tests (including Elementary dbt package and other packages)
+- Failures and/or warnings Elementary Anomaly Detection monitors
+- Failures and/or warning of custom SQL tests
+- dbt source freshness failures
diff --git a/docs/_snippets/cloud/features/anomaly-detection/all-anomalies-configuration.mdx b/docs/_snippets/cloud/features/anomaly-detection/all-anomalies-configuration.mdx
@@ -0,0 +1,2 @@
+- **Severity** - Should a failure be considered a warning or a failure. Default is warning. 
+- **Test metadata** - Add metadata such as tags and owner to the test.
diff --git a/docs/_snippets/cloud/features/anomaly-detection/automated-monitors-cards.mdx b/docs/_snippets/cloud/features/anomaly-detection/automated-monitors-cards.mdx
@@ -0,0 +1,10 @@
+<CardGroup cols={2}>
+<Card title="Automated Freshness" href="/features/anomaly-detection/automated-freshness">
+  Monitors updates to tables and how frequently a table is updated, 
+  and fails if there is an unexpected delay. 
+</Card>
+<Card title="Automated Volume" href="/features/anomaly-detection/automated-volume">
+  Monitors how many rows were added or removed to a table on each update,
+  and fails if there is an unexpected drop or spike in rows. 
+</Card>
+</CardGroup>
diff --git a/docs/_snippets/cloud/features/anomaly-detection/automated-monitors-intro.mdx b/docs/_snippets/cloud/features/anomaly-detection/automated-monitors-intro.mdx
@@ -0,0 +1,5 @@
+Out-of-the-box ML-powered monitoring for freshness and volume issues on all production tables.
+The automated monitors feature provides broad coverage and detection of critical pipeline issues, without any configuration effort.
+
+These monitors track updates to tables, and will detect data delays, incomplete updates, and significant volume changes.
+Additionally, there will be no increase in compute costs as the monitors leverage only warehouse metadata (e.g. information schema, query history).
diff --git a/docs/_snippets/cloud/features/anomaly-detection/freshness-configuration.mdx b/docs/_snippets/cloud/features/anomaly-detection/freshness-configuration.mdx
@@ -0,0 +1,6 @@
+You can choose between 2 detection methods for the Freshness monitor- Automatic and Manual.
+- **Automatic** - Elementary uses machine learning models to detect anomalies in the data freshness. This is the default setting. You can change the sensitivity level to *Low*, *Medium*, or *High*.
+For each level, you will see a simulation of the change impact on the latest result, and you can use the`Simulate Configuration` button to examine the change impact.
+- **Manual** - You can set the SLA breach threshold for the freshness monitor manually. This is useful for assets that are updated regularly at the same time every day, hour or week.
+
+<img width="500" src="https://res.cloudinary.com/dvmtceykh/image/upload/v1740599403/30861cc3-b743-4125-922c-25df0e78b75c.png" alt="Freshness monitor configuration" />
diff --git a/docs/_snippets/cloud/features/anomaly-detection/volume-configuration.mdx b/docs/_snippets/cloud/features/anomaly-detection/volume-configuration.mdx
@@ -0,0 +1,3 @@
+- **Anomaly Direction** - Whether you want the monitor to fail on anomalous drops, spikes, or both. Default is both.
+- **Sensitivity** -  You can set the monitor's sensitivity levels to *Low*, *Medium*, or *High*. In the future, we plan to allow for more nuanced adjustments to this parameter. You can use the `Simulate Configuration` button for testing how the change will affect the monitor.
+- **Detection Period** - The period in which the monitor look for anomalies. Default is the last 2 days.
diff --git a/docs/_snippets/cloud/features/data-health/data-health-intro.mdx b/docs/_snippets/cloud/features/data-health/data-health-intro.mdx
@@ -0,0 +1,11 @@
+Once you start sharing data with downstream consumers and stakeholders one of the most important things that you want to create is trust.
+Trust that the data that is being used is “healthy”. Imagine being a data analyst using a specific data asset but you constantly run into data quality issues.
+You will eventually lose trust.
+
+This is why we created **data health scores** in Elementary. It is a way to share an overview of the health of your data assets.
+
+To measure health we use an industry standard framework of [Data Quality Dimensions](/features/collaboration-and-communication/data-quality-dimensions#data-quality-dimensions).
+These dimensions help assess the reliability of data in various business contexts.
+Ensuring high-quality data across these dimensions is critical for accurate analysis, informed decision-making, and operational efficiency.
+
+<Tip>To learn more, **watch the webinar** [**Measuring Data Health with Elementary**](https://www.elementary-data.com/webinar/measuring-data-health-with-elementary)</Tip>
diff --git a/docs/_snippets/cloud/features/data-health/data-quality-dimensions.mdx b/docs/_snippets/cloud/features/data-health/data-quality-dimensions.mdx
@@ -0,0 +1,20 @@
+<CardGroup cols={3}>
+  <Card title="Freshness" icon="stopwatch">
+    Ensures that data is up to date and reflects the latest information.
+  </Card>
+  <Card title="Completeness" icon="ballot-check">
+    Ensures all required data is available, without missing values.
+  </Card>
+  <Card title="Accuracy" icon="bullseye-arrow">
+    Ensures that data represents the real-world scenario correctly.
+  </Card>
+  <Card title="Consistency" icon="code-compare">
+    The degree to which data remains uniform across multiple instances.
+  </Card>
+  <Card title="Uniqueness" icon="barcode-read">
+    Ensures that each entity is represented only once and there are no duplicates.
+  </Card>
+  <Card title="Validity" icon="puzzle">
+    Ensures that data conforms to rules or expectations, such as acceptable ranges or formats.
+  </Card>
+</CardGroup>
diff --git a/docs/_snippets/cloud/features/data-tests/benefits-dbt-tests.mdx b/docs/_snippets/cloud/features/data-tests/benefits-dbt-tests.mdx
@@ -0,0 +1,10 @@
+dbt tests are very powerful. The ease of use, simplicity, and usefulness in the dev process is unmatched.
+When you adopt any observability tool, you will still use dbt tests. This is why in Elementary, dbt tests are first class citizens.
+
+There are several benefits to this approach:
+
+- **Single interface for all observability** - Prevent the distribution of monitoring between different tools. All configuration is in code, all the results are in one interface.
+- **Avoid duplicate work and vendor lock in** - The tests you implemented already are effective in Elementary, as well as additional configuration. The future tests you add will remain in your code if you decide to offboard.
+- **Control of schedule and cost** - You have control of configuration and scheduling, tests can be executed when data is actually loaded and validation is needed.
+- **Prevent bad data from propagating** - As tests are in pipeline, you can leverage `dbt build` and fail the pipeline on critical test failures.
+- **Rich ecosystem** - The community of dbt users developes and supports various testing use cases.
diff --git a/docs/_snippets/cloud/features/data-tests/data-tests-cards.mdx b/docs/_snippets/cloud/features/data-tests/data-tests-cards.mdx
@@ -0,0 +1,14 @@
+<CardGroup cols={2}>
+<Card title="dbt tests" href="/features/data-tests/dbt-tests">
+  Native dbt tests such as `not_null`, `unique`, etc.
+</Card>
+<Card title="dbt packages" href="/features/data-tests/dbt-tests#supported-dbt-packages">
+  Tests of packages such as `dbt-expectations`, `dbt-utils`, etc.
+</Card>
+<Card title="Custom SQL tests" href="/features/data-tests/custom-sql-tests">
+  Tests to validate an explicit business logic.
+</Card>
+<Card title="Schema validation" href="/features/data-tests/schema-validation-test">
+  Schema tests by Elementary, implemented as dbt tests. 
+</Card>
+</CardGroup>
diff --git a/docs/_snippets/cloud/features/data-tests/dbt-test-hub.mdx b/docs/_snippets/cloud/features/data-tests/dbt-test-hub.mdx
@@ -0,0 +1,4 @@
+To help you find the test that is right for your use case, we created the [dbt Test Hub](https://www.elementary-data.com/dbt-test-hub).
+It's a searchable catalog of all the tests supported in Elementary, with their descriptions and example use cases.
+
+The tests are also segmented to use cases, so you can easily find the different options for addressing your detection use case.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		- Severity - Should a failure be considered a warning or a failure. Default is warning.
		- Test metadata - Add metadata such as tags and owner to the test.