You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Update README with AI-powered data validation section
Add documentation for AI-powered data validation and unstructured data tests, including an example configuration and a link to further documentation. Enhance the README's overview of Elementary's capabilities.
Copy file name to clipboardExpand all lines: README.md
+28-3Lines changed: 28 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,6 +23,7 @@ Available as self-hosted or Cloud service with premium features.
23
23
-[Get more out of Elementary](#get-more-out-of-elementary-dbt-package)
24
24
-[Run results and dbt artifacts](#run-results-and-dbt-artifacts)
25
25
-[Data anomaly detection as dbt tests](#data-anomaly-detection-as-dbt-tests)
26
+
-[AI-powered data validation and unstructured data tests](#ai-powered-data-validation-and-unstructured-data-tests)
26
27
-[How Elementary works?](#how-elementary-works)
27
28
-[Community & Support](#community--support)
28
29
-[Contribution](#contributions)
@@ -61,9 +62,9 @@ Elementary has 3 offerings: This dbt package, Elementary Community (OSS) and Ele
61
62
- **dbt package**
62
63
- For basic data monitoring and dbt artifacts collection, Elementary offers a dbt package. The package adds logging, artifacts uploading, and Elementary tests (anomaly detection and schema) to your project.
63
64
- **Elementary Community**
64
-
- An open-source CLI tool you can deploy and orchestrate to send alerts and self-host the Elementary report. Best for data and analytics engineers that require basic observability capabilities or for evaluating features without vendor approval. Our community can provide great support on[Slack](https://www.elementary-data.com/community)if needed.
65
+
- An open-source CLI tool you can deploy and orchestrate to send alerts and self-host the Elementary report. Best for data and analytics engineers that require basic observability capabilities or for evaluating features without vendor approval. Our community can provide great support on[Slack](https://www.elementary-data.com/community)if needed.
65
66
- **Elementary Cloud**
66
-
- Ideal for teams monitoring mission-critical data pipelines, requiring guaranteed uptime and reliability, short-time-to-value, advanced features, collaboration, and professional support. The solution is secure by design, and requires no access to your data from cloud. To learn more,[book a demo](https://cal.com/maayansa/elementary-intro-github-package)or [start a trial](https://www.elementary-data.com/signup).
67
+
- Ideal for teams monitoring mission-critical data pipelines, requiring guaranteed uptime and reliability, short-time-to-value, advanced features, collaboration, and professional support. The solution is secure by design, and requires no access to your data from cloud. To learn more,[book a demo](https://cal.com/maayansa/elementary-intro-github-package)or [start a trial](https://www.elementary-data.com/signup).
67
68
68
69
## Run Results and dbt artifacts
69
70
@@ -110,6 +111,30 @@ models:
110
111
111
112
Read about the available [Elementary tests and configuration](https://docs.elementary-data.com/data-tests/introduction).
112
113
114
+
## AI-powered data validation and unstructured data tests
115
+
116
+
Elementary leverages AI to enhance data reliability with natural language test definitions:
117
+
118
+
- **AI data validation**: Define expectations in plain English to validate structured data
119
+
- **Unstructured data validation**: Validate text, JSON, and other non-tabular data types
120
+
121
+
Example:
122
+
123
+
```yml
124
+
# AI data validation example
125
+
models:
126
+
- name: crm
127
+
description: "A table containing contract details."
128
+
columns:
129
+
- name: contract_date
130
+
description: "The date when the contract was signed."
131
+
tests:
132
+
- elementary.ai_data_validation:
133
+
expectation_prompt: "There should be no contract date in the future"
134
+
```
135
+
136
+
Learn more in our [AI data validation documentation](hhttps://docs.elementary-data.com/data-tests/ai-data-tests/ai_data_validations).
137
+
113
138
## How Elementary works?
114
139
115
140
Elementary dbt package creates tables of metadata and test results in your data warehouse, as part of your dbt runs.
@@ -127,6 +152,6 @@ The cloud service or the CLI tool read the data from these tables, send alerts a
127
152
128
153
## Contributions
129
154
130
-
Thank you :orange_heart: Whether it’s a bug fix, new feature, or additional documentation - we greatly appreciate contributions!
155
+
Thank you :orange_heart: Whether it's a bug fix, new feature, or additional documentation - we greatly appreciate contributions!
131
156
132
157
Check out the [contributions guide](https://docs.elementary-data.com/general/contributions) and [open issues](https://github.com/elementary-data/elementary/issues) in the main repo.
0 commit comments