Skip to content

Commit 285b4c0

Browse files
committed
Merge remote-tracking branch 'great-expectations/develop' into develop
2 parents c994555 + ca1a025 commit 285b4c0

35 files changed

+427
-2192
lines changed

.github/workflows/ci.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -635,7 +635,8 @@ jobs:
635635
fail-fast: false
636636
matrix:
637637
markers:
638-
- athena or openpyxl or pyarrow or project or sqlite or aws_creds
638+
- openpyxl or pyarrow or project or sqlite or aws_creds
639+
- athena
639640
- aws_deps
640641
- big
641642
- bigquery

docs/docusaurus/docs/cloud/connect/connect_databrickssql.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ import Tabs from '@theme/Tabs';
3535

3636
6. Click **Add x Asset(s)**.
3737

38-
7. Decide which [Anomaly Detection](/docs/cloud/overview/accelerating_test_coverage.md#anomaly-detection) options you want to enable. By default, GX Cloud adds [warning-severity](/docs/cloud/expectations/expectations_overview.md#failure-severity) Expectations to detect **Schema**, **Volume**, and **Completeness** anomalies. You can de-select recommendations you’d like to opt out of.
38+
7. Decide which [Anomaly Detection](/docs/cloud/overview/accelerating_test_coverage.md#anomaly-detection) options you want to enable. By default, GX Cloud adds [warning-severity](/docs/cloud/expectations/expectations_overview.md#failure-severity) Expectations to detect **Schema** and **Volume** anomalies. You can de-select recommendations you’d like to opt out of. You can choose to generate Expectations to detect **Completeness** anomalies.
3939

4040
8. Click **Start monitoring** or **Finish**.
4141

docs/docusaurus/docs/cloud/connect/connect_postgresql.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ import Tabs from '@theme/Tabs';
6060

6161
6. Click **Add x Asset(s)**.
6262

63-
7. Decide which [Anomaly Detection](/docs/cloud/overview/accelerating_test_coverage.md#anomaly-detection) options you want to enable. By default, GX Cloud adds [warning-severity](/docs/cloud/expectations/expectations_overview.md#failure-severity) Expectations to detect **Schema**, **Volume**, and **Completeness** anomalies. You can de-select recommendations you’d like to opt out of.
63+
7. Decide which [Anomaly Detection](/docs/cloud/overview/accelerating_test_coverage.md#anomaly-detection) options you want to enable. By default, GX Cloud adds [warning-severity](/docs/cloud/expectations/expectations_overview.md#failure-severity) Expectations to detect **Schema** and **Volume** anomalies. You can de-select recommendations you’d like to opt out of. You can choose to generate Expectations to detect **Completeness** anomalies.
6464

6565
8. Click **Start monitoring** or **Finish**.
6666

docs/docusaurus/docs/cloud/connect/connect_redshift.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ Run all queries in the [Amazon Redshift query editor v2](https://docs.aws.amazon
101101

102102
7. Click **Add x Asset(s)**.
103103

104-
8. Decide which [Anomaly Detection](/docs/cloud/overview/accelerating_test_coverage.md#anomaly-detection) options you want to enable. By default, GX Cloud adds [warning-severity](/docs/cloud/expectations/expectations_overview.md#failure-severity) Expectations to detect **Schema**, **Volume**, and **Completeness** anomalies. You can de-select recommendations you’d like to opt out of.
104+
8. Decide which [Anomaly Detection](/docs/cloud/overview/accelerating_test_coverage.md#anomaly-detection) options you want to enable. By default, GX Cloud adds [warning-severity](/docs/cloud/expectations/expectations_overview.md#failure-severity) Expectations to detect **Schema** and **Volume** anomalies. You can de-select recommendations you’d like to opt out of. You can choose to generate Expectations to detect **Completeness** anomalies.
105105

106106
9. Click **Start monitoring** or **Finish**.
107107

docs/docusaurus/docs/cloud/connect/connect_snowflake.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,7 @@ Then, you can use the GX Cloud UI to [add a Data Asset](/cloud/data_assets/manag
112112

113113
7. Click **Add x Asset(s)**.
114114

115-
8. Decide which [Anomaly Detection](/docs/cloud/overview/accelerating_test_coverage.md#anomaly-detection) options you want to enable. By default, GX Cloud adds [warning-severity](/docs/cloud/expectations/expectations_overview.md#failure-severity) Expectations to detect **Schema**, **Volume**, and **Completeness** anomalies. You can de-select recommendations you’d like to opt out of.
115+
8. Decide which [Anomaly Detection](/docs/cloud/overview/accelerating_test_coverage.md#anomaly-detection) options you want to enable. By default, GX Cloud adds [warning-severity](/docs/cloud/expectations/expectations_overview.md#failure-severity) Expectations to detect **Schema** and **Volume** anomalies. You can de-select recommendations you’d like to opt out of. You can choose to generate Expectations to detect **Completeness** anomalies.
116116

117117
9. Click **Start monitoring** or **Finish**.
118118

docs/docusaurus/docs/cloud/data_assets/manage_data_assets.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ To add a Data Asset from an existing Data Source, complete the following steps:
2424

2525
4. Click **Add x Asset(s)**.
2626

27-
5. Decide which [Anomaly Detection](/docs/cloud/overview/accelerating_test_coverage.md#anomaly-detection) options you want to enable. By default, GX Cloud adds [warning-severity](/docs/cloud/expectations/expectations_overview.md#failure-severity) Expectations to detect **Schema**, **Volume**, and **Completeness** anomalies. You can de-select recommendations you’d like to opt out of.
27+
5. Decide which [Anomaly Detection](/docs/cloud/overview/accelerating_test_coverage.md#anomaly-detection) options you want to enable. By default, GX Cloud adds [warning-severity](/docs/cloud/expectations/expectations_overview.md#failure-severity) Expectations to detect **Schema** and **Volume** anomalies. You can de-select recommendations you’d like to opt out of. You can choose to generate Expectations to detect **Completeness** anomalies.
2828

2929
6. Click **Start monitoring** or **Finish**.
3030

docs/docusaurus/docs/cloud/expectations/expectations_overview.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -176,9 +176,9 @@ Here is a comparison of key characteristics of GX-managed and API-managed Expect
176176
|--------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
177177
| Edit | [Edit parameters](/cloud/expectations/manage_expectations.md#edit-an-expectation) with the GX Cloud UI | [Edit parameters with the GX Cloud API](/reference/api/expectations/Expectation_class.mdx) or the GX Cloud UI |
178178
| Batch | [Define a Batch](/cloud/expectations/manage_expectations.md#optional-define-a-batch) in the GX Cloud UI | Define a Batch with the GX Cloud API when connecting to [SQL](/core/connect_to_data/sql_data/sql_data.md#create-a-batch-definition), [filesystem](/core/connect_to_data/filesystem_data/filesystem_data.md#create-a-batch-definition), or [dataframe](/core/connect_to_data/dataframes/dataframes.md#create-a-batch-definition) data |
179-
| Validate | [Run a Validation](/docs/cloud/validations/manage_validations.md#run-a-validation) using the GX Cloud UI or [run a Checkpoint](/core/trigger_actions_based_on_results/run_a_checkpoint.md) with the GX Cloud API | [Create a Validation Definition and run it](/core/run_validations/run_validations.md) with the GX Cloud API |
179+
| Ad hoc Validation | [Run a Validation](/docs/cloud/validations/manage_validations.md#run-a-validation) using the GX Cloud UI or [run a Checkpoint](/core/trigger_actions_based_on_results/run_a_checkpoint.md) with the GX Cloud API | [Create a Validation Definition and run it](/core/run_validations/run_validations.md) with the GX Cloud API |
180+
| Recurring Validations | [Schedule Validations](/cloud/schedules/manage_schedules.md) in the GX Cloud UI | Not supported, use an [orchestrator](/cloud/connect/connect_airflow.md) to control recurring validations |
180181
| Validation Results | [Access results in the Validations tab](/cloud/validations/manage_validations.md#view-validation-run-history) of the GX Cloud UI | [Access results with the GX Cloud API](/core/trigger_actions_based_on_results/choose_a_result_format/choose_a_result_format.md) or in the Validations tab of the GX Cloud UI |
181-
| Schedule | Keep default schedule or [edit schedule](/cloud/schedules/manage_schedules.md) in the GX Cloud UI | Not supported, use an [orchestrator](/cloud/connect/connect_airflow.md) to control recurring validations |
182182
| Expectation Suite | Automatically organized in a hidden default Expectation Suite | Manually grouped into [custom Expectation Suites](/core/define_expectations/organize_expectation_suites.md) via the GX Cloud API |
183183
| Delete | [Delete Expectation](/docs/cloud/expectations/manage_expectations/#delete-an-expectation) with the GX Cloud UI | [Delete Expectation with the GX Cloud API](/reference/api/ExpectationSuite_class.mdx#great_expectations.ExpectationSuite.delete_expectation) or the GX Cloud UI |
184184

docs/docusaurus/docs/cloud/expectations/manage_expectations.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,11 +32,13 @@ This page provides instructions for working with Expectations. To learn about Ex
3232

3333
5. Select an Expectation type.
3434

35-
6. Complete the mandatory and optional fields for the Expectation. A recurring [validation schedule](/cloud/schedules/manage_schedules.md) will be applied automatically to your Expectation.
35+
6. Complete the mandatory and optional fields for the Expectation.
3636

3737
7. Click **Save** or click **Save & Add More** and then repeat steps 4 through 7 to add additional Expectations.
3838

39-
8. Optional. Run a Validation. See [Run a Validation](/cloud/validations/manage_validations.md#run-a-validation).
39+
8. Optional. Run an ad hoc Validation. See [Run a Validation](/cloud/validations/manage_validations.md#run-a-validation).
40+
41+
9. Optional. Configure recurring Validations. See [Manage schedules](/docs/cloud/schedules/manage_schedules).
4042

4143

4244
## Save time with ExpectAI <span class="beta">Beta</span>

docs/docusaurus/docs/cloud/overview/accelerating_test_coverage.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ GX Cloud provides several conveniences to help you more quickly achieve test cov
1111

1212
## Anomaly Detection
1313

14-
When you [add a new Data Asset](/cloud/data_assets/manage_data_assets.md), GX Cloud by default generates Expectations to detect anomalies in the following data quality issues. You can also automate Anomaly Detection for these when you [add Expectations](/cloud/expectations/manage_expectations.md#add-an-expectation) for an existing Data Asset.
14+
When you [add a new Data Asset](/cloud/data_assets/manage_data_assets.md), you can have GX Cloud generate Expectations to detect anomalies in the following data quality issues. You can also automate Anomaly Detection for these when you [add Expectations](/cloud/expectations/manage_expectations.md#add-an-expectation) for an existing Data Asset.
1515

1616
- [Schema](#schema)
1717
- [Volume](#volume)
@@ -22,16 +22,16 @@ When you [add a new Data Asset](/cloud/data_assets/manage_data_assets.md), GX Cl
2222

2323
### Schema
2424

25-
To detect schema anomalies, we automatically generate a rule to [**expect table columns to match set**](/reference/learn/data_quality_use_cases/schema.md#expect-table-columns-to-match-set) using the Data Asset’s initial columns as the set to match. If the number or names of columns in the Data Asset change, this Expectation will fail.
25+
To detect schema anomalies, we generate a rule to [**expect table columns to match set**](/reference/learn/data_quality_use_cases/schema.md#expect-table-columns-to-match-set) using the Data Asset’s initial columns as the set to match. If the number or names of columns in the Data Asset change, this Expectation will fail.
2626

2727
### Volume
2828

29-
To detect anomalies in row count, we automatically generate a rule to [**expect table row count to be between**](/reference/learn/data_quality_use_cases/volume.md#expect-table-row-count-to-be-between) a forecasted range that tests that the current validation run doesn’t deviate significantly from historical patterns. For example, if there is a sudden increase when volume has been stable or stagnation in a season when volume typically grows, then this Expectation will fail.
29+
To detect anomalies in row count, we generate a rule to [**expect table row count to be between**](/reference/learn/data_quality_use_cases/volume.md#expect-table-row-count-to-be-between) a forecasted range that tests that the current validation run doesn’t deviate significantly from historical patterns. For example, if there is a sudden increase when volume has been stable or stagnation in a season when volume typically grows, then this Expectation will fail.
3030

3131

3232
### Completeness
3333

34-
To detect completeness anomalies, we automatically generate rules for every column to [**expect column proportion of non-null values to be between**](/reference/learn/data_quality_use_cases/missingness.md#expect-column-proportion-of-non-null-values-to-be-between) a forecasted range that depends on the column's initial proportion of non-null values.
34+
To detect completeness anomalies, we generate rules for every column to [**expect column proportion of non-null values to be between**](/reference/learn/data_quality_use_cases/missingness.md#expect-column-proportion-of-non-null-values-to-be-between) a forecasted range that depends on the column's initial proportion of non-null values.
3535

3636
- If a column initially has no null values, GX Cloud generates a rule to test that the column continues to have no null values.
3737
- If a column initially has all null values, GX Cloud generates a rule to test that the column continues to have all null values.

docs/docusaurus/docs/cloud/schedules/manage_schedules.md

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,29 @@ title: 'Manage schedules'
44
description: Create and manage schedules for Validations in GX Cloud.
55
---
66

7-
Use a schedule to automate data quality checks with GX-managed Expectations. When you add your first Expectation in the GX Cloud UI for a Data Asset, including when you choose to auto-generate Expectations for Anomaly Detection, we enable a default schedule for that Asset's GX-managed Expectations. By default, GX-managed Expectations are scheduled to run every 24 hours. The first run will be at the start of the next hour after you add your first Expectation in the Cloud UI. You can keep the default schedule, edit it, or disable it.
7+
Use a schedule to automate data quality checks with GX-managed Expectations.
88

99
:::note Schedules are for GX-managed Expectations only
1010
To automate data quality checks for [API-managed Expectations](/cloud/expectations/manage_expectations.md#gx-managed-vs-api-managed-expectations), use an [orchestrator](/cloud/connect/connect_airflow.md).
1111
:::
1212

1313

14+
## Enable a schedule
15+
16+
1. In GX Cloud, select the relevant **Workspace** and then click **Data Assets**.
17+
18+
2. In the **Data Assets** list, click the Data Asset name.
19+
20+
3. Next to **No scheduled validation**, click <img src="/img/pencil.png" alt="pencil icon" width="20" height="20"/> **Edit schedule**.
21+
22+
4. Turn the Validation schedule **ON**.
23+
24+
5. Select a **Frequency**.
25+
26+
6. Select a **Start time** for the first run of the new schedule.
27+
28+
7. Click **Save**.
29+
1430
## Edit a schedule
1531

1632
1. In GX Cloud, select the relevant **Workspace** and then click **Data Assets**.

0 commit comments

Comments
 (0)