Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions docs/pages/product/caching/matching-pre-aggregations.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,10 @@ the values should be the start and end days of the month, i.e.,
`['2020-01-01T00:00:00.000', '2020-01-31T23:59:59.999']`; when the granularity
is `day`, the values should be the start and end hours of the day, i.e.,
`['2020-01-01T00:00:00.000', '2020-01-01T23:59:59.999']`. Date ranges are
inclusive, and the minimum granularity is `second`. Use the
[`allow_non_strict_date_range_match`][ref-non-strict-date-range-match] to allow
a pre-aggregation to match a non-strict date range anyway.
inclusive, and the minimum granularity is `second`. By default, this is ensured
via the [`allow_non_strict_date_range_match`][ref-non-strict-date-range-match]
parameter of pre-aggregations: it allows to match _non-strict date ranges_ and is
set to `true` by default.
- **The time zone in the query must match the time zone of a pre-aggregation.**
You can configure a list of time zones that pre-aggregations will be built for
using the [`scheduled_refresh_time_zones`][ref-conf-scheduled-refresh-time-zones]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -117,15 +117,6 @@ The `COUNT(*)` aggregate function is being mapped to a measure of type
[count](/reference/data-model/types-and-formats#count) in Cube's
**Orders** data model file.

## Additional configuration

### Pre-aggregations

To allow queries from Superset to match pre-aggregations in Cube, [the
`allow_non_strict_date_range_match`
property][ref-schema-ref-preagg-allownonstrict] must be set to `true` in the
pre-aggregation definition. This is because Superset uses loose date ranges when
generating SQL queries.

[ref-getting-started]: /product/getting-started/cloud
[ref-schema-ref-preagg-allownonstrict]:
Expand Down
13 changes: 11 additions & 2 deletions docs/pages/reference/configuration/environment-variables.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1057,6 +1057,16 @@ to use for storing pre-aggregations.
It can be also set using the [`pre_aggregations_schema` configuration
option](/reference/configuration/config#pre_aggregations_schema).

## `CUBEJS_PRE_AGGREGATIONS_ALLOW_NON_STRICT_DATE_RANGE_MATCH`

If `true`, allows non-strict date range matching for pre-aggregations by default.
This can be overridden for individual pre-aggregations using the
[`allow_non_strict_date_range_match`][ref-schema-ref-preagg-allownonstrict] parameter.

| Possible Values | Default in Development | Default in Production |
| --------------- | ---------------------- | --------------------- |
| `true`, `false` | `true` | `true` |

## `CUBEJS_REFRESH_WORKER`

If `true`, this instance of Cube will **only** refresh pre-aggregations.
Expand All @@ -1078,8 +1088,6 @@ mode](/product/caching/using-pre-aggregations#rollup-only-mode) for details.
It can be also set using the [`orchestrator_options.rollupOnlyMode` configuration
option](/reference/configuration/config#orchestrator_options).



## `CUBEJS_SCHEDULED_REFRESH_TIMEZONES`

A comma-separated [list of timezones to schedule refreshes
Expand Down Expand Up @@ -1688,3 +1696,4 @@ The port for a Cube deployment to listen to API connections on.
[ref-sql-api-streaming]: /product/apis-integrations/sql-api#streaming
[ref-row-limit]: /product/apis-integrations/queries#row-limit
[mysql-server-tz-support]: https://dev.mysql.com/doc/refman/8.4/en/time-zone-support.html
[ref-schema-ref-preagg-allownonstrict]: /reference/data-model/pre-aggregations#allow_non_strict_date_range_match
24 changes: 16 additions & 8 deletions docs/pages/reference/data-model/pre-aggregations.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1110,14 +1110,20 @@ refreshes.

### `allow_non_strict_date_range_match`

The `allow_non_strict_date_range_match` property can be used to allow queries to
match this pre-aggregation when using non-strict date ranges.
By default Cube checks if requested date range exactly matches pre-aggregation granularity.
For example if you're requesting half of a day or your date range filter is just one millisecond off for daily granularity pre-aggregation Cube would not use such a pre-aggregation by default.
Use this flag to lift this strict check.
This is useful
when using a visualization tools such as Tableau or [Apache
Superset][ref-config-downstream-superset] that use loose date ranges by default.
The `allow_non_strict_date_range_match` parameter is used to allow queries to match a
pre-aggregation even if a query contains a _non-strict date range_. It is set to `true`
by default via the [`CUBEJS_PRE_AGGREGATIONS_ALLOW_NON_STRICT_DATE_RANGE_MATCH`][ref-env-allow-non-strict]
environment variable.

If this flag is set to `false`, Cube would check if requested date range exactly matches
pre-aggregation granularity. For example, if you're requesting half of a day or your date
range filter is just one millisecond off for a pre-aggregation with the daily granularity,
Cube would not use such a pre-aggregation.

With this flag set to `true`, that strict check is lifted. It allows queries from BI tools
to still match pre-aggregations at the cost of a slight potential data discrepancy.
This is convenient when using Cube with visualization tools such as [Tableau][ref-config-downstream-tableau]
or [Apache Superset][ref-config-downstream-superset] that use loose date ranges.

<CodeTabs>

Expand Down Expand Up @@ -1767,3 +1773,5 @@ cubes:
[ref-pre-aggs]: /product/caching/using-pre-aggregations
[ref-ref-cubes]: /reference/data-model/cube
[ref-custom-granularity]: /reference/data-model/dimensions#granularities
[ref-env-allow-non-strict]: /reference/configuration/environment-variables#cubejs-pre-aggregations-allow-non-strict-date-range-match
[ref-config-downstream-tableau]: /product/configuration/visualization-tools/tableau
3 changes: 3 additions & 0 deletions packages/cubejs-backend-shared/src/env.ts
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,9 @@
transpilationWorkerThreads: () => get('CUBEJS_TRANSPILATION_WORKER_THREADS')
.default('false')
.asBoolStrict(),
allowNonStrictDateRangeMatching: () => get('CUBEJS_PRE_AGGREGATIONS_ALLOW_NON_STRICT_DATE_RANGE_MATCH')

Check warning on line 229 in packages/cubejs-backend-shared/src/env.ts

View check run for this annotation

Codecov / codecov/patch

packages/cubejs-backend-shared/src/env.ts#L229

Added line #L229 was not covered by tests
.default('true')
.asBoolStrict(),
transpilationWorkerThreadsCount: () => get('CUBEJS_TRANSPILATION_WORKER_THREADS_COUNT')
.default('0')
.asInt(),
Expand Down
6 changes: 6 additions & 0 deletions packages/cubejs-schema-compiler/src/compiler/CubeSymbols.ts
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,12 @@ export class CubeSymbols {
preAggregation.type = 'rollup';
}

if (preAggregation.allowNonStrictDateRangeMatch === undefined &&
['originalSql', 'rollupJoin', 'rollup'].includes(preAggregation.type) &&
(preAggregation.timeDimension || preAggregation.timeDimensions)) {
preAggregation.allowNonStrictDateRangeMatch = getEnv('allowNonStrictDateRangeMatching');
}

if (preAggregation.scheduledRefresh === undefined && preAggregation.type !== 'rollupJoin' && preAggregation.type !== 'rollupLambda') {
preAggregation.scheduledRefresh = getEnv('scheduledRefreshDefault');
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,8 @@ describe('PreAggregations', () => {
countCustomGranularity: {
measures: [count],
timeDimension: createdAt,
granularity: 'hourTenMinOffset'
granularity: 'hourTenMinOffset',
allowNonStrictDateRangeMatch: false
},
sourceAndIdRollup: {
measures: [count],
Expand Down
4 changes: 4 additions & 0 deletions packages/cubejs-schema-compiler/test/unit/schema.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ describe('Schema Testing', () => {
refreshRangeEnd: {
sql: expect.any(Function),
},
allowNonStrictDateRangeMatch: true,
},
countCreatedAt: {
external: true,
Expand Down Expand Up @@ -102,6 +103,7 @@ describe('Schema Testing', () => {
refreshRangeEnd: {
sql: expect.any(Function),
},
allowNonStrictDateRangeMatch: true,
}
});
});
Expand All @@ -128,6 +130,7 @@ describe('Schema Testing', () => {
refreshRangeEnd: {
sql: expect.any(Function),
},
allowNonStrictDateRangeMatch: true,
},
countCreatedAt: {
// because preview
Expand Down Expand Up @@ -162,6 +165,7 @@ describe('Schema Testing', () => {
refreshRangeEnd: {
sql: expect.any(Function),
},
allowNonStrictDateRangeMatch: true,
}
});
});
Expand Down
Loading