Skip to content

Commit a5a04aa

Browse files
pulpdrewpulpdrew
authored
feat: Add materialized view support (Beta) (#1507)
Closes HDX-3082 # Summary This PR back-ports support for materialized views from the EE repo. Note that this feature is in **Beta**, and is subject to significant changes. This feature is intended to support: 1. Configuring AggregatingMergeTree (or SummingMergeTree) Materialized Views which are associated with a Source 2. Automatically selecting and querying an associated materialized view when a query supports it, in Chart Explorer, Custom Dashboards, the Services Dashboard, and the Search Page Histogram. 3. A UX for understanding what materialized views are available for a source, and whether (and why) it is or is not being used for a particular visualization. ## Note to Reviewer(s) This is a large PR, but the code has largely already been reviewed. - For net-new files, types, components, and utility functions, the code does not differ from the EE repo - Changes to the various services dashboard pages do not differ from the EE repo - Changes to `useOffsetPaginatedQuery`, `useChartConfig`, and `DBEditTimeChart` differ slightly due to unrelated (to MVs) drift between this repo and the EE repo, and due to the lack of feature toggles in this repo. **This is where slightly closer review would be most valuable.** ## Demo <details> <summary>Demo: MV Configuration</summary> https://github.com/user-attachments/assets/fedf3bcf-892c-4b8d-a788-7e231e23bcc3 </details> <details> <summary>Demo: Chart Explorer</summary> https://github.com/user-attachments/assets/fc8d1efa-7edc-42fc-98f0-75431cc056b8 </details> <details> <summary>Demo: Dashboards</summary> https://github.com/user-attachments/assets/f3cb247e-711f-4d90-95b8-cf977e94f065 </details> ## Known Limitations This feature is in Beta due to the following known limitations, which will be addressed in subsequent PRs: 1. Visualization start and end time, when not aligned with the granularity of MVs, will result in statistics based on the MV "time buckets" which fall inside the date range. This may not align exactly with the source table data which is in the selected date range. 2. Alerts do not make use of MVs, even if the associated visualization does. Due to (1), this means that alert values may not exactly match the values shown in the associated visualization. ## Differences in OSS vs EE Support - In OSS, there is a beta label on the MV configurations section - In EE there are feature toggles to enable MV support, in OSS the feature is enabled for all teams, but will only run for sources with MVs configured. ## Testing To test, a couple of MVs can be created on the default `otel_traces` table, directly in ClickHouse: <details> <summary>Example MVs DDL</summary> ```sql CREATE TABLE default.metrics_rollup_1m ( `Timestamp` DateTime, `ServiceName` LowCardinality(String), `SpanKind` LowCardinality(String), `StatusCode` LowCardinality(String), `count` SimpleAggregateFunction(sum, UInt64), `sum__Duration` SimpleAggregateFunction(sum, UInt64), `avg__Duration` AggregateFunction(avg, UInt64), `quantile__Duration` AggregateFunction(quantileTDigest(0.5), UInt64), `min__Duration` SimpleAggregateFunction(min, UInt64), `max__Duration` SimpleAggregateFunction(max, UInt64) ) ENGINE = AggregatingMergeTree PARTITION BY toDate(Timestamp) ORDER BY (Timestamp, StatusCode, SpanKind, ServiceName); CREATE MATERIALIZED VIEW default.metrics_rollup_1m_mv TO default.metrics_rollup_1m ( `Timestamp` DateTime, `ServiceName` LowCardinality(String), `SpanKind` LowCardinality(String), `version` LowCardinality(String), `StatusCode` LowCardinality(String), `count` UInt64, `sum__Duration` Int64, `avg__Duration` AggregateFunction(avg, UInt64), `quantile__Duration` AggregateFunction(quantileTDigest(0.5), UInt64), `min__Duration` SimpleAggregateFunction(min, UInt64), `max__Duration` SimpleAggregateFunction(max, UInt64) ) AS SELECT toStartOfMinute(Timestamp) AS Timestamp, ServiceName, SpanKind, StatusCode, count() AS count, sum(Duration) AS sum__Duration, avgState(Duration) AS avg__Duration, quantileTDigestState(0.5)(Duration) AS quantile__Duration, minSimpleState(Duration) AS min__Duration, maxSimpleState(Duration) AS max__Duration FROM default.otel_traces GROUP BY Timestamp, ServiceName, SpanKind, StatusCode; ``` ```sql CREATE TABLE default.span_kind_rollup_1m ( `Timestamp` DateTime, `ServiceName` LowCardinality(String), `SpanKind` LowCardinality(String), `histogram__Duration` AggregateFunction(histogram(20), UInt64) ) ENGINE = AggregatingMergeTree PARTITION BY toDate(Timestamp) ORDER BY (Timestamp, ServiceName, SpanKind); CREATE MATERIALIZED VIEW default.span_kind_rollup_1m_mv TO default.span_kind_rollup_1m ( `Timestamp` DateTime, `ServiceName` LowCardinality(String), `SpanKind` LowCardinality(String), `histogram__Duration` AggregateFunction(histogram(20), UInt64) ) AS SELECT toStartOfMinute(Timestamp) AS Timestamp, ServiceName, SpanKind, histogramState(20)(Duration) AS histogram__Duration FROM default.otel_traces GROUP BY Timestamp, ServiceName, SpanKind; ``` </details> Then you'll need to configure the materialized views in your source settings: <details> <summary>Source Configuration (should auto-infer when MVs are selected)</summary> <img width="949" height="1011" alt="Screenshot 2025-12-19 at 10 26 54 AM" src="https://github.com/user-attachments/assets/fc46a1b9-de8b-4b95-a8ef-ba5fee905685" /> </details>
1 parent 99e7ce2 commit a5a04aa

27 files changed

+3777
-117
lines changed

‎.changeset/olive-snails-shake.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
"@hyperdx/common-utils": minor
3+
"@hyperdx/api": minor
4+
"@hyperdx/app": minor
5+
---
6+
7+
feat: Add materialized view support (Beta)

‎packages/api/src/models/source.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,9 @@ export const Source = mongoose.model<ISource>(
7272
highlightedRowAttributeExpressions: {
7373
type: mongoose.Schema.Types.Array,
7474
},
75+
materializedViews: {
76+
type: mongoose.Schema.Types.Array,
77+
},
7578

7679
metricTables: {
7780
type: {

‎packages/app/src/DBDashboardPage.tsx‎

Lines changed: 71 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ import {
7575
} from '@/dashboard';
7676

7777
import DBSqlRowTableWithSideBar from './components/DBSqlRowTableWithSidebar';
78+
import MVOptimizationIndicator from './components/MaterializedViews/MVOptimizationIndicator';
7879
import OnboardingModal from './components/OnboardingModal';
7980
import { Tags } from './components/Tags';
8081
import useDashboardFilters from './hooks/useDashboardFilters';
@@ -256,66 +257,77 @@ const Tile = forwardRef(
256257
<Text size="sm" ms="xs">
257258
{chart.config.name}
258259
</Text>
259-
{hovered ? (
260-
<Flex gap="0px">
261-
{(chart.config.displayType === DisplayType.Line ||
262-
chart.config.displayType === DisplayType.StackedBar) && (
263-
<Indicator
264-
size={alert?.state === AlertState.OK ? 6 : 8}
265-
zIndex={1}
266-
color={alertIndicatorColor}
267-
processing={alert?.state === AlertState.ALERT}
268-
label={!alert && <span className="fs-8">+</span>}
269-
mr={4}
270-
>
271-
<Tooltip label={alertTooltip} withArrow>
272-
<Button
273-
data-testid={`tile-alerts-button-${chart.id}`}
274-
variant="subtle"
275-
color="gray"
276-
size="xxs"
277-
onClick={onEditClick}
278-
>
279-
<IconBell size={16} />
280-
</Button>
281-
</Tooltip>
282-
</Indicator>
283-
)}
260+
<Group>
261+
{hovered ? (
262+
<Flex gap="0px" onMouseDown={e => e.stopPropagation()}>
263+
{(chart.config.displayType === DisplayType.Line ||
264+
chart.config.displayType === DisplayType.StackedBar) && (
265+
<Indicator
266+
size={alert?.state === AlertState.OK ? 6 : 8}
267+
zIndex={1}
268+
color={alertIndicatorColor}
269+
processing={alert?.state === AlertState.ALERT}
270+
label={!alert && <span className="fs-8">+</span>}
271+
mr={4}
272+
>
273+
<Tooltip label={alertTooltip} withArrow>
274+
<Button
275+
data-testid={`tile-alerts-button-${chart.id}`}
276+
variant="subtle"
277+
color="gray"
278+
size="xxs"
279+
onClick={onEditClick}
280+
>
281+
<IconBell size={16} />
282+
</Button>
283+
</Tooltip>
284+
</Indicator>
285+
)}
284286

285-
<Button
286-
data-testid={`tile-duplicate-button-${chart.id}`}
287-
variant="subtle"
288-
color="gray"
289-
size="xxs"
290-
onClick={onDuplicateClick}
291-
title="Duplicate"
292-
>
293-
<IconCopy size={14} />
294-
</Button>
295-
<Button
296-
data-testid={`tile-edit-button-${chart.id}`}
297-
variant="subtle"
298-
color="gray"
299-
size="xxs"
300-
onClick={onEditClick}
301-
title="Edit"
302-
>
303-
<IconPencil size={14} />
304-
</Button>
305-
<Button
306-
data-testid={`tile-delete-button-${chart.id}`}
307-
variant="subtle"
308-
color="gray"
309-
size="xxs"
310-
onClick={onDeleteClick}
311-
title="Delete"
312-
>
313-
<IconTrash size={14} />
314-
</Button>
315-
</Flex>
316-
) : (
317-
<Box h={22} />
318-
)}
287+
<Button
288+
data-testid={`tile-duplicate-button-${chart.id}`}
289+
variant="subtle"
290+
color="gray"
291+
size="xxs"
292+
onClick={onDuplicateClick}
293+
title="Duplicate"
294+
>
295+
<IconCopy size={14} />
296+
</Button>
297+
<Button
298+
data-testid={`tile-edit-button-${chart.id}`}
299+
variant="subtle"
300+
color="gray"
301+
size="xxs"
302+
onClick={onEditClick}
303+
title="Edit"
304+
>
305+
<IconPencil size={14} />
306+
</Button>
307+
<Button
308+
data-testid={`tile-delete-button-${chart.id}`}
309+
variant="subtle"
310+
color="gray"
311+
size="xxs"
312+
onClick={onDeleteClick}
313+
title="Delete"
314+
>
315+
<IconTrash size={14} />
316+
</Button>
317+
</Flex>
318+
) : (
319+
<Box h={22} />
320+
)}
321+
{source?.materializedViews?.length && queriedConfig && (
322+
<Box onMouseDown={e => e.stopPropagation()}>
323+
<MVOptimizationIndicator
324+
config={queriedConfig}
325+
source={source}
326+
variant="icon"
327+
/>
328+
</Box>
329+
)}
330+
</Group>
319331
</div>
320332
<div
321333
className="fs-7 text-muted flex-grow-1 overflow-hidden"

‎packages/app/src/DBSearchPage.tsx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -518,6 +518,7 @@ function useSearchedConfigToChartConfig({
518518
data: {
519519
select: select || (sourceObj.defaultTableSelectExpression ?? ''),
520520
from: sourceObj.from,
521+
source: sourceObj.id,
521522
...(sourceObj.tableFilterExpression != null
522523
? {
523524
filters: [

‎packages/app/src/ServicesDashboardPage.tsx‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,7 @@ function ServiceSelectControlled({
138138
const { expressions } = useServiceDashboardExpressions({ source });
139139

140140
const queriedConfig = {
141+
source: source?.id,
141142
timestampValueExpression: source?.timestampValueExpression || '',
142143
from: {
143144
databaseName: source?.from.databaseName || '',
@@ -250,6 +251,7 @@ export function EndpointLatencyChart({
250251
'avg_duration_ns',
251252
]}
252253
config={{
254+
source: source.id,
253255
...pick(source, [
254256
'timestampValueExpression',
255257
'connection',
@@ -303,6 +305,7 @@ export function EndpointLatencyChart({
303305
) : (
304306
<DBHistogramChart
305307
config={{
308+
source: source.id,
306309
...pick(source, [
307310
'timestampValueExpression',
308311
'connection',
@@ -311,9 +314,15 @@ export function EndpointLatencyChart({
311314
where: appliedConfig.where || '',
312315
whereLanguage: appliedConfig.whereLanguage || 'sql',
313316
select: [
317+
{
318+
alias: 'data_nanoseconds',
319+
aggFn: 'histogram',
320+
level: 20,
321+
valueExpression: expressions.duration,
322+
},
314323
{
315324
alias: 'data',
316-
valueExpression: `histogram(20)(${expressions.durationInMillis})`,
325+
valueExpression: `arrayMap(bin -> (bin.1 / ${expressions.durationDivisorForMillis}, bin.2 / ${expressions.durationDivisorForMillis}, bin.3), data_nanoseconds)`,
317326
},
318327
],
319328
filters: [
@@ -361,6 +370,7 @@ function HttpTab({
361370
if (!source || !expressions) return null;
362371
if (reqChartType === 'overall') {
363372
return {
373+
source: source.id,
364374
...pick(source, ['timestampValueExpression', 'connection', 'from']),
365375
where: appliedConfig.where || '',
366376
whereLanguage: appliedConfig.whereLanguage || 'sql',
@@ -392,6 +402,7 @@ function HttpTab({
392402
return {
393403
timestampValueExpression: 'series_time_bucket',
394404
connection: source.connection,
405+
source: source.id,
395406
with: [
396407
{
397408
name: 'error_series',
@@ -557,6 +568,7 @@ function HttpTab({
557568
<DBTimeChart
558569
sourceId={source.id}
559570
config={{
571+
source: source.id,
560572
...pick(source, [
561573
'timestampValueExpression',
562574
'connection',
@@ -601,6 +613,7 @@ function HttpTab({
601613
'error_requests',
602614
]}
603615
config={{
616+
source: source.id,
604617
...pick(source, [
605618
'timestampValueExpression',
606619
'connection',
@@ -726,6 +739,7 @@ function HttpTab({
726739
'error_count',
727740
]}
728741
config={{
742+
source: source.id,
729743
...pick(source, [
730744
'timestampValueExpression',
731745
'connection',
@@ -951,6 +965,7 @@ function DatabaseTab({
951965
dateRange: searchedTimeRange,
952966
timestampValueExpression: 'series_time_bucket',
953967
connection: source.connection,
968+
source: source.id,
954969
} satisfies ChartConfigWithDateRange;
955970
}, [appliedConfig, expressions, searchedTimeRange, source]);
956971

@@ -1071,6 +1086,7 @@ function DatabaseTab({
10711086
dateRange: searchedTimeRange,
10721087
timestampValueExpression: 'series_time_bucket',
10731088
connection: source.connection,
1089+
source: source.id,
10741090
} satisfies ChartConfigWithDateRange;
10751091
}, [appliedConfig, expressions, searchedTimeRange, source]);
10761092

@@ -1149,6 +1165,7 @@ function DatabaseTab({
11491165
'p50_duration_ns',
11501166
]}
11511167
config={{
1168+
source: source.id,
11521169
...pick(source, [
11531170
'timestampValueExpression',
11541171
'connection',
@@ -1229,6 +1246,7 @@ function DatabaseTab({
12291246
'p50_duration_ns',
12301247
]}
12311248
config={{
1249+
source: source.id,
12321250
...pick(source, [
12331251
'timestampValueExpression',
12341252
'connection',
@@ -1327,6 +1345,7 @@ function ErrorsTab({
13271345
<DBTimeChart
13281346
sourceId={source.id}
13291347
config={{
1348+
source: source.id,
13301349
...pick(source, [
13311350
'timestampValueExpression',
13321351
'connection',

0 commit comments

Comments
 (0)