Skip to content

Commit 18327dc

Browse files
[Streams] Show field data types in Processing table and step editor (#241825)
Closes #240954 ## Summary Added field type icons throughout the Streams Processing UI to improve field identification and user experience. Icons are sourced from DataView field types (ES types) for reliability and are cached across components using React Query. ### Before <img width="1166" height="497" alt="image" src="https://github.com/user-attachments/assets/461a3ac1-b0ec-479c-be29-1653ff3a1a68" /> ### After <img width="1782" height="478" alt="image" src="https://github.com/user-attachments/assets/694baed0-0ab9-459b-bc6c-b796cd764d3b" /> ## Changes - Created `useStreamDataViewFieldTypes` hook using `@kbn/react-query` for shared DataView fetching - Implemented 5-minute cache TTL to minimize duplicate requests - Added `QueryClientProvider` to app root for React Query support - Caching the DataView enabled a single request to be used through all the components needed ### Testing - Updated `processor_field_selector.test.tsx` with mocks for new hooks - Updated `field_selector.test.tsx` with coverage for icon rendering - Added test cases for icon display with/without type information ```bash yarn test:jest x-pack/platform/plugins/shared/streams_app/public/components/data_management/shared/field_selector.test.tsx yarn test:jest x-pack/platform/plugins/shared/streams_app/public/components/data_management/stream_detail_enrichment/steps/blocks/action/processor_field_selector.test.tsx ``` ## Possible improvements ### Graceful loading state for field type icons Currently, field type icons appear immediately once the DataView data is fetched, which can cause a slight visual "pop-in" effect. **Considered approaches:** **Icon-level loading** (with spinners per column) - Table renders immediately with loading spinners in each column header - Icons replace spinners when DataView loads - **Pros**: Content-first, users can interact with data immediately - **Cons**: Visual noise from multiple spinners **Table-level loading** (wait for DataView) - Single loading spinner until DataView request completes - Table renders complete with icons - **Pros**: Polished, complete appearance on first render, no layout shifts - **Cons**: Blocks content access during DataView fetch ### Handling flattened nested field types **Current behavior**: All fields display field type icons. Fields without explicit DataView mappings show a gray question mark icon (undefined type). While this provides consistent visual feedback, it's not semantically accurate for flattened nested fields. **The challenge**: Some Elasticsearch field types (like `geo_point`) get flattened into sub-fields in document responses, which lack direct mappings in the DataView. **Example**: - DataView mapping: `source.geo.location` → `geo_point` - Flattened document fields: `source.geo.location.lat`, `source.geo.location.lon` → **no mapping** → show as "undefined" **Current solution**: These fields display with a gray question mark icon, indicating they lack type information. This is not ideal, as `.lat` and `.lon` are semantically numeric fields derived from a known parent type.
1 parent 4d969df commit 18327dc

File tree

9 files changed

+292
-24
lines changed

9 files changed

+292
-24
lines changed

x-pack/platform/plugins/shared/streams_app/public/components/data_management/shared/field_selector.test.tsx

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,10 @@ import { render, screen } from '@testing-library/react';
1010
import userEvent from '@testing-library/user-event';
1111
import { FieldSelector } from './field_selector';
1212

13+
jest.mock('@kbn/react-field', () => ({
14+
FieldIcon: ({ type }: { type: string }) => <span data-test-subj={`field-icon-${type}`} />,
15+
}));
16+
1317
describe('FieldSelector', () => {
1418
const mockSuggestions = [
1519
{ name: '@timestamp' },
@@ -151,4 +155,61 @@ describe('FieldSelector', () => {
151155
expect(input).toHaveAttribute('disabled');
152156
});
153157
});
158+
159+
describe('Field Type Icons', () => {
160+
it('renders field type icons when suggestions include type information', async () => {
161+
const suggestionsWithTypes = [
162+
{ name: '@timestamp', type: 'date' },
163+
{ name: 'log.level', type: 'keyword' },
164+
{ name: 'message', type: 'text' },
165+
];
166+
167+
render(<FieldSelector {...defaultProps} suggestions={suggestionsWithTypes} />);
168+
169+
const toggleButton = screen.getByTestId('comboBoxToggleListButton');
170+
await userEvent.click(toggleButton);
171+
172+
// Verify that field icons are rendered for fields with types
173+
expect(screen.getByTestId('field-icon-date')).toBeInTheDocument();
174+
expect(screen.getByTestId('field-icon-keyword')).toBeInTheDocument();
175+
expect(screen.getByTestId('field-icon-text')).toBeInTheDocument();
176+
});
177+
178+
it('renders unknown icons for fields without type information', async () => {
179+
const suggestionsWithoutTypes = [{ name: '@timestamp' }, { name: 'log.level' }];
180+
181+
render(<FieldSelector {...defaultProps} suggestions={suggestionsWithoutTypes} />);
182+
183+
const toggleButton = screen.getByTestId('comboBoxToggleListButton');
184+
await userEvent.click(toggleButton);
185+
186+
// Verify that unknown field icons are rendered for fields without types
187+
expect(screen.getAllByTestId('field-icon-unknown')).toHaveLength(2);
188+
});
189+
190+
it('shows icon for selected field when type is available', () => {
191+
const suggestionsWithTypes = [
192+
{ name: 'log.level', type: 'keyword' },
193+
{ name: 'message', type: 'text' },
194+
];
195+
196+
render(
197+
<FieldSelector {...defaultProps} value="log.level" suggestions={suggestionsWithTypes} />
198+
);
199+
200+
// Icon should be visible in the selected value
201+
expect(screen.getByTestId('field-icon-keyword')).toBeInTheDocument();
202+
});
203+
204+
it('shows unknown icon for selected field when type is not available', () => {
205+
const suggestionsWithoutTypes = [{ name: 'log.level' }, { name: 'message' }];
206+
207+
render(
208+
<FieldSelector {...defaultProps} value="log.level" suggestions={suggestionsWithoutTypes} />
209+
);
210+
211+
// Unknown icon should be visible in the selected value
212+
expect(screen.getByTestId('field-icon-unknown')).toBeInTheDocument();
213+
});
214+
});
154215
});

x-pack/platform/plugins/shared/streams_app/public/components/data_management/shared/field_selector.tsx

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import React, { useCallback, useMemo } from 'react';
99
import { EuiFormRow, EuiComboBox } from '@elastic/eui';
1010
import { i18n } from '@kbn/i18n';
1111
import type { EuiComboBoxOptionOption } from '@elastic/eui';
12+
import { FieldIcon } from '@kbn/react-field';
1213

1314
export interface FieldSuggestion {
1415
name: string;
@@ -56,6 +57,9 @@ export const FieldSelector = ({
5657
suggestions.map((suggestion) => ({
5758
label: suggestion.name,
5859
value: suggestion.name,
60+
prepend: (
61+
<FieldIcon type={suggestion.type || 'unknown'} size="s" className="eui-alignMiddle" />
62+
),
5963
'data-test-subj': `field-suggestion-${suggestion.name}`,
6064
})),
6165
[suggestions]
@@ -65,8 +69,26 @@ export const FieldSelector = ({
6569
if (!value) return [];
6670

6771
const matchingSuggestion = comboBoxOptions.find((option) => option.value === value);
68-
return matchingSuggestion ? [matchingSuggestion] : [{ label: value, value }];
69-
}, [value, comboBoxOptions]);
72+
if (matchingSuggestion) {
73+
return [matchingSuggestion];
74+
}
75+
76+
// For custom values not in suggestions, try to find the type
77+
const suggestionWithType = suggestions.find((s) => s.name === value);
78+
return [
79+
{
80+
label: value,
81+
value,
82+
prepend: (
83+
<FieldIcon
84+
type={suggestionWithType?.type || 'unknown'}
85+
size="s"
86+
className="eui-alignMiddle"
87+
/>
88+
),
89+
},
90+
];
91+
}, [value, comboBoxOptions, suggestions]);
7092

7193
const handleSelectionChange = useCallback(
7294
(newSelectedOptions: Array<EuiComboBoxOptionOption<string>>) => {

x-pack/platform/plugins/shared/streams_app/public/components/data_management/shared/preview_flyout.tsx

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,8 @@
88
import React from 'react';
99
import { UnifiedDocViewerFlyout } from '@kbn/unified-doc-viewer-plugin/public';
1010
import type { DataTableRecord } from '@kbn/discover-utils';
11-
import useAsync from 'react-use/lib/useAsync';
1211
import type { DocViewsRegistry } from '@kbn/unified-doc-viewer';
12+
import type { DataView } from '@kbn/data-views-plugin/common';
1313
import { useKibana } from '../../../hooks/use_kibana';
1414

1515
export const FLYOUT_WIDTH_KEY = 'streamsEnrichment:flyoutWidth';
@@ -24,22 +24,16 @@ export const PreviewFlyout = ({
2424
setExpandedDoc,
2525
docViewsRegistry,
2626
streamName,
27+
streamDataView,
2728
}: {
2829
currentDoc?: DataTableRecordWithIndex;
2930
hits: DataTableRecordWithIndex[];
3031
setExpandedDoc: (doc?: DataTableRecordWithIndex) => void;
3132
docViewsRegistry: DocViewsRegistry;
3233
streamName: string;
34+
streamDataView?: DataView;
3335
}) => {
34-
const { core, dependencies } = useKibana();
35-
const { data } = dependencies.start;
36-
37-
const { value: streamDataView } = useAsync(() =>
38-
data.dataViews.create({
39-
title: streamName,
40-
timeFieldName: '@timestamp',
41-
})
42-
);
36+
const { core } = useKibana();
4337

4438
return (
4539
currentDoc &&

x-pack/platform/plugins/shared/streams_app/public/components/data_management/shared/preview_table.tsx

Lines changed: 43 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,14 @@ import {
1616
EuiButtonIcon,
1717
EuiDataGrid,
1818
EuiFlexGroup,
19+
EuiFlexItem,
1920
EuiToolTip,
2021
useEuiTheme,
2122
} from '@elastic/eui';
2223
import { i18n } from '@kbn/i18n';
2324
import type { SampleDocument } from '@kbn/streams-schema';
2425
import ColumnHeaderTruncateContainer from '@kbn/unified-data-table/src/components/column_header_truncate_container';
26+
import { FieldIcon } from '@kbn/react-field';
2527
import React, { useMemo, useState, useCallback, createContext, useContext } from 'react';
2628
import type {
2729
IgnoredField,
@@ -93,6 +95,7 @@ export function PreviewTable({
9395
showLeadingControlColumns = true,
9496
originalSamples,
9597
cellActions,
98+
dataViewFieldTypes,
9699
}: {
97100
documents: SampleDocument[] | DocumentWithIgnoredFields[];
98101
displayColumns?: string[];
@@ -112,8 +115,27 @@ export function PreviewTable({
112115
showLeadingControlColumns?: boolean;
113116
originalSamples?: SampleDocumentWithUIAttributes[];
114117
cellActions?: EuiDataGridColumnCellAction[];
118+
dataViewFieldTypes?: Array<{ name: string; type: string; esType?: string }>;
115119
}) {
116120
const { euiTheme: theme } = useEuiTheme();
121+
122+
// Create a map of field names to their ES types for quick lookup from DataView
123+
const fieldTypeMap = useMemo(() => {
124+
const typeMap = new Map<string, string>();
125+
126+
if (dataViewFieldTypes && dataViewFieldTypes.length > 0) {
127+
dataViewFieldTypes.forEach((field) => {
128+
// Use esType if available (more specific), otherwise use type
129+
const fieldType = field.esType || field.type;
130+
if (fieldType) {
131+
typeMap.set(field.name, fieldType);
132+
}
133+
});
134+
}
135+
136+
return typeMap;
137+
}, [dataViewFieldTypes]);
138+
117139
// Determine canonical column order
118140
const canonicalColumnOrder = useMemo(() => {
119141
const cols = new Set<string>();
@@ -255,12 +277,22 @@ export function PreviewTable({
255277
return [...acc, '.', <wbr key={index} />, part];
256278
}, [] as React.ReactNode[]);
257279

280+
// Get the field type from the map
281+
const fieldType = fieldTypeMap.get(column);
282+
258283
return {
259284
id: column,
260285
display: (
261-
<ColumnHeaderTruncateContainer wordBreak="normal">
262-
{interleavedColumnParts}
263-
</ColumnHeaderTruncateContainer>
286+
<EuiFlexGroup gutterSize="s" alignItems="center" responsive={false}>
287+
<EuiFlexItem grow={false}>
288+
<FieldIcon type={fieldType || 'unknown'} size="s" />
289+
</EuiFlexItem>
290+
<EuiFlexItem>
291+
<ColumnHeaderTruncateContainer wordBreak="normal">
292+
{interleavedColumnParts}
293+
</ColumnHeaderTruncateContainer>
294+
</EuiFlexItem>
295+
</EuiFlexGroup>
264296
),
265297
actions:
266298
Boolean(setVisibleColumns) || Boolean(setSorting)
@@ -276,7 +308,14 @@ export function PreviewTable({
276308
cellActions,
277309
};
278310
});
279-
}, [cellActions, canonicalColumnOrder, setSorting, setVisibleColumns, columnWidths]);
311+
}, [
312+
cellActions,
313+
canonicalColumnOrder,
314+
fieldTypeMap,
315+
setSorting,
316+
setVisibleColumns,
317+
columnWidths,
318+
]);
280319

281320
return (
282321
<EuiDataGrid

x-pack/platform/plugins/shared/streams_app/public/components/data_management/stream_detail_enrichment/processor_condition_editor.tsx

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,12 @@
55
* 2.0.
66
*/
77

8-
import React from 'react';
8+
import React, { useMemo } from 'react';
99
import { useEnrichmentFieldSuggestions } from '../../../hooks/use_field_suggestions';
1010
import type { ConditionEditorProps } from '../shared/condition_editor';
1111
import { ConditionEditor } from '../shared/condition_editor';
12+
import { useStreamDataViewFieldTypes } from '../../../hooks/use_stream_data_view_field_types';
13+
import { useSimulatorSelector } from './state_management/stream_enrichment_state_machine/use_stream_enrichment';
1214

1315
export type ProcessorConditionEditorProps = Omit<
1416
ConditionEditorProps,
@@ -17,5 +19,20 @@ export type ProcessorConditionEditorProps = Omit<
1719

1820
export function ProcessorConditionEditorWrapper(props: ProcessorConditionEditorProps) {
1921
const fieldSuggestions = useEnrichmentFieldSuggestions();
20-
return <ConditionEditor status="enabled" {...props} fieldSuggestions={fieldSuggestions} />;
22+
const streamName = useSimulatorSelector((state) => state.context.streamName);
23+
24+
// Fetch DataView field types with automatic caching via React Query
25+
const { fieldTypeMap } = useStreamDataViewFieldTypes(streamName);
26+
27+
// Enrich field suggestions with types from DataView
28+
const enrichedFieldSuggestions = useMemo(() => {
29+
return fieldSuggestions.map((suggestion) => ({
30+
...suggestion,
31+
type: fieldTypeMap.get(suggestion.name),
32+
}));
33+
}, [fieldSuggestions, fieldTypeMap]);
34+
35+
return (
36+
<ConditionEditor status="enabled" {...props} fieldSuggestions={enrichedFieldSuggestions} />
37+
);
2138
}

x-pack/platform/plugins/shared/streams_app/public/components/data_management/stream_detail_enrichment/processor_outcome_preview.tsx

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ import type { GrokProcessor } from '@kbn/streamlang';
2323
import { isActionBlock } from '@kbn/streamlang';
2424
import { useDocViewerSetup } from '../../../hooks/use_doc_viewer_setup';
2525
import { useDocumentExpansion } from '../../../hooks/use_document_expansion';
26+
import { useStreamDataViewFieldTypes } from '../../../hooks/use_stream_data_view_field_types';
2627
import { getPercentageFormatter } from '../../../util/formatters';
2728
import type { PreviewDocsFilterOption } from './state_management/simulation_state_machine';
2829
import {
@@ -173,8 +174,13 @@ const PreviewDocumentsGroupBy = () => {
173174
};
174175

175176
const OutcomePreviewTable = ({ previewDocuments }: { previewDocuments: FlattenRecord[] }) => {
177+
// Original logic - used for column determination
176178
const detectedFields = useSimulatorSelector((state) => state.context.simulation?.detected_fields);
177179
const streamName = useSimulatorSelector((state) => state.context.streamName);
180+
181+
// Fetch DataView field types with automatic caching via React Query
182+
const { fieldTypes: dataViewFieldTypes, dataView: streamDataView } =
183+
useStreamDataViewFieldTypes(streamName);
178184
const previewDocsFilter = useSimulatorSelector((state) => state.context.previewDocsFilter);
179185
const previewColumnsSorting = useSimulatorSelector(
180186
(state) => state.context.previewColumnsSorting
@@ -392,6 +398,7 @@ const OutcomePreviewTable = ({ previewDocuments }: { previewDocuments: FlattenRe
392398
setSorting={setPreviewColumnsSorting}
393399
columnOrderHint={previewColumnsOrder}
394400
renderCellValue={renderCellValue}
401+
dataViewFieldTypes={dataViewFieldTypes}
395402
/>
396403
</RowSelectionContext.Provider>
397404
<DocViewerContext.Provider value={docViewerContext}>
@@ -401,6 +408,7 @@ const OutcomePreviewTable = ({ previewDocuments }: { previewDocuments: FlattenRe
401408
setExpandedDoc={setExpandedDoc}
402409
docViewsRegistry={docViewsRegistry}
403410
streamName={streamName}
411+
streamDataView={streamDataView}
404412
/>
405413
</DocViewerContext.Provider>
406414
</>

0 commit comments

Comments
 (0)