Skip to content

Commit e8ef633

Browse files
flash1293kibanamachineKerry350
authored
🌊 Streams: Classic streams field mappings (#228415)
This PR adds the ability to map fields in classic streams. This fits really well into the existing flow - it's basically just enabling the schema editor for classic streams. Under the hood this is using the new data stream mappings API instead of changing component templates. Changes that are compatible with the write index mapping are applied directly, otherwise a rollover is performed automatically. Schema editor for a classic stream. Fields mapped via streams are marked as "mapped". Fields that are mapped via the underlying index template are shown as "unmanaged", with a type that comes from field caps (the tooltip is calling it out). <img width="1248" height="647" alt="Screenshot 2025-07-17 at 17 23 40" src="https://github.com/user-attachments/assets/c8e20652-6394-43e2-b119-b42aa78418a6" /> <img width="743" height="223" alt="Screenshot 2025-07-17 at 17 26 53" src="https://github.com/user-attachments/assets/e2fcb6f7-6e5d-46da-b2e5-1698b31b4a1d" /> Previously it was called "unmapped" - it's unmapped on the streams level, but not actually unmapped in the underlying data. I'm sure that will be confusing. I'm not sure whether "unmanaged" is good enough, happy to hear other opinions. In the processing view, the fields are shown in the same way <img width="1247" height="411" alt="Screenshot 2025-07-17 at 17 28 05" src="https://github.com/user-attachments/assets/cc251181-4bfd-44a4-952d-6ce2326f0159" /> It's possible that a mapping isn't compatible with what's defined in the template of a stream. In this case an toast error is shown when the user is trying to save (hopefully it explains well enough what's going on, it's basically what comes from Elasticsearch): <img width="399" height="314" alt="Screenshot 2025-07-17 at 17 46 58" src="https://github.com/user-attachments/assets/a7ec64c2-40dd-4045-b499-7d77a11c4a0a" /> ## Notes ### Discover I think the types were incomplete here - the boolean flag in question is definitely returned in these objects, that's why adjusted upstream. ### Simulation reset too early A little drive-by fix: If the saving of the processing changes failed, the simulation would have been reset already, which makes us loose the field mappings that are staged there. This PR moves the reset to the success state transition, so everything sticks around on failure so the user can fix and try again. ### Rollover logic If the mapping is changed, the data stream is rolled over. ### Error handling If setting the mapping on the data stream level doesn't work, we bail out. This should trigger a resync with #227738 , which will bring the system into a valid state again. However, this is considered a 500 instead of a 400. By using the dry run in the validation phase we can catch it early, but not sure whether it's worth it... (it also takes longer of course). ### Sourcing fields Fetching the fields for the schema editor, it's now also sourcing them from the field caps along with the definition and unmapped fields API, to give as complete of a picture as possible. ### Field simulation Had to adjust the field simulation logic a bit, I think I actually simplified it successfully so it works the same for wired and classic streams, but worth taking another look. --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Kerry Gallagher <[email protected]> Co-authored-by: Kerry Gallagher <[email protected]>
1 parent dcdc9d7 commit e8ef633

File tree

30 files changed

+1465
-292
lines changed

30 files changed

+1465
-292
lines changed

‎oas_docs/bundle.json‎

Lines changed: 216 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55583,7 +55583,114 @@
5558355583
"properties": {
5558455584
"classic": {
5558555585
"additionalProperties": false,
55586-
"properties": {},
55586+
"properties": {
55587+
"field_overrides": {
55588+
"additionalProperties": {
55589+
"allOf": [
55590+
{
55591+
"additionalProperties": {
55592+
"anyOf": [
55593+
{
55594+
"anyOf": [
55595+
{
55596+
"type": "string"
55597+
},
55598+
{
55599+
"type": "number"
55600+
},
55601+
{
55602+
"type": "boolean"
55603+
},
55604+
{
55605+
"enum": [
55606+
"null"
55607+
],
55608+
"nullable": true
55609+
},
55610+
{
55611+
"not": {}
55612+
}
55613+
]
55614+
},
55615+
{
55616+
"items": {
55617+
"anyOf": [
55618+
{
55619+
"type": "string"
55620+
},
55621+
{
55622+
"type": "number"
55623+
},
55624+
{
55625+
"type": "boolean"
55626+
},
55627+
{
55628+
"enum": [
55629+
"null"
55630+
],
55631+
"nullable": true
55632+
},
55633+
{
55634+
"not": {}
55635+
}
55636+
]
55637+
},
55638+
"type": "array"
55639+
},
55640+
{}
55641+
]
55642+
},
55643+
"type": "object"
55644+
},
55645+
{
55646+
"anyOf": [
55647+
{
55648+
"additionalProperties": false,
55649+
"properties": {
55650+
"format": {
55651+
"minLength": 1,
55652+
"type": "string"
55653+
},
55654+
"type": {
55655+
"enum": [
55656+
"keyword",
55657+
"match_only_text",
55658+
"long",
55659+
"double",
55660+
"date",
55661+
"boolean",
55662+
"ip"
55663+
],
55664+
"type": "string"
55665+
}
55666+
},
55667+
"required": [
55668+
"type"
55669+
],
55670+
"type": "object"
55671+
},
55672+
{
55673+
"additionalProperties": false,
55674+
"properties": {
55675+
"type": {
55676+
"enum": [
55677+
"system"
55678+
],
55679+
"type": "string"
55680+
}
55681+
},
55682+
"required": [
55683+
"type"
55684+
],
55685+
"type": "object"
55686+
}
55687+
]
55688+
}
55689+
]
55690+
},
55691+
"type": "object"
55692+
}
55693+
},
5558755694
"type": "object"
5558855695
}
5558955696
},
@@ -65017,7 +65124,114 @@
6501765124
"properties": {
6501865125
"classic": {
6501965126
"additionalProperties": false,
65020-
"properties": {},
65127+
"properties": {
65128+
"field_overrides": {
65129+
"additionalProperties": {
65130+
"allOf": [
65131+
{
65132+
"additionalProperties": {
65133+
"anyOf": [
65134+
{
65135+
"anyOf": [
65136+
{
65137+
"type": "string"
65138+
},
65139+
{
65140+
"type": "number"
65141+
},
65142+
{
65143+
"type": "boolean"
65144+
},
65145+
{
65146+
"enum": [
65147+
"null"
65148+
],
65149+
"nullable": true
65150+
},
65151+
{
65152+
"not": {}
65153+
}
65154+
]
65155+
},
65156+
{
65157+
"items": {
65158+
"anyOf": [
65159+
{
65160+
"type": "string"
65161+
},
65162+
{
65163+
"type": "number"
65164+
},
65165+
{
65166+
"type": "boolean"
65167+
},
65168+
{
65169+
"enum": [
65170+
"null"
65171+
],
65172+
"nullable": true
65173+
},
65174+
{
65175+
"not": {}
65176+
}
65177+
]
65178+
},
65179+
"type": "array"
65180+
},
65181+
{}
65182+
]
65183+
},
65184+
"type": "object"
65185+
},
65186+
{
65187+
"anyOf": [
65188+
{
65189+
"additionalProperties": false,
65190+
"properties": {
65191+
"format": {
65192+
"minLength": 1,
65193+
"type": "string"
65194+
},
65195+
"type": {
65196+
"enum": [
65197+
"keyword",
65198+
"match_only_text",
65199+
"long",
65200+
"double",
65201+
"date",
65202+
"boolean",
65203+
"ip"
65204+
],
65205+
"type": "string"
65206+
}
65207+
},
65208+
"required": [
65209+
"type"
65210+
],
65211+
"type": "object"
65212+
},
65213+
{
65214+
"additionalProperties": false,
65215+
"properties": {
65216+
"type": {
65217+
"enum": [
65218+
"system"
65219+
],
65220+
"type": "string"
65221+
}
65222+
},
65223+
"required": [
65224+
"type"
65225+
],
65226+
"type": "object"
65227+
}
65228+
]
65229+
}
65230+
]
65231+
},
65232+
"type": "object"
65233+
}
65234+
},
6502165235
"type": "object"
6502265236
}
6502365237
},

0 commit comments

Comments
 (0)