Skip to content

Update [email protected] with new GenAI fields #129122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jun 16, 2025
Merged

Update [email protected] with new GenAI fields #129122

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
852d2b8
Update [email protected] with new GenAI fields
eyalkoren Jun 8, 2025
2cc5df4
Update docs/changelog/129122.yaml
eyalkoren Jun 8, 2025
d807638
Merge remote-tracking branch 'upstream/main' into support-double-EcsD…
eyalkoren Jun 15, 2025
72cec7c
Update gen_ai string field mappings
eyalkoren Jun 15, 2025
1fb9af6
Merge branch 'main' into support-double-EcsDynamicTemplatesIT
eyalkoren Jun 16, 2025
851568f
Merge branch 'main' into support-double-EcsDynamicTemplatesIT
eyalkoren Jun 16, 2025
8db3da8
Merge branch 'main' into support-double-EcsDynamicTemplatesIT
eyalkoren Jun 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/changelog/129122.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pr: 129122
summary: Update [email protected] with new GenAI fields
area: Data streams
type: feature
issues: []
39 changes: 39 additions & 0 deletions x-pack/plugin/core/template-resources/src/main/resources/[email protected]
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,45 @@
"match_mapping_type": "object"
}
},
{
"ecs_gen_ai_integers": {
"mapping": {
"type": "integer"
},
"path_match": [
"*gen_ai.request.max_tokens",
"*gen_ai.usage.input_tokens",
"*gen_ai.usage.output_tokens",
"*gen_ai.request.choice.count",
"*gen_ai.request.seed"
]
}
},
{
"ecs_gen_ai_doubles": {
"mapping": {
"type": "double"
},
"path_match": [
"*gen_ai.request.temperature",
"*gen_ai.request.top_k",
"*gen_ai.request.frequency_penalty",
"*gen_ai.request.presence_penalty",
"*gen_ai.request.top_p"
]
}
},
{
"ecs_gen_ai_text": {
"mapping": {
"type": "text"
Copy link
Member

@felixbarny felixbarny Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am also not sure why gen_ai.request.model is text and not keyword...

That's a bit weird indeed. Using text for metric dimensions is quite unusual. Especially since TSDB doesn't support text fields as dimensions.

Copy link
Contributor Author

@eyalkoren eyalkoren Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@susan-shu-c this PR is a followup for elastic/ecs#2475 - it updates our ECS dynamic templates (ecs@mappings) to include the new gen_ai.* fields.
Can you explain the rationale of using text for these two?

Copy link
Member

@susan-shu-c susan-shu-c Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eyalkoren thanks for catching this. Looking through it, I do agree keyword would be better for gen_ai.request.model. I missed it earlier...

Copy link
Member

@susan-shu-c susan-shu-c Jun 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the next steps?
Should I make an update on top of elastic/ecs#2475 with a new PR before you can update this PR?

Copy link
Member

@susan-shu-c susan-shu-c Jun 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thought process on gen_ai.agent.description being text is that it might be 1-2 sentences depending on how verbose it is, say describing a lot of components. Would love to hear what you think as well as your team would be much more aware of best practices!

Copy link
Contributor Author

@eyalkoren eyalkoren Jun 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the next steps?

Yes, it would make more sense if you change it first and we won't have to merge an "erroneous" state of the mappings. Please let me know how long you expect it to get approved and merged though, because until then we have failing test notifications, so if it's going to take a while, we may prefer merging as is and then fixing. Our change should be very quick.

Would love to hear what you think as well as your team would be much more aware of best practices!

My thinking was similar to yours, but what @felixbarny says is that if gen_ai.agent.description is used as metic dimensions, it probably shouldn't be text, because it won't work as expected in TSDB. I see that this field has a semantic conventions counterpart, so the definition there may indicate what it's used for. Its description in your PR says: "Free-form description of the GenAI agent provided by the application", which doesn't sound like something intended to be used as dimension. @felixbarny WDYT?

Copy link
Member

@felixbarny felixbarny Jun 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thinking was similar to yours, but what @felixbarny says is that if gen_ai.agent.description is used as metic dimensions, it probably shouldn't be text, because it won't work as expected in TSDB.

Thinking about it again, there's a dedicated version of the dynamic ECS mappings for TSDB, incorporating the limited field types available: ecs-tsdb@template. So maybe it's not as big of a deal, even if this field is used as a metric attribute. But is it really used as a metric attribute or is it something that would rather be attached to other signals, such as logs and traces? A free-form description does sound a bit curious to add as a metric attribute (aka dimension).

My thought process on gen_ai.agent.description being text is that it might be 1-2 sentences depending on how verbose it is, say describing a lot of components.

The text field type is useful for full-text search use cases where phrase queries are used a lot. It's powerful but also rather expensive from a storage and indexing perspective. Irrespective of the length of the field, I'd only use it if you have concrete use cases for doing a full-text searches on it. If you just want to store the value to provide additional context, I'd recommend the keyword field type. While it has a limit on how many characters it can store in doc_values it falls back to storing the value in a stored field if ignore_above is set, which seems like a good compromise here. You could potentially disable doc_values and enable store, however, you wouldn't be able to benefit from the dictionary encoding (ordinals) and other optimizations, like run-length encoding for shorter descriptions. Also, if you don't intend to filter on that field (with an exact value), you may want to set index to false to save space and ingest time overhead.

Copy link
Member

@susan-shu-c susan-shu-c Jun 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that this field has a semantic conventions counterpart, so the definition there may indicate what it's used for.

The documentation for the OTel field is here, it's of their type "string" so we have to decide ourselves what's best in ECS. I ported the same definition "Free-form description of the GenAI agent provided by the application."

While it has a limit on how many characters it can store in doc_values it falls back to storing the value in a stored field if ignore_above is set, which seems like a good compromise here.

I've created a new PR, could you check if this covers what you mentioned about ignore_above?

elastic/ecs#2489

},
"path_match": [
"*gen_ai.request.model",
"*gen_ai.agent.description"
]
}
},
{
"all_strings_to_keywords": {
"mapping": {
Expand Down
5 changes: 4 additions & 1 deletion ...ugin/stack/src/javaRestTest/java/org/elasticsearch/xpack/stack/EcsDynamicTemplatesIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -376,9 +376,12 @@ private Object generateTestValue(String type) {
case "long" -> {
return randomLong();
}
case "int" -> {
case "integer" -> {
return randomInt();
}
case "double" -> {
return randomDouble();
}
case "float", "scaled_float" -> {
return randomFloat();
}
Expand Down
Loading