Skip to content

Add restrictions for synthetic _source #1622

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jan 21, 2025
Merged

Add restrictions for synthetic _source #1622

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9312207
Add restrictions for synthetic
kilfoyle Jan 10, 2025
f7d5f69
Fixes for Martijn's review
kilfoyle Jan 20, 2025
5a30d2c
Remove note about synthetic _source not supported on TSDS
kilfoyle Jan 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 24 additions & 21 deletions docs/en/ingest-management/data-streams.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -691,22 +691,6 @@ You can modify your pipeline API request as needed to apply custom processing at
Refer to <<data-streams-pipelines>> to learn more.


















[[data-streams-advanced-features]]
== Enabling and disabling advanced indexing features for {fleet}-managed data streams

Expand All @@ -720,6 +704,7 @@ Refer to <<data-streams-pipelines>> to learn more.
* link:{ref}/mapping-source-field.html#synthetic-source[Synthetic `_source`]

These features can be enabled and disabled for {fleet}-managed data streams by using the index template API and a few key settings.
Note that in versions 8.17.0 and later, Synthetic `_source` requires an Enterprise license. As well, synthetic `_source` is supported only on non-TSDS data streams.

NOTE: If you are already making use of `@custom` component templates for ingest or retention customization (as shown for example in <<data-streams-ilm-tutorial,Tutorial: Customize data retention policies>>), exercise care to ensure you don't overwrite your customizations when making these requests.

Expand Down Expand Up @@ -878,10 +863,11 @@ For example, the following payload disables TSDS on `nginx.stubstatus`:
[[data-streams-advanced-synthetic-enable]]
== Enable synthetic `_source`

Note that synthetic `_source` is supported only on non-TSDS data streams.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Synthetic source is also supported on tsds data streams. I think this line can be removed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @martijnvg. I can remove the line, but from @lucabelluccini's issue: I've tested it on 8.17 and it's impossible to disable Synthetic _source on a TSDS data stream.

For now I've changed it to Note that disabling synthetic _source is supported only on non-TSDS data streams.
Please let me know if that's not what we want.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I commented on the issue: #1618 (comment)

Let's see what went wrong here.


[source,json]
----
PUT _component_template/<NAME>@custom

{
"template": {
"mappings": {
Expand All @@ -891,20 +877,37 @@ PUT _component_template/<NAME>@custom
}
}
}

PUT _component_template/<NAME>@custom
{
"settings": {
"index": {
"mapping": {
"source": {
"mode": "synthetic"
}
}
}
}
}

----

[discrete]
[[data-streams-advanced-synthetic-disable]]
== Disable synthetic `_source`

Note that synthetic `_source` is supported only on non-TSDS data streams.

[source,json]
----
PUT _component_template/<NAME>@custom

{
"template": {
"mappings": {
"_source": {}
"settings": {
"index": {
"mapping": {
"source": {}
}
}
}
}
Expand Down
Loading