Skip to content

[streams][lifecycle] ilm for classic streams #221364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 16 commits into from
Jun 4, 2025
Merged

[streams][lifecycle] ilm for classic streams #221364

merged 16 commits into from
Jun 4, 2025

Conversation

@klacabane
Copy link
Contributor

@klacabane klacabane commented May 23, 2025 •
edited by kibanamachine
Loading

Summary

Enable ILM for classic streams.

We can now overwrite template settings at the data stream level with PUT _data_stream/{name}/_settings so we can enable full lifecycle capabilities on classic stream.

@klacabane
Copy link
Contributor Author

klacabane commented May 26, 2025 •
edited
Loading

This is functionally complete but I'll leave this pending until elastic/elasticsearch#128375 is available since it will simplify the implementation and fix inconsistent configurations. The latter does not break the functionality but break some assertions in api tests

@flash1293
Copy link
Contributor

@klacabane is this blocked by anything?

@klacabane
Copy link
Contributor Author

@flash1293 the 9.1.0 ES snapshot did not contain the prefer_ilm change last time I've tested. I can still test locally with a source build but CI will still fail

@flash1293
Copy link
Contributor

flash1293 commented Jun 2, 2025 •
edited
Loading

Seems like the Kibana ES snapshot promotion is held back by some unrelated ML problem. Let's check back

klacabane
klacabane commented Jun 2, 2025
View reviewed changes
...latform/plugins/shared/streams/server/lib/streams/state_management/streams/unwired_stream.ts
export class UnwiredStream extends StreamActiveRecord<Streams.UnwiredStream.Definition> {
private _processingChanged: boolean = false;
private _lifeCycleChanged: boolean = false;
private _lifecycleChanged: boolean = false;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to align naming

klacabane
klacabane commented Jun 2, 2025
View reviewed changes
x-pack/platform/plugins/shared/streams/server/lib/streams/data_streams/manage_data_streams.ts
// TODO: use client method once available
esClient.transport.request({
method: 'PUT',
path: `/_data_stream/${names.join(',')}/_settings`,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this updates backing indices settings as well

@klacabane klacabane marked this pull request as ready for review June 2, 2025 19:52
@klacabane klacabane requested review from a team as code owners June 2, 2025 19:52
@klacabane klacabane added release_note:skip Skip the PR/issue when compiling release notes Team:obs-ux-logs Observability Logs User Experience Team backport:version Backport to applied version labels Feature:Streams This is the label for the Streams Project v9.1.0 v8.19.0 labels Jun 2, 2025
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

@klacabane klacabane mentioned this pull request Jun 3, 2025
Closed
@elasticmachine
Copy link
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #60 / Observability AI Assistant Functional tests knowledge_base_management/index.spec.ts Knowledge management tab User instruction management cancels editing without saving changes

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
streamsApp 548.1KB 548.0KB -94.0B

History

@klacabane
Copy link
Contributor Author

@flash1293 this is ready for another review. Inherit is now blocked on unwired once the lifecycle has been updated, and the reset to default option is removed from the UI

flash1293
flash1293 approved these changes Jun 4, 2025
View reviewed changes
Copy link
Contributor

@flash1293 flash1293 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works as expected, LGTM.

I added #222440 to MS3 for now since we didn't make a call yet on how important it is.

@klacabane klacabane merged commit 1abb811 into elastic:main Jun 4, 2025
10 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/15444070936

kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Jun 4, 2025
@klacabane @kibanamachine
[streams][lifecycle] ilm for classic streams (elastic#221364)
b90fc96 
## Summary

Enable ILM for classic streams.

We can now overwrite template settings at the data stream level with
`PUT _data_stream/{name}/_settings` so we can enable full lifecycle
capabilities on classic stream.

---------

Co-authored-by: kibanamachine <[email protected]>
(cherry picked from commit 1abb811)
@kibanamachine kibanamachine mentioned this pull request Jun 4, 2025
Merged
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
✅ 8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

zacharyparikh pushed a commit to zacharyparikh/kibana that referenced this pull request Jun 4, 2025
@klacabane @kibanamachine @zacharyparikh
[streams][lifecycle] ilm for classic streams (elastic#221364)
ec2e453 
## Summary

Enable ILM for classic streams.

We can now overwrite template settings at the data stream level with
`PUT _data_stream/{name}/_settings` so we can enable full lifecycle
capabilities on classic stream.

---------

Co-authored-by: kibanamachine <[email protected]>
@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jun 6, 2025
@kibanamachine
Copy link
Contributor

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @klacabane

2 similar comments
@kibanamachine
Copy link
Contributor

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @klacabane

@kibanamachine
Copy link
Contributor

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @klacabane

@klacabane
Copy link
Contributor Author

This backport needs this ES change to be backported first which is WIP

@kibanamachine
Copy link
Contributor

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @klacabane

1 similar comment
@kibanamachine
Copy link
Contributor

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @klacabane

nickpeihl pushed a commit to nickpeihl/kibana that referenced this pull request Jun 12, 2025
@klacabane @kibanamachine @nickpeihl
[streams][lifecycle] ilm for classic streams (elastic#221364)
fa573c8 
## Summary

Enable ILM for classic streams.

We can now overwrite template settings at the data stream level with
`PUT _data_stream/{name}/_settings` so we can enable full lifecycle
capabilities on classic stream.

---------

Co-authored-by: kibanamachine <[email protected]>
@kibanamachine
Copy link
Contributor

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @klacabane

5 similar comments
@kibanamachine
Copy link
Contributor

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @klacabane

@kibanamachine
Copy link
Contributor

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @klacabane

@kibanamachine
Copy link
Contributor

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @klacabane

@kibanamachine
Copy link
Contributor

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @klacabane

@kibanamachine
Copy link
Contributor

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @klacabane

@flash1293
Copy link
Contributor

@masseyke @klacabane where are we with backporting this PR?

@masseyke
Copy link
Member

@masseyke @klacabane where are we with backporting this PR?

The elasticsearch piece is in -- elastic/elasticsearch#129213

kibanamachine added a commit that referenced this pull request Jun 23, 2025
@kibanamachine @klacabane @flash1293
[8.19] [streams][lifecycle] ilm for classic streams (#221364) (#222578)
63020c2 
# Backport

This will backport the following commits from `main` to `8.19`:
- [[streams][lifecycle] ilm for classic streams
(#221364)](#221364)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Kevin
Lacabane","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-06-04T13:48:00Z","message":"[streams][lifecycle]
ilm for classic streams (#221364)\n\n## Summary\n\nEnable ILM for
classic streams.\n\nWe can now overwrite template settings at the data
stream level with\n`PUT _data_stream/{name}/_settings` so we can enable
full lifecycle\ncapabilities on classic
stream.\n\n---------\n\nCo-authored-by: kibanamachine
<[email protected]>","sha":"1abb811b2ea0e9cdd30e2b7c21bcfb32488fee84","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:obs-ux-logs","backport:version","Feature:Streams","v9.1.0","v8.19.0"],"title":"[streams][lifecycle]
ilm for classic
streams","number":221364,"url":"https://github.com/elastic/kibana/pull/221364","mergeCommit":{"message":"[streams][lifecycle]
ilm for classic streams (#221364)\n\n## Summary\n\nEnable ILM for
classic streams.\n\nWe can now overwrite template settings at the data
stream level with\n`PUT _data_stream/{name}/_settings` so we can enable
full lifecycle\ncapabilities on classic
stream.\n\n---------\n\nCo-authored-by: kibanamachine
<[email protected]>","sha":"1abb811b2ea0e9cdd30e2b7c21bcfb32488fee84"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/221364","number":221364,"mergeCommit":{"message":"[streams][lifecycle]
ilm for classic streams (#221364)\n\n## Summary\n\nEnable ILM for
classic streams.\n\nWe can now overwrite template settings at the data
stream level with\n`PUT _data_stream/{name}/_settings` so we can enable
full lifecycle\ncapabilities on classic
stream.\n\n---------\n\nCo-authored-by: kibanamachine
<[email protected]>","sha":"1abb811b2ea0e9cdd30e2b7c21bcfb32488fee84"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Kevin Lacabane <[email protected]>
Co-authored-by: Joe Reuter <[email protected]>
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jun 23, 2025
@flash1293
Copy link
Contributor

Thanks @masseyke , kicked off another build and it all worked out

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@flash1293 flash1293 flash1293 approved these changes

Assignees

No one assigned

Labels

backport:version Backport to applied version labels Feature:Streams This is the label for the Streams Project release_note:skip Skip the PR/issue when compiling release notes Team:obs-ux-logs Observability Logs User Experience Team v8.19.0 v9.1.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@klacabane @flash1293 @elasticmachine @kibanamachine @masseyke