Commit d37cb19
# Backport
This will backport the following commits from `main` to `9.2`:
- [π Streams: Retention pages fixes
(#238683)](#238683)
<!--- Backport version: 9.6.6 -->
### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)
<!--BACKPORT [{"author":{"name":"Joe
Reuter","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-10-15T15:49:45Z","message":"π
Streams: Retention pages fixes (#238683)\n\nFixes
https://github.com/elastic/kibana/issues/238277\nFixes
https://github.com/elastic/kibana/issues/238643\n\n* Subtract failure
store size from storage size when possible\n* Do not hide the chart data
while it's loading after a refresh - no\njumpy chart\n* Only sample if
there more than 10k docs in total:\n* On page load, start fetching with
a sampling of 0.1 and fetch the data\nstream stats in parallel (this
already happened)\n* If data stream stats arrive and there are less than
10k docs, refetch\nthe chart data without sampling\n\nThis has the
following effect:\n* If there isn't much data, we fetch twice, but it
doesn't matter much\nbecause there is very little data (e.g. for failure
store) and the chart\nis meaningful because it has the actual counts\n*
If there is a lot of data, nothing changes compared to todays
version\n\n\nThis also does another improvement - for estimating the
ingest rate, we\ncurrently take the current time range as the basis to
extrapolate. But\nwe also know the creation date of the data stream,
which we can take\ninto account. E.g. if the user picks a time range of
\"last 10 years\",\nthen we can be smart and only take 4 days of that
range if we know the\ndata stream only existed since 4 days.\n\n###
Testing `(storage size - failure store)` change\n\n**Full
permissions**\n- ingest documents in a stream and in its failure
store\n- verify the \"Storage size\" corresponds to the sum of the
regular\n(`.ds-..`) indices' of the stream\n- verify the \"Failure
storage size\" corresponds to the failure store\n(`.fs-..`) size of the
stream\n\n**No failure store permissions**\n- create user with no
permissions to manage failure store\n```\n// 1. create role\nPOST
_security/role/streams_no_failure_store\n{\n \"cluster\": [\n
\"manage_index_templates\",\n \"manage_ingest_pipelines\"\n ],\n
\"indices\": [\n {\n \"names\": [\n \"logs*\"\n ],\n \"privileges\": [\n
\"view_index_metadata\",\n \"read\",\n \"write\",\n \"monitor\"\n ],\n
\"field_security\": {\n \"grant\": [\n \"*\"\n ],\n \"except\": []\n
},\n \"allow_restricted_indices\": false\n }\n ],\n \"applications\":
[\n {\n \"application\": \"kibana-.kibana\",\n \"privileges\": [\n
\"feature_streams.all\"\n ],\n \"resources\": [\n \"*\"\n ]\n }\n ],\n
\"run_as\": [],\n \"metadata\": {},\n \"transient_metadata\": {\n
\"enabled\": true\n }\n}\n\n// 2. create user\nPOST
_security/user/streams_user_no_fs\n{\n \"password\" : \"changeme\",\n
\"roles\" : [ \"streams_no_failure_store\" ]\n}\n```\n\n2. log in as
user and visit the stream that contains regular and failure\nstore
data\n3. verify that \"Storage size\" is the sum of regular (`.ds-..`)
and\n(`.fs-..`) indices\n4. verify that Storage size has a tooltip and
Ingestion rate's tooltip\nmentions that metrics may be
inaccurate\n\n---------\n\nCo-authored-by: klacabane
<[email protected]>\nCo-authored-by: kibanamachine
<[email protected]>","sha":"389ef265945a0a9a5247bbb1ea993800917d61dc","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:obs-ux-logs","backport:version","Feature:Streams","v9.2.0","Team:streams-program","v9.3.0"],"title":"π
Streams: Retention pages
fixes","number":238683,"url":"https://github.com/elastic/kibana/pull/238683","mergeCommit":{"message":"π
Streams: Retention pages fixes (#238683)\n\nFixes
https://github.com/elastic/kibana/issues/238277\nFixes
https://github.com/elastic/kibana/issues/238643\n\n* Subtract failure
store size from storage size when possible\n* Do not hide the chart data
while it's loading after a refresh - no\njumpy chart\n* Only sample if
there more than 10k docs in total:\n* On page load, start fetching with
a sampling of 0.1 and fetch the data\nstream stats in parallel (this
already happened)\n* If data stream stats arrive and there are less than
10k docs, refetch\nthe chart data without sampling\n\nThis has the
following effect:\n* If there isn't much data, we fetch twice, but it
doesn't matter much\nbecause there is very little data (e.g. for failure
store) and the chart\nis meaningful because it has the actual counts\n*
If there is a lot of data, nothing changes compared to todays
version\n\n\nThis also does another improvement - for estimating the
ingest rate, we\ncurrently take the current time range as the basis to
extrapolate. But\nwe also know the creation date of the data stream,
which we can take\ninto account. E.g. if the user picks a time range of
\"last 10 years\",\nthen we can be smart and only take 4 days of that
range if we know the\ndata stream only existed since 4 days.\n\n###
Testing `(storage size - failure store)` change\n\n**Full
permissions**\n- ingest documents in a stream and in its failure
store\n- verify the \"Storage size\" corresponds to the sum of the
regular\n(`.ds-..`) indices' of the stream\n- verify the \"Failure
storage size\" corresponds to the failure store\n(`.fs-..`) size of the
stream\n\n**No failure store permissions**\n- create user with no
permissions to manage failure store\n```\n// 1. create role\nPOST
_security/role/streams_no_failure_store\n{\n \"cluster\": [\n
\"manage_index_templates\",\n \"manage_ingest_pipelines\"\n ],\n
\"indices\": [\n {\n \"names\": [\n \"logs*\"\n ],\n \"privileges\": [\n
\"view_index_metadata\",\n \"read\",\n \"write\",\n \"monitor\"\n ],\n
\"field_security\": {\n \"grant\": [\n \"*\"\n ],\n \"except\": []\n
},\n \"allow_restricted_indices\": false\n }\n ],\n \"applications\":
[\n {\n \"application\": \"kibana-.kibana\",\n \"privileges\": [\n
\"feature_streams.all\"\n ],\n \"resources\": [\n \"*\"\n ]\n }\n ],\n
\"run_as\": [],\n \"metadata\": {},\n \"transient_metadata\": {\n
\"enabled\": true\n }\n}\n\n// 2. create user\nPOST
_security/user/streams_user_no_fs\n{\n \"password\" : \"changeme\",\n
\"roles\" : [ \"streams_no_failure_store\" ]\n}\n```\n\n2. log in as
user and visit the stream that contains regular and failure\nstore
data\n3. verify that \"Storage size\" is the sum of regular (`.ds-..`)
and\n(`.fs-..`) indices\n4. verify that Storage size has a tooltip and
Ingestion rate's tooltip\nmentions that metrics may be
inaccurate\n\n---------\n\nCo-authored-by: klacabane
<[email protected]>\nCo-authored-by: kibanamachine
<[email protected]>","sha":"389ef265945a0a9a5247bbb1ea993800917d61dc"}},"sourceBranch":"main","suggestedTargetBranches":["9.2"],"targetPullRequestStates":[{"branch":"9.2","label":"v9.2.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/238683","number":238683,"mergeCommit":{"message":"π
Streams: Retention pages fixes (#238683)\n\nFixes
https://github.com/elastic/kibana/issues/238277\nFixes
https://github.com/elastic/kibana/issues/238643\n\n* Subtract failure
store size from storage size when possible\n* Do not hide the chart data
while it's loading after a refresh - no\njumpy chart\n* Only sample if
there more than 10k docs in total:\n* On page load, start fetching with
a sampling of 0.1 and fetch the data\nstream stats in parallel (this
already happened)\n* If data stream stats arrive and there are less than
10k docs, refetch\nthe chart data without sampling\n\nThis has the
following effect:\n* If there isn't much data, we fetch twice, but it
doesn't matter much\nbecause there is very little data (e.g. for failure
store) and the chart\nis meaningful because it has the actual counts\n*
If there is a lot of data, nothing changes compared to todays
version\n\n\nThis also does another improvement - for estimating the
ingest rate, we\ncurrently take the current time range as the basis to
extrapolate. But\nwe also know the creation date of the data stream,
which we can take\ninto account. E.g. if the user picks a time range of
\"last 10 years\",\nthen we can be smart and only take 4 days of that
range if we know the\ndata stream only existed since 4 days.\n\n###
Testing `(storage size - failure store)` change\n\n**Full
permissions**\n- ingest documents in a stream and in its failure
store\n- verify the \"Storage size\" corresponds to the sum of the
regular\n(`.ds-..`) indices' of the stream\n- verify the \"Failure
storage size\" corresponds to the failure store\n(`.fs-..`) size of the
stream\n\n**No failure store permissions**\n- create user with no
permissions to manage failure store\n```\n// 1. create role\nPOST
_security/role/streams_no_failure_store\n{\n \"cluster\": [\n
\"manage_index_templates\",\n \"manage_ingest_pipelines\"\n ],\n
\"indices\": [\n {\n \"names\": [\n \"logs*\"\n ],\n \"privileges\": [\n
\"view_index_metadata\",\n \"read\",\n \"write\",\n \"monitor\"\n ],\n
\"field_security\": {\n \"grant\": [\n \"*\"\n ],\n \"except\": []\n
},\n \"allow_restricted_indices\": false\n }\n ],\n \"applications\":
[\n {\n \"application\": \"kibana-.kibana\",\n \"privileges\": [\n
\"feature_streams.all\"\n ],\n \"resources\": [\n \"*\"\n ]\n }\n ],\n
\"run_as\": [],\n \"metadata\": {},\n \"transient_metadata\": {\n
\"enabled\": true\n }\n}\n\n// 2. create user\nPOST
_security/user/streams_user_no_fs\n{\n \"password\" : \"changeme\",\n
\"roles\" : [ \"streams_no_failure_store\" ]\n}\n```\n\n2. log in as
user and visit the stream that contains regular and failure\nstore
data\n3. verify that \"Storage size\" is the sum of regular (`.ds-..`)
and\n(`.fs-..`) indices\n4. verify that Storage size has a tooltip and
Ingestion rate's tooltip\nmentions that metrics may be
inaccurate\n\n---------\n\nCo-authored-by: klacabane
<[email protected]>\nCo-authored-by: kibanamachine
<[email protected]>","sha":"389ef265945a0a9a5247bbb1ea993800917d61dc"}}]}]
BACKPORT-->
Co-authored-by: Joe Reuter <[email protected]>
Co-authored-by: klacabane <[email protected]>
1 parent fedea09 commit d37cb19
File tree
19 files changed
+583
-684
lines changed19 files changed
+583
-684
lines changedLines changed: 18 additions & 37 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
30 | 29 | | |
31 | | - | |
| 30 | + | |
32 | 31 | | |
33 | | - | |
34 | 32 | | |
| 33 | + | |
35 | 34 | | |
36 | | - | |
| 35 | + | |
37 | 36 | | |
38 | 37 | | |
39 | 38 | | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
40 | 42 | | |
41 | 43 | | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | 44 | | |
62 | 45 | | |
63 | 46 | | |
64 | 47 | | |
65 | | - | |
66 | 48 | | |
67 | | - | |
| 49 | + | |
68 | 50 | | |
69 | 51 | | |
70 | | - | |
| 52 | + | |
71 | 53 | | |
72 | 54 | | |
73 | | - | |
74 | | - | |
| 55 | + | |
| 56 | + | |
75 | 57 | | |
76 | 58 | | |
77 | 59 | | |
| |||
98 | 80 | | |
99 | 81 | | |
100 | 82 | | |
101 | | - | |
102 | 83 | | |
103 | | - | |
| 84 | + | |
104 | 85 | | |
105 | 86 | | |
106 | | - | |
| 87 | + | |
107 | 88 | | |
108 | 89 | | |
109 | | - | |
110 | | - | |
| 90 | + | |
| 91 | + | |
111 | 92 | | |
112 | 93 | | |
113 | 94 | | |
| |||
151 | 132 | | |
152 | 133 | | |
153 | 134 | | |
154 | | - | |
| 135 | + | |
155 | 136 | | |
156 | 137 | | |
157 | 138 | | |
| |||
215 | 196 | | |
216 | 197 | | |
217 | 198 | | |
218 | | - | |
| 199 | + | |
219 | 200 | | |
220 | 201 | | |
221 | 202 | | |
| |||
266 | 247 | | |
267 | 248 | | |
268 | 249 | | |
269 | | - | |
| 250 | + | |
270 | 251 | | |
271 | 252 | | |
272 | 253 | | |
273 | 254 | | |
274 | 255 | | |
275 | 256 | | |
276 | | - | |
| 257 | + | |
277 | 258 | | |
278 | 259 | | |
279 | 260 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
| 15 | + | |
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
| 23 | + | |
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
14 | 13 | | |
| 14 | + | |
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
Lines changed: 5 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
18 | | - | |
| 17 | + | |
| 18 | + | |
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
30 | | - | |
31 | 29 | | |
32 | 30 | | |
33 | 31 | | |
34 | 32 | | |
35 | 33 | | |
36 | | - | |
| 34 | + | |
37 | 35 | | |
38 | 36 | | |
39 | | - | |
40 | | - | |
41 | | - | |
| 37 | + | |
42 | 38 | | |
43 | 39 | | |
44 | 40 | | |
| |||
72 | 68 | | |
73 | 69 | | |
74 | 70 | | |
75 | | - | |
76 | | - | |
| 71 | + | |
77 | 72 | | |
78 | 73 | | |
79 | 74 | | |
| |||
Lines changed: 14 additions & 31 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
18 | 17 | | |
19 | | - | |
| 18 | + | |
20 | 19 | | |
21 | 20 | | |
22 | 21 | | |
| |||
27 | 26 | | |
28 | 27 | | |
29 | 28 | | |
| 29 | + | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| |||
37 | 38 | | |
38 | 39 | | |
39 | 40 | | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | 41 | | |
50 | 42 | | |
51 | 43 | | |
| |||
58 | 50 | | |
59 | 51 | | |
60 | 52 | | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | 53 | | |
69 | 54 | | |
70 | 55 | | |
| |||
89 | 74 | | |
90 | 75 | | |
91 | 76 | | |
92 | | - | |
| 77 | + | |
93 | 78 | | |
94 | 79 | | |
95 | 80 | | |
96 | 81 | | |
97 | 82 | | |
98 | 83 | | |
99 | 84 | | |
100 | | - | |
| 85 | + | |
101 | 86 | | |
102 | 87 | | |
103 | 88 | | |
104 | 89 | | |
105 | | - | |
106 | | - | |
107 | | - | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
108 | 93 | | |
109 | 94 | | |
110 | 95 | | |
111 | | - | |
| 96 | + | |
112 | 97 | | |
113 | 98 | | |
114 | 99 | | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
119 | 104 | | |
120 | | - | |
121 | | - | |
122 | | - | |
| 105 | + | |
123 | 106 | | |
124 | 107 | | |
125 | 108 | | |
| |||
Lines changed: 30 additions & 20 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
9 | 16 | | |
10 | 17 | | |
11 | 18 | | |
12 | | - | |
13 | 19 | | |
14 | 20 | | |
15 | | - | |
| 21 | + | |
| 22 | + | |
16 | 23 | | |
17 | 24 | | |
18 | 25 | | |
19 | 26 | | |
20 | 27 | | |
21 | 28 | | |
22 | | - | |
23 | 29 | | |
24 | | - | |
| 30 | + | |
25 | 31 | | |
26 | 32 | | |
27 | | - | |
| 33 | + | |
28 | 34 | | |
29 | 35 | | |
30 | | - | |
31 | | - | |
32 | | - | |
| 36 | + | |
| 37 | + | |
33 | 38 | | |
34 | 39 | | |
35 | 40 | | |
36 | 41 | | |
37 | 42 | | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
47 | 58 | | |
48 | 59 | | |
49 | 60 | | |
| |||
65 | 76 | | |
66 | 77 | | |
67 | 78 | | |
68 | | - | |
69 | | - | |
| 79 | + | |
70 | 80 | | |
71 | 81 | | |
72 | 82 | | |
| |||
0 commit comments