-
Notifications
You must be signed in to change notification settings - Fork 159
[Upgrade 9.0] Kibana updates #991
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 8 commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
4cc22b3
Kibana updates.
jmikell821 d80c0e6
Update saved-object-migrations.md
jmikell821 a6cc6a4
Merge branch 'main' into kib-updates-upgrade
jmikell821 eba2a10
Splits index aliases and names into two tables.
jmikell821 1dd1a43
Rewrite evolution of saved object indices
gsoldevila cb0c022
Polish some bits
gsoldevila bf86bb2
above -> below
gsoldevila aebfe97
Uniformise to present tense
gsoldevila 6cd64b9
Merge branch 'main' into kib-updates-upgrade
jmikell821 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
85 changes: 62 additions & 23 deletions
85
deploy-manage/upgrade/deployment-or-cluster/saved-object-migrations.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,52 +1,91 @@ | ||
--- | ||
mapped_pages: | ||
- https://www.elastic.co/guide/en/kibana/current/saved-object-migrations.html | ||
applies_to: | ||
stack: | ||
deployment: | ||
eck: | ||
ess: | ||
ece: | ||
self: | ||
--- | ||
|
||
# Saved object migrations [saved-object-migrations] | ||
|
||
Each time you upgrade {{kib}}, an upgrade migration is performed to ensure that all [saved objects](/explore-analyze/find-and-organize/saved-objects.md) are compatible with the new version. | ||
|
||
::::{note} | ||
{{kib}} includes an [**Upgrade Assistant**](../prepare-to-upgrade/upgrade-assistant.md) to help you prepare for an upgrade. To access the assistant, go to **Stack Management > Upgrade Assistant**. | ||
::::{note} | ||
{{kib}} includes an [**Upgrade Assistant**](../prepare-to-upgrade/upgrade-assistant.md) to help you prepare to upgrade. To access the assistant, go to **Stack Management > Upgrade Assistant**. | ||
:::: | ||
|
||
|
||
::::{warning} | ||
{{kib}} 7.12.0 and later uses a new migration process and index naming scheme. Before you upgrade, read the documentation for your version of {{kib}}. | ||
:::: | ||
% ::::{warning} | ||
% {{kib}} 7.12.0 and later uses a new migration process and index naming scheme. % Before you upgrade, read the documentation for your version of {{kib}}. | ||
% :::: | ||
|
||
## How saved object migrations work [upgrade-migrations-process] | ||
|
||
::::{warning} | ||
The `kibana.index` and `xpack.tasks.index` configuration settings are obsolete and no longer taken into account in 8.x. If you are using custom index names, please perform the necessary adaptations before attempting to upgrade to 8.x. | ||
:::: | ||
When you start a new {{kib}} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic. Before you upgrade, shut down old nodes to prevent losing acknowledged writes. | ||
If the upgrade includes breaking changes, the old saved objects will be reindexed into new indices. Otherwise, the existing indices will be reused, and saved object documents will be updated. | ||
|
||
Saved objects are stored in multiple indices. While they all start with the `.kibana*` prefix, other `.kibana*` indices exist but are not used to store saved objects. | ||
The indices used to store saved objects and the conventions on their names and aliases have evolved as follows: | ||
|
||
### Kibana 6.5.0 | ||
|
||
The `.kibana_N` saved object index is created. Saved objects are reindexed into a new index, and the `N` suffix is incremented each time an upgrade is performed. A `.kibana` alias is maintained, which points to the latest version of the index. | ||
|
||
## How saved objects migrations work [upgrade-migrations-process] | ||
### Kibana 7.4.0 | ||
|
||
When you start a new {{kib}} installation, an upgrade migration is performed before starting plugins or serving HTTP traffic. Before you upgrade, shut down old nodes to prevent losing acknowledged writes. To reduce the likelihood of old nodes losing acknowledged writes, {{kib}} 7.12.0 and later adds a write block to the outdated index. | ||
A new `.kibana_task_manager_N` is introduced, which hosts the `task` saved objects. It also has the corresponding `.kibana_task_manager` alias, pointing to the latest version of the index. | ||
|
||
Saved objects are stored in multiple indices. Whilst all of them start with the `.kibana*` prefix, other `.kibana*` indices exist, which are not used to store saved objects. The following tables lists the saved objects indices used by each {{kib}} version. | ||
### Kibana 7.11.0 | ||
|
||
| Upgrading from version | Index | Aliases | | ||
Starting with 7.11.0 the naming convention evolves, and the indices names contain the version number, along with a `_001` suffix. | ||
Each of the saved objects indices now has a couple of aliases. For example, the `.kibana_7.11.0_001` index has a *default* `.kibana` alias and a `.kibana_7.11.0` *version* alias. The *default* aliases (such as `.kibana` and `.kibana_task_manager`) always point to the most up-to-date saved object indices. Then, *version* aliases are aligned with the deployed {{kib}} version: | ||
|
||
| Alias | Version alias | Index name | | ||
| --- | --- | --- | | ||
| 6.5.0 through 7.3.x | `.kibana_N` | `.kibana` | | ||
| 7.4.0 through 7.11.x | `.kibana_N`<br>`.kibana_task_manager_N` | `.kibana`<br>`.kibana_task_manager` | | ||
| 7.11.x through 8.7.x | `.kibana_{{kibana_version}}_001`<br>`.kibana_task_manager_{{kibana_version}}_001` | `.kibana`, `.kibana_{{kibana_version}}`<br>`.kibana_task_manager`, `.kibana_task_manager_{{kibana_version}}` | | ||
| 8.8.0+ | `.kibana_{{kibana_version}}_001`<br>`.kibana_alerting_cases_{{kibana_version}}_001``.kibana_analytics_{{kibana_version}}_001``.kibana_ingest_{{kibana_version}}_001``.kibana_task_manager_{{kibana_version}}_001``.kibana_security_solution_{{kibana_version}}_001` | `.kibana`, `.kibana_{{kibana_version}}`<br>`.kibana_alerting_cases`, `.kibana_alerting_cases_{{kibana_version}}``.kibana_analytics`, `.kibana_analytics_{{kibana_version}}``.kibana_ingest`, `.kibana_ingest_{{kibana_version}}``.kibana_task_manager`, `.kibana_task_manager_{{kibana_version}}``.kibana_security_solution`, `.kibana_security_solution_{{kibana_version}}` | | ||
| .kibana | .kibana_7.11.0 | .kibana_7.11.0_001 | | ||
| .kibana_task_manager | .kibana_task_manager_7.11.0 | .kibana_task_manager_7.11.0_001 | | ||
|
||
### Kibana 8.6.0 | ||
|
||
Starting on 7.11.0, each of the saved objects indices has a couple of aliases, e.g. the `.kibana_8.8.0_001` index has a *default* alias `.kibana` and a *version* alias `.kibana_8.8.0`. The *default* aliases (e.g. `.kibana` and `.kibana_task_manager`) always point to the most up-to-date saved object indices. Then, *version* aliases are aligned with the deployed {{kib}} version. | ||
In versions 8.6.0 and newer, compatible migrations are introduced, allowing to reuse existing saved object indices as long as changes in the mappings are compatible. After this change, index names aren’t necessarily aligned with the deployed {{kib}} version. When updates on a certain index are compatible, {{kib}} will keep the existing index instead of creating a new one. This allows for a more efficient upgrade process. The following example illustrates a completely valid state for an 8.7.0 deployment: | ||
|
||
Starting on 8.6.0, index names aren’t necessarily aligned with the deployed {{kib}} version. When updates on a certain index are compatible, {{kib}} will keep the existing index instead of creating a new one. This allows for a more efficient upgrade process. The following example illustrates a completely valid state for a 8.8.0 deployment: | ||
| Alias | Versioned alias | Index name | | ||
| --- | --- | --- | | ||
| .kibana | .kibana_8.7.0 | .kibana_8.7.0_001 | | ||
| .kibana_task_manager | .kibana_task_manager_8.7.0 | .kibana_task_manager_7.17.0_001 | | ||
|
||
* `.kibana_8.8.0_001` index, with `.kibana` and `.kibana_8.8.0` aliases. | ||
* `.kibana_task_manager_8.7.0_001` index, with `.kibana_task_manager` and `.kibana_task_manager_8.8.0` aliases. | ||
### Kibana 8.8.0 | ||
|
||
Starting on 8.8.0, {{kib}} splits the main saved object index into multiple ones, as depicted on the table above. When upgrading from a previous version, the {{kib}} migration process will reindex some saved objects from the `.kibana` index into the new indices, depending on their types. Note that the `.kibana` index still exists, and it continues to store multiple saved object types. | ||
Starting with 8.8.0, {{kib}} splits the main saved object index into multiple ones, as depicted in the table below. When upgrading from a previous version, the {{kib}} migration process will reindex some saved objects from the `.kibana` index into the new indices, depending on their types. Note that the `.kibana` index still exists and continues storing multiple saved object types. | ||
|
||
| Alias | Version alias | Index name | | ||
| --- | --- | --- | | ||
| .kibana | .kibana_8.8.0 | .kibana_8.8.0_001 | | ||
| .kibana_task_manager | .kibana_task_manager_8.8.0 | .kibana_task_manager_7.17.0_001 | | ||
| .kibana_alerting_cases | .kibana_alerting_cases_8.8.0 | .kibana_alerting_cases_8.8.0_001 | | ||
| .kibana_analytics | .kibana_analytics_8.8.0 | .kibana_analytics_8.8.0_001 | | ||
| .kibana_ingest | .kibana_ingest_8.8.0 | .kibana_ingest_8.8.0_001 | | ||
| .kibana_security_solution | .kibana_security_solution_8.8.0 | .kibana_security_solution_8.8.0_001 | | ||
|
||
### Kibana 8.16.0 and newer | ||
|
||
A new index is introduced, with the goal of storing the `usage-counter` saved object type, which is expected to have a large number of documents. | ||
|
||
| Alias | Version alias | Index name | | ||
| --- | --- | --- | | ||
| .kibana | .kibana_8.8.0 | .kibana_8.8.0_001 | | ||
| .kibana_task_manager | .kibana_task_manager_8.8.0 | .kibana_task_manager_7.17.0_001 | | ||
| .kibana_alerting_cases | .kibana_alerting_cases_8.8.0 | .kibana_alerting_cases_8.8.0_001 | | ||
| .kibana_analytics | .kibana_analytics_8.8.0 | .kibana_analytics_8.8.0_001 | | ||
| .kibana_ingest | .kibana_ingest_8.8.0 | .kibana_ingest_8.8.0_001 | | ||
| .kibana_security_solution | .kibana_security_solution_8.8.0 | .kibana_security_solution_8.8.0_001 | | ||
| .kibana_usage_counters | .kibana_usage_counters_8.8.0 | .kibana_usage_counters_8.8.0_001 | | ||
|
||
## Old {{kib}} indices [upgrade-migrations-old-indices] | ||
|
||
As a deployment is gradually upgraded, multiple {{kib}} indices are created in {{es}}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0_001`, `.kibana_7.13.0_001`, `.kibana_8.0.0_001` etc). {{kib}} only uses those indices that the *default* and *version* aliases point to. The other, older {{kib}} saved object indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {{kib}} back to a previous version. | ||
## Old {{kib}} indices [upgrade-migrations-old-indices] | ||
|
||
When you upgrade a deployment, Elastic creates multiple {{kib}} indices in {{es}}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0_001`, `.kibana_7.13.0_001`, `.kibana_8.0.0_001`, etc.). {{kib}} only uses those indices that the *default* and *version* aliases point to. You can safely delete the older {{kib}} saved object indices, but they're retained for historical records and to facilitate rolling {{kib}} back to a previous version. |
26 changes: 16 additions & 10 deletions
26
deploy-manage/upgrade/prepare-to-upgrade/upgrade-assistant.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,32 +1,38 @@ | ||
--- | ||
mapped_pages: | ||
- https://www.elastic.co/guide/en/kibana/current/upgrade-assistant.html | ||
applies_to: | ||
stack: | ||
deployment: | ||
eck: | ||
ess: | ||
ece: | ||
self: | ||
--- | ||
|
||
# Upgrade Assistant [upgrade-assistant] | ||
|
||
The Upgrade Assistant helps you prepare for your upgrade to the next version of the {{stack}}. To access the assistant, go to **{{stack-manage-app}} > Upgrade Assistant**. | ||
|
||
The assistant identifies deprecated settings in your configuration and guides you through the process of resolving issues if any deprecated features are enabled. | ||
The Upgrade Assistant helps you [prepare to upgrade](/deploy-manage/upgrade/prepare-to-upgrade.md) to the next version of the {{stack}}. To access the assistant, go to **{{stack-manage-app}} → Upgrade Assistant**. | ||
|
||
The assistant identifies deprecated settings in your configuration, and if any of those settings are enabled, it guides you through resolving issues that could prevent a successful upgrade. The Upgrade Assistant also helps resolve issues with older indices created before version 8.0.0, providing options to reindex older indices or mark them as read-only. | ||
|
||
## Required permissions [_required_permissions_11] | ||
|
||
The `manage` cluster privilege is required to access the **Upgrade assistant**. Additional privileges may be needed to perform certain actions. | ||
To access the Upgrade Assistant, you need the `manage` cluster privilege. You may also need additional privileges to perform specific actions. | ||
|
||
|
||
## Feature set [_feature_set] | ||
|
||
Some features of the Upgrade assistant are only needed when upgrading to a new major version. The feature set enabled by default are those for the very next version from the one Kibana currently runs on. | ||
Some features of the Upgrade Assistant are only needed when upgrading to a new major version. The features enabled by default are those for the next version from the one {{kib}} currently runs on. | ||
|
||
|
||
## Deprecations [_deprecations] | ||
|
||
The Upgrade assistant pulls information about deprecations from the following sources: | ||
The Upgrade Assistant pulls information about deprecations from the following sources: | ||
|
||
* Elasticsearch Deprecation Info API | ||
* Elasticsearch deprecation logs | ||
* Kibana deprecations API | ||
* {{es}} deprecation info API | ||
* {{es}} deprecation logs | ||
* {{kib}} deprecations API | ||
|
||
For more information about Upgrade Assistant APIs, refer to [Upgrade Assistant APIs](https://www.elastic.co/guide/en/kibana/current/upgrade-assistant-api.html). | ||
For more information about Upgrade Assistant APIs, refer to [Upgrade Assistant APIs](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-upgrade). | ||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.