Skip to content

Updates "Reading indices from older Elasticsearch versions" topic #919

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 26, 2025
Merged

Updates "Reading indices from older Elasticsearch versions" topic #919

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
434a00d
Update reading-indices-from-older-elasticsearch-versions.md
jmikell821 Mar 25, 2025
ded0e04
Update reading-indices-from-older-elasticsearch-versions.md
jmikell821 Mar 26, 2025
d9b916a
Incorporating feedback, adding applies_to tags.
jmikell821 Mar 26, 2025
ef067aa
Merge branch 'main' into upgrade-es-indices
jmikell821 Mar 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 20 additions & 16 deletions ...rade/deployment-or-cluster/reading-indices-from-older-elasticsearch-versions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,50 +3,54 @@ mapped_pages:
- https://www.elastic.co/guide/en/elasticsearch/reference/current/archive-indices.html
---

# Reading indices from older Elasticsearch versions [archive-indices]
# Reading indices from older {{es}} versions [archive-indices]

{{es}} has full query and write support for indices created in the previous major version. If you have indices created in {{es}} versions 5 or 6, you can now use the archive functionality to import them into newer {{es}} versions as well.
{{es}} has full query and write support for indices created in the previous major version. If you have indices created in {{es}} versions 5, 6, or 7, you can now use the archive functionality to import them into newer {{es}} versions as well.

The archive functionality provides slower read-only access to older {{es}} data, for compliance or regulatory reasons, the occasional lookback or investigation, or to rehydrate parts of it. Access to the data is expected to be infrequent, and can therefore happen with limited performance and query capabilities.

For this, {{es}} has the ability to access older snapshot repositories (going back to version 5). The legacy indices in the [snapshot repository](../../tools/snapshot-and-restore.md) can either be [restored](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-restore), or can be directly accessed via [searchable snapshots](../../tools/snapshot-and-restore/searchable-snapshots.md) so that the archived data won’t even need to fully reside on local disks for access.
For this, {{es}} can access older snapshot repositories going back to version 5. The legacy indices in the [snapshot repository](../../tools/snapshot-and-restore.md) can either be [restored](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-restore) or directly accessed through [searchable snapshots](../../tools/snapshot-and-restore/searchable-snapshots.md) so that the archived data won’t need to fully reside on local disks for access.


## Supported field types [archive-indices-supported-field-types]

Old mappings are imported as much "as-is" as possible into {{es}} 8, but only provide regular query capabilities on a select subset of fields:
Old mappings are imported as much as possible into {{es}} 9, but only provide regular query capabilities on a select subset of fields:

* [Numeric types](elasticsearch://reference/elasticsearch/mapping-reference/number.md)
* [`boolean` type](elasticsearch://reference/elasticsearch/mapping-reference/boolean.md)
* [`ip` type](elasticsearch://reference/elasticsearch/mapping-reference/ip.md)
* [`geo_point` type](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md)
* [`date` types](elasticsearch://reference/elasticsearch/mapping-reference/date.md): the date `format` setting on date fields is supported as long as it behaves similarly across these versions. In case it is not, for example [when using custom date formats](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/migrate-to-java-time.html), this field can be updated on legacy indices so that it can be changed by a user if need be.
* [`keyword` type](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md#keyword-field-type): the `normalizer` setting on keyword fields is supported as long as it behaves similarly across these versions. In case it is not, this field can be updated on legacy indices if need be.
* [`text` type](elasticsearch://reference/elasticsearch/mapping-reference/text.md#text-field-type): scoring capabilities are limited, and all queries return constant scores that are equal to 1.0. The `analyzer` settings on text fields are supported as long as they behave similarly across these versions. In case they do not, they can be updated on legacy indices if need be.
* [`date` types](elasticsearch://reference/elasticsearch/mapping-reference/date.md): the date `format` setting on date fields is supported as long as it behaves similarly across these versions. If it is not, for example, [when using custom date formats](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/migrate-to-java-time.html), this field can be updated on legacy indices so a user can change it if needed.
* [`keyword` type](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md#keyword-field-type): The `normalizer` setting on keyword fields is supported as long as it behaves similarly across these versions. If they do not, they can be updated on legacy indices if necessary.
* [`text` type](elasticsearch://reference/elasticsearch/mapping-reference/text.md#text-field-type): Scoring capabilities are limited, and all queries return constant scores equal to 1.0. The `analyzer` settings on text fields are supported as long as they behave similarly across these versions. If they don't, they can be updated on legacy indices.
* [Multi-fields](elasticsearch://reference/elasticsearch/mapping-reference/multi-fields.md)
* [Field aliases](elasticsearch://reference/elasticsearch/mapping-reference/field-alias.md)
* [`object`](elasticsearch://reference/elasticsearch/mapping-reference/object.md) fields
* some basic metadata fields, e.g. `_type` for querying {{es}} 5 indices
* some basic metadata fields, such as `_type` for querying {{es}} 5 indices
* [runtime fields](../../../manage-data/data-store/mapping/map-runtime-field.md)
* [`_source` field](elasticsearch://reference/elasticsearch/mapping-reference/mapping-source-field.md)

{{es}} 5 indices with mappings that have [multiple mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html) are collapsed together on a best-effort basis before they are imported.
{{es}} 5 indices with mappings that have [multiple mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html) are collapsed together on a best-effort basis before being imported.

In case the auto-import of mappings does not work, or the new {{es}} version can’t make sense of the mapping, it falls back to importing the index without the mapping, but stores the original mapping in the [_meta](elasticsearch://reference/elasticsearch/mapping-reference/mapping-meta-field.md) section of the imported index. The legacy mapping can then be introspected using the [GET mapping](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-mapping) API and an updated mapping can be manually put in place using the [update mapping](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-mapping) API, copying and adapting relevant sections of the legacy mapping to work with the current {{es}} version. While auto-import is expected to work in most cases, failures of doing so should be [raised](https://github.com/elastic/elasticsearch/issues/new/choose) with the Elastic team for future improvements.
If auto-importing mappings does not work, or the new {{es}} version doesn't understand the mapping, it falls back to importing the index without the mapping, but stores the original mapping in the [_meta](elasticsearch://reference/elasticsearch/mapping-reference/mapping-meta-field.md) section of the imported index. Users can then examine the legacy mapping using the [GET mapping](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-mapping) API, and manually update the mapping using the [update mapping](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-mapping) API, copying and adapting relevant sections of the legacy mapping to work with the current {{es}} version. While auto-import is expected to work in most cases, any failures should be [raised](https://github.com/elastic/elasticsearch/issues/new/choose) with the Elastic team for future improvements.


## Supported APIs [_supported_apis]

Archive indices are read-only, and provide data access via the [search](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) and [field capabilities](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-field-caps) APIs. They do not support the [Get API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-get) nor any write APIs.
Archive indices are read-only and provide data access through the [search](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search) and [field capabilities](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-field-caps) APIs. They do not support the [Get API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-get) or any write APIs.

Archive indices allow running queries as well as aggregations in so far as they are [supported by the given field type](#archive-indices-supported-field-types).
Archive indices allow running queries and aggregations if the [field type](#archive-indices-supported-field-types) supports them.

Due to `_source` access the data can also be [reindexed](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) to a new index that has full compatibility with the current {{es}} version.
Due to `_source` access, the data can also be [reindexed](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) to a new index that's fully compatible with the current {{es}} version.


## How to upgrade older {{es}} 5 or 6 clusters? [_how_to_upgrade_older_es_5_or_6_clusters]
## Upgrade older {{es}} clusters [_how_to_upgrade_older_es_clusters]

Take a snapshot of the indices in the old cluster, delete indices that are not directly supported by ES 8 (i.e. indices older than 7.0), upgrade the cluster without the old indices, and then [restore](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-restore) the legacy indices from the snapshot or [mount](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-searchable-snapshots-mount) them via searchable snapshots.
To upgrade older {{es}} 5, 6, or 7 clusters:

In the future, we plan on streamlining the upgrade process going forward, making it easier to take legacy indices along when going to future major {{es}} versions.
1. Take a snapshot of the indices in the old cluster.
2. Delete any indices created before 8.0.0.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get confused about which ES version we are now covering. If it's about upgrading from 5.x or 6.x, the instructions don't change in 9.0 compared to before? I think that this 8.0.0 should actually still be 7.0.0. Does that make sense?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That depends probably if you need to write to the index or not. 7.0 indices won't block the upgrade if in read-only mode so maybe we could expand this to describe this or link to relevant docs elsewhere?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that yes, but I think the version that was incremented from 7.0 to 8.0 per-se is wrong?

3. Upgrade the cluster without the old indices, then [restore](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-restore) the legacy indices from the snapshot or [mount](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-searchable-snapshots-mount) them using searchable snapshots.

% In the future, we plan on streamlining the upgrade process going forward, making it easier to take legacy indices along when going to future major {{es}} versions.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This last note no longer makes sense, isn't that what we have done with 9.0, meaning that you can bring along indices that were previously archived in 8.x (from 5.x or 6.x), as well as mark 7.x indices read-only and take them with you ? @cbuescher does that make sense to you?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"take legacy indices along when going to future major {{es}} versions"

I think this statement is ambiguous, like e.g. I could mean being able to write to legacy indices as well. We know that's not what it means or will probably ever mean but I think the intention when this was added was to say we will keep improving this. I'm not sure we need to keep this around, but I also don't think we've reached with with v9 yet, e.g. you cannot simply keep version 5/6 indices through an upgrade without remounting a snapshot as far as I know. Or am I wrong here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you cannot simply keep version 5/6 indices through an upgrade without remounting a snapshot as far as I know

yes you can!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then lets scratch that last note here, it that what you suggest?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes that's what I'd suggest, if you agree :)


Loading