Skip to content

Upgrade to Lucene 10 #114741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 580 commits into from
Oct 21, 2024
Merged

Upgrade to Lucene 10 #114741

merged 580 commits into from
Oct 21, 2024

Conversation

javanna
Copy link
Member

@javanna javanna commented Oct 14, 2024

No description provided.

elasticsearchmachine and others added 30 commits September 9, 2024 06:11
[Automated] Update Lucene snapshot to 10.0.0-snapshot-dc47adbbe73
5ff75bb
[Automated] Update Lucene snapshot to 9.12.0-snapshot-371fa57d9c7
e0e2e1e
@cbuescher
Remove IndexVersions.V_7_6_0
3267fd0
@cbuescher
Remove IndexVersions.V_7_7_0 - V_7_11_0
74040a5
@cbuescher
Remove remainign V_7x IndexVersions
70ee548
@cbuescher
Uncomment some tests for better mergability with main
db7463f
@cbuescher
Merge branch 'main' into lucene_snapshot_10
95bc9d3
@cbuescher
Fix errors in imports
d0a1110
[Automated] Update Lucene snapshot to 10.0.0-snapshot-64f5697f537
afec00b
[Automated] Update Lucene snapshot to 9.12.0-snapshot-ce23e15eb54
304a1c6
@cbuescher
Merge branch 'main' into lucene_snapshot_10
d104bc3
@cbuescher
Remove usages of random versions before V_8_0_0 in tests
8fe6c3a
@cbuescher
More test fixed related to legacy versions
8a01884
@cbuescher
Mute CompositeRolesStoreTests.testXPackUserCanAccessNonRestrictedIndices
09b68e2
@cbuescher
Fix failing knn yaml test
ca97d86
@cbuescher
Fix docs test related to knn queries
84d4e86
@cbuescher
Null out missing codec in BWCLucene70Codec
77f24ae 
This temporarily gets us past missing codec that we attempt to load at node
startup, even if we don't use it.
@cbuescher
Partially revert previous commit
3cea216
[Automated] Update Lucene snapshot to 9.12.0-snapshot-7964682ddf5
8bf41b8
[Automated] Update Lucene snapshot to 10.0.0-snapshot-7c529ce092d
fff3fbb
@cbuescher
Fix compilation issues after last Lucene 10 snapshot merge
0ee5e4f
@cbuescher
Merge branch 'main' into lucene_snapshot_10
2d31a70
@cbuescher
Fix UOE in ContextIndexSearcher after last Lucene merge
ced0f7b
[Automated] Update Lucene snapshot to 10.0.0-snapshot-74e3c44063a
2430fff
[Automated] Update Lucene snapshot to 9.12.0-snapshot-ab262f917d4
d45dbc7
@cbuescher
Fix compile issues after latest Lucene snapshot update
d24200e
@cbuescher
Merge branch 'main' into lucene_snapshot_10
42c4ed1 
 Conflicts:
	build-tools-internal/version.properties
	modules/analysis-common/src/test/java/org/elasticsearch/analysis/common/EdgeNGramTokenizerTests.java
	modules/analysis-common/src/test/java/org/elasticsearch/analysis/common/SynonymsAnalysisTests.java
	modules/reindex/src/internalClusterTest/java/org/elasticsearch/migration/MultiFeatureMigrationIT.java
	plugins/analysis-phonetic/src/test/java/org/elasticsearch/plugin/analysis/phonetic/AnalysisPhoneticFactoryTests.java
	qa/rolling-upgrade-legacy/src/test/java/org/elasticsearch/upgrades/RecoveryIT.java
	server/src/internalClusterTest/java/org/elasticsearch/repositories/IndexSnapshotsServiceIT.java
	server/src/test/java/org/elasticsearch/action/admin/cluster/stats/VersionStatsTests.java
	server/src/test/java/org/elasticsearch/cluster/metadata/MetadataCreateIndexServiceTests.java
	server/src/test/java/org/elasticsearch/cluster/metadata/MetadataTests.java
	server/src/test/java/org/elasticsearch/index/IndexVersionTests.java
	server/src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java
	server/src/test/java/org/elasticsearch/index/mapper/DateFieldMapperTests.java
	server/src/test/java/org/elasticsearch/index/mapper/DynamicTemplatesTests.java
	server/src/test/java/org/elasticsearch/index/mapper/FieldNamesFieldMapperTests.java
	server/src/test/java/org/elasticsearch/index/mapper/ParametrizedMapperTests.java
	server/src/test/java/org/elasticsearch/index/mapper/vectors/BinaryDenseVectorScriptDocValuesTests.java
	server/src/test/java/org/elasticsearch/index/similarity/SimilarityServiceTests.java
	server/src/test/java/org/elasticsearch/indices/analysis/AnalysisModuleTests.java
	server/src/test/java/org/elasticsearch/script/VectorScoreScriptUtilsTests.java
	server/src/test/java/org/elasticsearch/script/field/vectors/DenseVectorTests.java
	test/framework/src/main/java/org/elasticsearch/index/mapper/MetadataMapperTestCase.java
	test/framework/src/main/java/org/elasticsearch/snapshots/AbstractSnapshotIntegTestCase.java
	x-pack/plugin/deprecation/src/test/java/org/elasticsearch/xpack/deprecation/IndexDeprecationChecksTests.java
	x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/ShapeFieldMapperTests.java
	x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/query/LegacyGeoShapeWithDocValuesQueryTests.java
@cbuescher
spotless
8800107
@cbuescher
Fixing more checkstyle and spotless issues after merging main
84bb2b4
@mayya-sharipova
Remove awaitsFix, as the issue was fixed
e916319
cbuescher
cbuescher approved these changes Oct 15, 2024
View reviewed changes
Copy link
Member

@cbuescher cbuescher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've done a final pass (non-test code only), left a few notes where I am unsure but nothing really sticks out. LGTM

uschindler
uschindler reviewed Oct 15, 2024
View reviewed changes
Copy link
Contributor

@uschindler uschindler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine. 🍺🚀

@cbuescher cbuescher mentioned this pull request Oct 16, 2024
Merged
@javanna javanna added test-windows Trigger CI checks on Windows test-full-bwc Trigger full BWC version matrix tests test-arm Pull Requests that should be tested against arm agents labels Oct 17, 2024
@javanna javanna removed the request for review from benwtrent October 17, 2024 07:57
@javanna
Copy link
Member Author

javanna commented Oct 20, 2024

run elasticsearch-ci/part-3-windows

@javanna
Copy link
Member Author

javanna commented Oct 20, 2024

run elasticsearch-ci/part-1-windows

@javanna
Copy link
Member Author

javanna commented Oct 20, 2024

run elasticsearch-ci/part-2-windows

@javanna
Copy link
Member Author

javanna commented Oct 20, 2024

run elasticsearch-ci/part-3-windows

@javanna
Copy link
Member Author

javanna commented Oct 20, 2024

run elasticsearch-ci/part-4-windows

@javanna
Copy link
Member Author

javanna commented Oct 21, 2024

The checkstyle errors from windows builds are present in main too, and are being fixed by #115185 .

@javanna
Copy link
Member Author

javanna commented Oct 21, 2024

Windows tests failed with the following which is unrelated to the changes made in this PR:

REPRODUCE WITH: gradlew ":server:test" --tests "org.elasticsearch.reservedstate.service.FileSettingsServiceTests.testProcessFileChanges" -Dtests.seed=98579A98909F778 -Dtests.locale=ro-MD -Dtests.timezone=America/Kentucky/Louisville -Druntime.java=22
FileSettingsServiceTests > testProcessFileChanges FAILED
    java.nio.file.AccessDeniedException: C:\bk\server\build\testrun\test\temp\org.elasticsearch.reservedstate.service.FileSettingsServiceTests_98579A98909F778-001\tempFile-001.tmp -> C:\bk\server\build\testrun\test\temp\org.elasticsearch.reservedstate.service.FileSettingsServiceTests_98579A98909F778-001\tempDir-024\config\operator\settings.json
        at __randomizedtesting.SeedInfo.seed([98579A98909F778:210DAE66E062FF98]:0)
        at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
        at java.base/sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:328)
        at java.base/sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:291)
        at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:146)
        at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:146)
        at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:146)
        at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:146)
        at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:146)
        at java.base/java.nio.file.Files.move(Files.java:1430)
        at org.elasticsearch.reservedstate.service.FileSettingsServiceTests.writeTestFile(FileSettingsServiceTests.java:358)
        at org.elasticsearch.reservedstate.service.FileSettingsServiceTests.testProcessFileChanges(FileSettingsServiceTests.java:253)

The checkstyle errors on kql are also present in main, like I mentioned above.

@javanna javanna merged commit 8efd08b into elastic:main Oct 21, 2024
21 of 26 checks passed
@javanna javanna deleted the lucene_snapshot_10 branch October 21, 2024 11:38
@arteam arteam mentioned this pull request Oct 23, 2024
Merged
georgewallace pushed a commit to georgewallace/elasticsearch that referenced this pull request Oct 25, 2024
@javanna @cbuescher
@mayya-sharipova @ChrisHegarty @brianseeders @original-brownbear @pmpailis @benwtrent @georgewallace
Upgrade to Lucene 10 (elastic#114741)
5b0d31f 
The most relevant ES changes that upgrading to Lucene 10 requires are:

- use the appropriate IOContext
- Scorer / ScorerSupplier breaking changes
- Regex automaton are no longer determinized by default
- minimize moved to test classes
- introduce Elasticsearch900Codec
- adjust slicing code according to the added support for intra-segment concurrency
- disable intra-segment concurrency in tests
- adjust accessor methods for many Lucene classes that became a record
- adapt to breaking changes in the analysis area

Co-authored-by: Christoph Büscher <[email protected]>
Co-authored-by: Mayya Sharipova <[email protected]>
Co-authored-by: ChrisHegarty <[email protected]>
Co-authored-by: Brian Seeders <[email protected]>
Co-authored-by: Armin Braun <[email protected]>
Co-authored-by: Panagiotis Bailis <[email protected]>
Co-authored-by: Benjamin Trent <[email protected]>
jfreden pushed a commit to jfreden/elasticsearch that referenced this pull request Nov 4, 2024
@javanna @cbuescher
@mayya-sharipova @ChrisHegarty @brianseeders @original-brownbear @pmpailis @benwtrent @jfreden
Upgrade to Lucene 10 (elastic#114741)
f9872fe 
The most relevant ES changes that upgrading to Lucene 10 requires are:

- use the appropriate IOContext
- Scorer / ScorerSupplier breaking changes
- Regex automaton are no longer determinized by default
- minimize moved to test classes
- introduce Elasticsearch900Codec
- adjust slicing code according to the added support for intra-segment concurrency
- disable intra-segment concurrency in tests
- adjust accessor methods for many Lucene classes that became a record
- adapt to breaking changes in the analysis area

Co-authored-by: Christoph Büscher <[email protected]>
Co-authored-by: Mayya Sharipova <[email protected]>
Co-authored-by: ChrisHegarty <[email protected]>
Co-authored-by: Brian Seeders <[email protected]>
Co-authored-by: Armin Braun <[email protected]>
Co-authored-by: Panagiotis Bailis <[email protected]>
Co-authored-by: Benjamin Trent <[email protected]>
@javanna javanna mentioned this pull request Feb 13, 2025
Closed
@pawankartik-elastic pawankartik-elastic mentioned this pull request May 9, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rjernst rjernst rjernst left review comments

@cbuescher cbuescher cbuescher approved these changes

@ChrisHegarty ChrisHegarty ChrisHegarty approved these changes

+1 more reviewer

@uschindler uschindler uschindler left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

:Search Foundations/Search Catch all for Search Foundations serverless-linked Added by automation, don't add manually Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch test-arm Pull Requests that should be tested against arm agents test-full-bwc Trigger full BWC version matrix tests test-windows Trigger CI checks on Windows >upgrade v9.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.