Single loop for FieldfInfo processing #137967

burqen · 2025-11-12T14:40:57Z

Previously both DeduplicatingFieldInfosFormat and TSDBSyntheticIdCodec.RewriteFieldInfosFormat would iterate over FieldInfos.

This is now optimised by letting TSDBSyntheticIdCodec.RewriteFieldInfosFormat extend DeduplicatingFieldInfosFormat in order to let RewriteFieldInfosFormat utilise the iteration done by DeduplicatingFieldInfosFormat.

Also let TSDBSyntheticIdCodec extend DeduplicateFieldInfosCodec so that we can simplify the codec wrapping to always use only one of them.

Additional changes:

Elasticsearch versioned codex are always wrapped by DeduplicateFieldInfosCodec, instead of extending it. This makes it possible for TSDBSyntheticIdCodec to extend DeduplicateFieldInfosCodec and hopefully as a side effect make it easier to reason about.
Move DecouplingFieldInfosCodec to upper level.

No codec should extend DeduplicatingFieldInfoCodec, instead always wrap it.

Previously both DeduplicatingFieldInfosFormat and TSDBSyntheticIdCodec.RewriteFieldInfosFormat would iterate over FieldInfos. This is now optimised by letting TSDBSyntheticIdCodec.RewriteFieldInfosFormat extend DeduplicatingFieldInfosFormat in order to let RewriteFieldInfosFormat utilise the iteration done by DeduplicatingFieldInfosFormat. Also let TSDBSyntheticIdCodec extend DeduplicateFieldInfosCodec so that we can simplify the codec wrapping to always use only one of them.

elasticsearchmachine · 2025-11-12T14:41:22Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

elasticsearchmachine · 2025-11-12T14:41:23Z

Hi @burqen, I've created a changelog YAML for you.

…452-iterate-fields-once--inline

tlrx

LGTM

tlrx · 2025-11-13T08:45:06Z

server/src/main/java/org/elasticsearch/index/codec/CodecService.java

-            return codec;
+            String name = e.getValue().getName();
+            Codec codec = e.getValue();
+            return useTsdbSyntheticId ? new TSDBSyntheticIdCodec(name, codec) : new DeduplicateFieldInfosCodec(name, codec);


nit:

Suggested change

return useTsdbSyntheticId ? new TSDBSyntheticIdCodec(name, codec) : new DeduplicateFieldInfosCodec(name, codec);

return useTsdbSyntheticId ? new TSDBSyntheticIdCodec(codec) : new DeduplicateFieldInfosCodec(codec);

tlrx · 2025-11-13T08:47:11Z

server/src/main/java/org/elasticsearch/index/codec/DeduplicateFieldInfosCodec.java

+    protected DeduplicateFieldInfosCodec(String name, Codec delegate) {
+        super(name, delegate);
+        this.fieldInfosFormat = createFieldInfosFormat(delegate.fieldInfosFormat());
+    }


Suggested change

protected DeduplicateFieldInfosCodec(String name, Codec delegate) {

super(name, delegate);

this.fieldInfosFormat = createFieldInfosFormat(delegate.fieldInfosFormat());

}

protected DeduplicateFieldInfosCodec(Codec delegate) {

super(delegate.getName(), delegate);

this.fieldInfosFormat = createFieldInfosFormat(delegate.fieldInfosFormat());

}

tlrx · 2025-11-13T08:48:53Z

server/src/main/java/org/elasticsearch/index/codec/DeduplicatingFieldInfosFormat.java


+    protected void validateFieldInfos(FieldInfos fieldInfos) {}
+
+    protected FieldInfo processFieldInfo(FieldInfo fi) {


nit:

Suggested change

protected FieldInfo processFieldInfo(FieldInfo fi) {

protected FieldInfo wrapFieldInfo(FieldInfo fi) {

tlrx · 2025-11-13T08:52:33Z

server/src/test/java/org/elasticsearch/index/codec/CodecTests.java


+    private static Codec unwrappedCodec(CodecService codecService, String codecName) {
+        Codec codec = codecService.codec(codecName);
+        if (codec instanceof DeduplicateFieldInfosCodec deduplicatingCodec) {


I think this can be:

Suggested change

if (codec instanceof DeduplicateFieldInfosCodec deduplicatingCodec) {

if (codec instanceof FilterCoded filtered) {

Unfortunately not, the delegate is hidden from us there.

fcofdez

I'm afraid that we should revert the change around not extending DeduplicateFieldInfosCodec in the default codecs. The reason is that Lucene would use SPI to load the Codec and it will just instantiate the codec with the no-args constructor and thus we won't get to deduplicate the fields. This only applies when a node is restarted for example and we need to read the codec from the SegmentInfo and I guess that it applies to search nodes in serverless too.

burqen · 2025-11-13T13:45:44Z

I'm afraid that we should revert the change around not extending DeduplicateFieldInfosCodec in the default codecs. The reason is that Lucene would use SPI to load the Codec and it will just instantiate the codec with the no-args constructor and thus we won't get to deduplicate the fields. This only applies when a node is restarted for example and we need to read the codec from the SegmentInfo and I guess that it applies to search nodes in serverless too.

I can see that Lucene is used to load the codecs here, but they will all be wrapped afterwards. Is there some other place as well where the codecs are service loaded outside of CodecService? https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/index/codec/CodecService.java#L66-L69

fcofdez · 2025-11-13T14:02:14Z

Is there some other place as well where the codecs are service loaded outside of CodecService?

Yes, in Lucene, when a commit is read from disk (see https://github.com/apache/lucene/blob/e02bdb4c3c547488342b423e1b9b2b25519bd427/lucene/core/src/java/org/apache/lucene/index/SegmentInfos.java#L409-L412). We kind of rely implicitly that for reading a segment with a particular codec we can use the SPI loaded codec.

burqen · 2025-11-13T16:16:42Z

Is there some other place as well where the codecs are service loaded outside of CodecService?

Yes, in Lucene, when a commit is read from disk (see https://github.com/apache/lucene/blob/e02bdb4c3c547488342b423e1b9b2b25519bd427/lucene/core/src/java/org/apache/lucene/index/SegmentInfos.java#L409-L412). We kind of rely implicitly that for reading a segment with a particular codec we can use the SPI loaded codec.

Got it! Thanks for pointing that out. I'll revert and find some other approach.

burqen added 3 commits November 12, 2025 14:07

Decouple DeduplicatingFieldInfoCodec

c3f8de5

No codec should extend DeduplicatingFieldInfoCodec, instead always wrap it.

Move DecouplingFieldInfosCodec to upper level

46c9566

burqen added >enhancement :StorageEngine/TSDB You know, for Metrics v9.3.0 labels Nov 12, 2025

elasticsearchmachine added the Team:StorageEngine label Nov 12, 2025

Update docs/changelog/137967.yaml

21e6faf

burqen changed the title ~~Single loop for FielfInfo processing~~ Single loop for FieldfInfo processing Nov 12, 2025

burqen requested review from fcofdez and tlrx November 12, 2025 14:43

burqen and others added 4 commits November 12, 2025 16:54

Fix test failures expecting inheritance

cb9fdb7

Fix more test failures expecting inheritance

57da3f7

Merge remote-tracking branch 'upstream/main' into ap/2025.11.10.ES-13…

47cd28d

…452-iterate-fields-once--inline

[CI] Auto commit changes from spotless

bf75792

tlrx approved these changes Nov 13, 2025

View reviewed changes

fcofdez reviewed Nov 13, 2025

View reviewed changes

Address PR comments

8f21294

	return useTsdbSyntheticId ? new TSDBSyntheticIdCodec(name, codec) : new DeduplicateFieldInfosCodec(name, codec);
	return useTsdbSyntheticId ? new TSDBSyntheticIdCodec(codec) : new DeduplicateFieldInfosCodec(codec);


		protected void validateFieldInfos(FieldInfos fieldInfos) {}

		protected FieldInfo processFieldInfo(FieldInfo fi) {

	protected FieldInfo processFieldInfo(FieldInfo fi) {
	protected FieldInfo wrapFieldInfo(FieldInfo fi) {

	if (codec instanceof DeduplicateFieldInfosCodec deduplicatingCodec) {
	if (codec instanceof FilterCoded filtered) {

Single loop for FieldfInfo processing #137967

Are you sure you want to change the base?

Single loop for FieldfInfo processing #137967

Uh oh!

Conversation

burqen commented Nov 12, 2025

Uh oh!

elasticsearchmachine commented Nov 12, 2025

Uh oh!

elasticsearchmachine commented Nov 12, 2025

Uh oh!

tlrx left a comment

Choose a reason for hiding this comment

Uh oh!

tlrx Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

tlrx Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

tlrx Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

tlrx Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

burqen Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

fcofdez left a comment

Choose a reason for hiding this comment

Uh oh!

burqen commented Nov 13, 2025

Uh oh!

fcofdez commented Nov 13, 2025

Uh oh!

burqen commented Nov 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants