Switching to a fixed CFS threshold

[shubhamsrkdev]

Problem

Lucene uses CFS ratio (which is by default 10%) to determine whether to use Compound Files.

Solution

In this PR we are doing the following:

• Switching from a CFS ratio to a fixed threshold which is:

  • 64MB for byte-size-based merge policies
  • 65,536 docs for doc-based merge policies

• Moved CFS configuration to CompoundFormat.java from Merge policies.

Fixes #14959

[epotyom]

It would be great if we can avoid customizing it for specific policies, otherwise it might be tricky to maintain in the future, if e.g. there is another policy that is based on doc size not bytes.

Maybe we can add a enum and a method to MergePolicy which returns its unit (bytes/docs) , and use it here to decide which threshold to use?

Or do we want to always choose compound format based on size in bytes even for LogDocMergePolicy? In this case we might be able to use merge.getMergeInfo().sizeInBytes() when we call this method and avoid relying on MergePolicy#size all together?

[shubhamsrkdev]

+1 to the idea of Enums, I will wait if anyone else has other suggestions here but having an enum makes most sense to me.

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

[stefanvodita]

I had looked at this PR before, but I see it's gone stale. Hoping to revive it.
Surprised how many changes we needed for this!

[stefanvodita]

Thank you for addressing my comments! There are two questions left for me from the comments:

1. Whether the thresholds we're starting with are good enough for a first iteration.
2. Whether we can avoid branching logic based on which merge policy is in use.

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

[shubhamsrkdev]

We had to disable compound files here. The reason being, before this change we had a CFS ratio (10%) which in this case would not result in compound file creation. However since we switched to a fixed threshold (64 MB now) these files are way below the threshold leading to compound file creation and thus increase in file sizes. The increase is minimal (4 bytes for me below the original threshold) but enough to fail the tests. Threefore, I have disabled compound files here.

[stefanvodita]

Thank you for following up on the remaining comments. I'll try to merge soon to avoid more merge conflicts for this large PR.

[romseygeek]

This looks like it's causing failures in some backwards-codecs tests:

TestLucene87StoredFieldsFormatHighCompression > testMergeStability FAILED
--
java.lang.AssertionError: expected:<{fnm=77, fdt=37004, fdx=64, fdm=159}> but was:<{cfs=37366, cfe=150}>
at __randomizedtesting.SeedInfo.seed([DF7E25E7C981E11B:AB3263C8C46BE3AD]:0)
at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:120)
at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:146)
at org.apache.lucene.test_framework@11.0.0-SNAPSHOT/org.apache.lucene.tests.index.BaseIndexFileFormatTestCase.testMergeStability(BaseIndexFileFormatTestCase.java:306)

gradlew test --tests TestLucene87StoredFieldsFormatHighCompression.testMergeStability -Dtests.seed=DF7E25E7C981E11B -Dtests.nightly=true -Dtests.locale=ig-Latn-NG -Dtests.timezone=Asia/Qostanay -Dtests.asserts=true -Dtests.file.encoding=UTF-8

I think these are nightly-only so may have been missed by normal test runs.

[stefanvodita]

Thank you for pointing out the issue @romseygeek and thank you for addressing it quickly @shubhamsrkdev!