[common] Introduce BitmapFileIndexMetaV2. by hang8929201 · Pull Request #5028 · apache/paimon

hang8929201 · 2025-02-06T08:24:00Z

Purpose

When the bitmap-indexed column cardinality is high, using the first version of the bitmap index format will take a lot of time to read the entire dictionary. But in fact we don't need a full dictionary when dealing with a small number of predicates, the performance of predicate hits on the bitmap can be improved by creating a secondary index on the dictionary.

https://docs.google.com/document/d/11dJlGlSX3dwYKKrPN0DQ2XQTsx6d9wI6DTBIiiBwomM/edit?tab=t.0

performance

cardinality 1000:

cardinality 10000:

cardinality 30000:

cardinality 50000:

cardinality 80000:

cardinality 100000:

Tests

org.apache.paimon.fileindex.bitmapindex.TestBitmapFileIndex
org.apache.paimon.spark.SparkFileIndexITCase
org.apache.paimon.benchmark.bitmap.BitmapIndexBenchmark

API and Format

Documentation

docs/content/concepts/spec/fileindex.md

…apFileIndex.

paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndexMetaV2.java

paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndex.java

Tan-JiaLiang

@hang8929201 Thanks for your patience!

+1 cc @JingsongLi

2k124 · 2025-02-08T02:04:49Z

+1

hang8929201 · 2025-02-08T03:39:33Z

I found another question that needs to be discussed. Should the index version default to v1 or v2? @Tan-JiaLiang @JingsongLi

docs/content/append-table/query-performance.md

docs/content/concepts/spec/fileindex.md

JingsongLi · 2025-02-09T11:50:26Z

I found another question that needs to be discussed. Should the index version default to v1 or v2? @Tan-JiaLiang @JingsongLi

I don't have a clear idea, although it may cause compatibility issues, bitmap should not be used too much. Maintaining the default version to 1 is a relatively conservative choice.

JingsongLi · 2025-02-09T12:16:10Z

paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndex.java

+    public static final String VERSION = "version";
+    public static final String INDEX_BLOCK_SIZE = "index-block-size";
+    public static final String ENABLE_BUFFERED_INPUT = "enable-buffered-input";
+    public static final String ENABLE_NEXT_OFFSET_TO_SIZE = "enable-next-offset-to-size";


What is enable next offset to size? Under what circumstances can it not be size?

This parameter is just a switch for the benchmark test.

JingsongLi · 2025-02-09T12:19:30Z

paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndexMetaV2.java

+ * ｜ ...                                            ｜
+ * +-------------------------------------------------+-----------------
+ *
+ * index block format:


Maybe we should have ability to compress index block?

I feel that it is usually more appropriate to compress the data block, our 'data block' is the roaring bitmap, which is relatively well compressed. The index block is relatively small and has high query performance requirements. I guess there is no need to compress it. Maybe in the future when there is a bad case in the business, we will design a compressed format.

JingsongLi · 2025-02-10T01:29:57Z

paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndexMetaV2.java

+        Options options;
+
+        void tryDeserialize() {
+            if (entryList == null) {


Maybe you can take a look to https://cwiki.apache.org/confluence/display/PAIMON/PIP-25%3A+Introduce+a+key-value+file+format+for+paimon+primary+key+table

Here just something like LevelDB sst block, we can store length to avoid deserialize all entries.

Sorry, I don't quite understand what you mean. The purpose of designing secondary indexes is to read only part of the entries.

hang8929201 · 2025-02-08T03:46:58Z

paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndex.java

+                                    bitmapOffsets,
+                                    offsetRef[0]);
+                } else {
+                    throw new RuntimeException("invalid version: " + version);


When the writer is upgraded to the latest version and the default index version is v2, if the reader is not upgraded, an error will be reported here. Is it more appropriate to use the v1 format by default or force users to upgrade the paimon version by reporting an error? @Tan-JiaLiang @JingsongLi

hang8929201 · 2025-02-10T03:29:02Z

paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndex.java

+    public static final String VERSION = "version";
+    public static final String INDEX_BLOCK_SIZE = "index-block-size";
+    public static final String ENABLE_BUFFERED_INPUT = "enable-buffered-input";
+    public static final String ENABLE_NEXT_OFFSET_TO_SIZE = "enable-next-offset-to-size";


This parameter is just a switch for the benchmark test.

hang8929201 · 2025-02-10T07:30:19Z

paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndexMetaV2.java

+ * ｜ ...                                            ｜
+ * +-------------------------------------------------+-----------------
+ *
+ * index block format:


I feel that it is usually more appropriate to compress the data block, our 'data block' is the roaring bitmap, which is relatively well compressed. The index block is relatively small and has high query performance requirements. I guess there is no need to compress it. Maybe in the future when there is a bad case in the business, we will design a compressed format.

hang8929201 · 2025-02-10T07:37:30Z

paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndexMetaV2.java

+        Options options;
+
+        void tryDeserialize() {
+            if (entryList == null) {


Sorry, I don't quite understand what you mean. The purpose of designing secondary indexes is to read only part of the entries.

docs/content/concepts/spec/fileindex.md

docs/content/append-table/query-performance.md

JingsongLi

+1

guanshi added 13 commits February 6, 2025 15:49

[common] Introduce BitmapFileIndexMetaV2.

70e8d76

[common] Introduce BitmapFileIndexMetaV2.

494af0b

[common] Introduce BitmapFileIndexMetaV2.

efd3545

[common] Introduce BitmapFileIndexMetaV2.

32e9568

[common] Introduce BitmapFileIndexMetaV2.

313a5e2

[common] Add BitmapIndexBenchmark.

8279d71

[common] Add BitmapIndexBenchmark.

61815c0

[common] Add BitmapIndexBenchmark.

81eec8e

[common] Add BitmapIndexBenchmark.

ae96b55

[common] Support use byte buffer to deserialize RoaringBitmap in Bitm…

531ab01

…apFileIndex.

[common] Support use byte buffer to deserialize RoaringBitmap in Bitm…

d1daff3

…apFileIndex.

[common] BitmapFileIndexMetaV2

51d3edd

[common] BitmapFileIndexMetaV2

326b32e

hang8929201 mentioned this pull request Feb 6, 2025

[common] Introduce BitmapFileIndexMetaV2. #4956

Closed

Tan-JiaLiang reviewed Feb 6, 2025

View reviewed changes

paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndexMetaV2.java Show resolved Hide resolved

Tan-JiaLiang reviewed Feb 6, 2025

View reviewed changes

paimon-common/src/main/java/org/apache/paimon/fileindex/bitmap/BitmapFileIndex.java Show resolved Hide resolved

guanshi added 2 commits February 7, 2025 11:06

[common] binary search

63472e0

[common] find entry.

03b5310

Tan-JiaLiang approved these changes Feb 7, 2025

View reviewed changes

JingsongLi reviewed Feb 9, 2025

View reviewed changes

docs/content/append-table/query-performance.md Outdated Show resolved Hide resolved

JingsongLi reviewed Feb 9, 2025

View reviewed changes

docs/content/concepts/spec/fileindex.md Show resolved Hide resolved

JingsongLi reviewed Feb 9, 2025

View reviewed changes

JingsongLi reviewed Feb 10, 2025

View reviewed changes

guanshi added 2 commits February 10, 2025 15:40

[common] Change default bitmap index format version to V1.

3f5de5b

[common] Change default bitmap index format version to V1.

35cb8e8

hang8929201 commented Feb 24, 2025

View reviewed changes

JingsongLi approved these changes Feb 24, 2025

View reviewed changes

JingsongLi merged commit 958c1ec into apache:master Feb 24, 2025
13 checks passed

hang8929201 deleted the bitmap-multi-level-index branch March 12, 2025 11:11

Conversation

hang8929201 commented Feb 6, 2025

Purpose

Tests

API and Format

Documentation

Uh oh!

Uh oh!

Uh oh!

Tan-JiaLiang left a comment

Choose a reason for hiding this comment

Uh oh!

2k124 commented Feb 8, 2025

Uh oh!

hang8929201 commented Feb 8, 2025

Uh oh!

Uh oh!

Uh oh!

JingsongLi commented Feb 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants