Exact Cardinality Count extension (#18021)

* Add Bitmap64 extension

* Update max intermediate size

* Changes to Bitmap64 after build

* Replace cardinality count name

* Unit tests for bitmap64

* Unit tests for bitmap aggregator

* tidy up for tests and Counter

* Fix checkstyle

* Tidy README

* Tidy review suggestions

* Fix checkstyle

* Clean up dependencies in pom file

* Docs for extension

* Fix Spelling check

* Add wikipedia datasource walkthrough

* Add SQL Test

Revert "Add wikipedia datasource walkthrough"

This reverts commit 83dfef9.

Revert "Add SQL Test"

This reverts commit e81a0fdc2f07b71958bc32734abe16bacbac920d.

* Fix Cannot find class error

* Fix Assert cannot be found

* Fix test case

* Fix SqlAggregator test cases

* Fix checkstyle

* Fix registerSerde problem

* Add JavaDocs for interface

* Fix null check not passing

* Add dockerfiles for DruidExactCardinalityIT

* Add integration tests for Druid Exact Cardinality

* Checkstyle

* Change groupId to contrib

* Change docker file to keep things running

* Set fullDatasourceName after initialization to prevent NPE with config

* Fix adding contrib pom changes to distribution/pom.xml

* Setup IT for contrib packages

* Add DruidExactCardinality into IT

* Add case to check cardinality ignores duplicates

* Add case to check cardinality works on rolled-up columns:

* Fix fullDatasourceName can be a local variable

* Bitmap64 exactcount update

* Add ExactCount extension into github actions

* Fix resources/cluster docker file

* Add additional line to eof

* Change docs to specify columns

* Eliminate unnecessary byte array copying

* Change Bitmap64 function to filter out numeric types

* Prevent copying of array in Output stream

* Fix checkstyle

* Fix UnsupportedOperationException

* Fix SQL type allowable

* Remove unnecessary build config

* Add unit test for String Column

* Type checking for String in ExactCount

* Implement unit tests to confirm String column check

* Update difference between exact count + distinct count

* Fix spelling mistake

* Address review feedback P1 - Rework ExposedByteArrayOutputStream

* Address unclear exception message when decoding using base64

* Comments for 1KiB max intermediate size

* Allow Bitmap64 to take directly from data input

* Use underlying byte array

* Checkstyle

* Fix array writing

* Rename to druid-bitmap-exact-count

* Missing rename

* Update to snapshot 35

* Renamed to druid-exact-count-bitmap

* Druid exact count bitmap

---------

Co-authored-by: Phua Guan Wei <[email protected]>

Author: GWphua

.github/workflows/revised-its.yml

@@ -67,7 +67,7 @@ jobs:
       fail-fast: false
       matrix:
         jdk: [17]
-        it: [HighAvailability, MultiStageQuery, Catalog, BatchIndex, MultiStageQueryWithMM, InputSource, InputFormat, Security, Query]
+        it: [HighAvailability, MultiStageQuery, Catalog, BatchIndex, MultiStageQueryWithMM, InputSource, InputFormat, Security, Query, DruidExactCountBitmap]
         indexer: [middleManager]
     uses: ./.github/workflows/reusable-revised-its.yml
     if: ${{ needs.changes.outputs.core == 'true' || needs.changes.outputs.common-extensions == 'true' }}

distribution/pom.xml

@@ -462,6 +462,8 @@
                                         <argument>org.apache.druid.extensions.contrib:grpc-query</argument>
                                         <argument>-c</argument>
                                         <argument>org.apache.druid.extensions.contrib:druid-ranger-security</argument>
+                                        <argument>-c</argument>
+                                        <argument>org.apache.druid.extensions.contrib:druid-exact-count-bitmap</argument>
                                     </arguments>
                                 </configuration>
                             </execution>

docs/configuration/extensions.md

@@ -86,6 +86,7 @@ All of these community extensions can be downloaded using [pull-deps](../operati
 |druid-ddsketch|Support for DDSketch approximate quantiles based on [DDSketch](https://github.com/datadog/sketches-java) | [link](../development/extensions-contrib/ddsketch-quantiles.md)|
 |druid-deltalake-extensions|Support for ingesting Delta Lake tables.|[link](../development/extensions-contrib/delta-lake.md)|
 |druid-distinctcount|DistinctCount aggregator|[link](../development/extensions-contrib/distinctcount.md)|
+|druid-exact-count-bitmap|Support for exact cardinality counting using Roaring Bitmap over a Long column.|[link](../development/extensions-contrib/druid-exact-count-bitmap.md)|
 |druid-iceberg-extensions|Support for ingesting Iceberg tables.|[link](../development/extensions-contrib/iceberg.md)|
 |druid-redis-cache|A cache implementation for Druid based on Redis.|[link](../development/extensions-contrib/redis-cache.md)|
 |druid-time-min-max|Min/Max aggregator for timestamp.|[link](../development/extensions-contrib/time-min-max.md)|