[Improvement] Too many blocks generated by a single task for a partition can cause memory overflow

[xianjingfeng]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

Below is the result I printed using jmap.

 [root@xxx arthas]# grep 'org.roaringbitmap' ~/jmap_heap
     2:     241973220     5807357280  org.roaringbitmap.ArrayContainer
     3:      10627671     1560565296  [Lorg.roaringbitmap.Container;
     9:      10627671      255064104  org.roaringbitmap.RoaringArray
    12:      10627671      170042736  org.roaringbitmap.RoaringBitmap
   245:          3277          78648  org.roaringbitmap.BitmapContainer
   256:          1292          51680  org.roaringbitmap.longlong.Roaring64NavigableMap
   306:          1292          20672  org.roaringbitmap.RoaringBitmapSupplier
   307:          1292          20672  org.roaringbitmap.longlong.RoaringIntPacking$1
   325:           645          15480  [Lorg.roaringbitmap.longlong.Roaring64NavigableMap; 

We can see that there are few org.roaringbitmap.longlong.Roaring64NavigableMap, but a lot of org.roaringbitmap.RoaringBitmap.
Based on the current blockId design, this situation will arise when a single task generates many blocks for a single partition.

How should we improve?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!