RCORE-2057 New builder for assessing that compression is working correctly (#7667)

* update next-major

* specified part of new layout (new width encoding)

* new header format for compressed arrays

* code review

* code review

* start of classifying arrays for compression

* classification down to column types

* first attempt to cut through the BPlusTree madness

* [wip] start on 'type driven' write process

* all tests passing (but no compression enabled)

* enabled compression for signed integer leafs only

* removed some dubious constructions in cluster tree

* delete tmp array while classifying arrays

* enabled compression of links and backlinks (excl collections)

* also compress bplustree of integers/links (experimental)

* pref for compressing dicts (not working)

* wip

* wip

* finally: compressing collections (incl dicts)

* compressing timestamps now

* enabled compression on ObjectID, TypedLink and UUID

* also compressing Mixed properties (not list/dicts of Mixed)

* Array compression with collections in Mixed (#7412)

---------

Co-authored-by: Finn Schiermer Andersen <[email protected]>

* merge next-major + collection in mixed

* enable dynamic choice of compression method

* moved typed_write/typed_print for bptree into class

* Merge pull request #7432 from realm/fsa/clean_typed_write

moved typed_write/typed_print for bptree into class

* cleanup unrelated code changes

* fix compilation

* cleanup

* code review

* code review

* scaffolding for accessing unaligned memory inside compressed arrays

* introducing parallel scan for scanning in one go multiple compressed arrays

* Silence warning

* tests + point fix for upper bound

* fix ubsan

* added clickbench data ingestor

* Added 2 queries to clickbench

* separated out a clickquery benchmark

* separated out a clickquery benchmark

* update to clickbench and query

* fix warnings

* add new builder with compression always enabled for verifying that compression works correctly (#7524)

* attempt to fix windows builders

* silence warning windows

* silence warnings

---------

Co-authored-by: Finn Schiermer Andersen <[email protected]>
Co-authored-by: Jørgen Edelbo <[email protected]>
Co-authored-by: Finn Schiermer Andersen <[email protected]>

Author: 3 people

CMakeLists.txt

@@ -261,12 +261,13 @@ option(REALM_ENABLE_ALLOC_SET_ZERO "Zero all allocations." OFF)
 if(NOT EMSCRIPTEN)
     option(REALM_ENABLE_ENCRYPTION "Enable encryption." ON)
 endif()
-option(REALM_ENABLE_MEMDEBUG "Add additional memory checks" OFF)
-option(REALM_VALGRIND "Tell the test suite we are running with valgrind" OFF)
-option(REALM_SYNC_MULTIPLEXING "Enables/disables sync session multiplexing by default" ON)
+option(REALM_ENABLE_MEMDEBUG "Add additional memory checks." OFF)
+option(REALM_VALGRIND "Tell the test suite we are running with valgrind." OFF)
+option(REALM_SYNC_MULTIPLEXING "Enables/disables sync session multiplexing by default." ON)
 set(REALM_MAX_BPNODE_SIZE "1000" CACHE STRING "Max B+ tree node size.")
 option(REALM_ENABLE_GEOSPATIAL "Enable geospatial types and queries." ON)
 option(REALM_APP_SERVICES "Enable the default app services implementation." ON)
+option(REALM_COMPRESS "Compress all the arrays by default in flex format." OFF)
 
 # Find dependencies
 set(THREADS_PREFER_PTHREAD_FLAG ON)

evergreen/config.yml

@@ -139,6 +139,10 @@ functions:
 
           set_cmake_var realm_vars REALM_BUILD_COMMANDLINE_TOOLS BOOL "${build_command_line_tools|On}"
           set_cmake_var realm_vars REALM_ENABLE_ENCRYPTION BOOL "${enable_realm_encryption|On}"
+          if [[ -n "${compress|}" ]]; then
+              set_cmake_var realm_vars REALM_COMPRESS PATH "${cmake_toolchain_file}"
+          fi
+
 
           if [[ -n "${fetch_missing_dependencies|}" ]]; then
               set_cmake_var realm_vars REALM_FETCH_MISSING_DEPENDENCIES BOOL On
@@ -1704,6 +1708,23 @@ buildvariants:
   - name: compile_test_coverage
   - name: finalize_coverage_data
 
+- name: macos-array-compression
+  display_name: "MacOS 11 arm64 (Compress Arrays)"
+  run_on: macos-1100-arm64
+  expansions:
+    cmake_url: "https://s3.amazonaws.com/static.realm.io/evergreen-assets/cmake-3.26.3-macos-universal.tar.gz"
+    cmake_bindir: "./cmake_binaries/CMake.app/Contents/bin"
+    cmake_toolchain_file: "./tools/cmake/xcode.toolchain.cmake"
+    cmake_generator: Xcode
+    max_jobs: $(sysctl -n hw.logicalcpu)
+    xcode_developer_dir: /Applications/Xcode13.1.app/Contents/Developer
+    extra_flags: -DCMAKE_SYSTEM_NAME=Darwin -DCMAKE_OSX_ARCHITECTURES=arm64
+    compress: On
+    cmake_build_type: Debug
+    coveralls_flag_name: "macos-arm64"
+  tasks:
+  - name: compile_test
+
 - name: windows-64-vs2019
   display_name: "Windows x86_64 (VS 2019)"
   run_on: windows-vsCurrent-large

src/realm/util/config.h.in

@@ -21,3 +21,4 @@
 #cmakedefine01 REALM_VALGRIND
 #cmakedefine01 REALM_ASAN
 #cmakedefine01 REALM_TSAN
+#cmakedefine01 REALM_COMPRESS