Clean out rejected experiments #3

Author: thomasmueller

README.md

@@ -6,13 +6,8 @@ This is a research library currently. It is not meant for production use.
 
 Developers might want to consider our [Header-only Xor Filter library in C](https://github.com/FastFilter/xor_singleheader/).
 
-## Reference
 
-* Thomas Mueller Graf,  Daniel Lemire, [Xor Filters: Faster and Smaller Than Bloom and Cuckoo Filters](https://arxiv.org/abs/1912.08258), Journal of Experimental Algorithmics 25 (1), 2020. DOI: 10.1145/3376122
-
-
-
-## Prerequisites_fastfilter
+## Prerequisites
 
 - A  C++11 compiler such as GNU G++ or LLVM Clang++
 - Make
@@ -44,17 +39,17 @@ Your results will depend on the hardware, on the compiler and how the system is
 
 ```
 $ ./bulk-insert-and-query.exe 10000000
-                                                    find    find    find    find    find                        optimal  wasted million
-                                     add  remove      0%     25%     50%     75%    100%        ε  bits/item  bits/item   space    keys
-
-add    cycles: 325.5/key, instructions: (303.2/key, 0.93/cycle) cache misses: 12.41/key branch misses: 1.17/key
-0.00%  cycles:  81.7/key, instructions: ( 48.0/key, 0.59/cycle) cache misses:  3.06/key branch misses: 0.00/key
-0.25%  cycles:  81.8/key, instructions: ( 48.0/key, 0.59/cycle) cache misses:  3.06/key branch misses: 0.00/key
-0.50%  cycles:  81.8/key, instructions: ( 48.0/key, 0.59/cycle) cache misses:  3.06/key branch misses: 0.00/key
-0.75%  cycles:  82.0/key, instructions: ( 48.0/key, 0.59/cycle) cache misses:  3.06/key branch misses: 0.00/key
-1.00%  cycles:  81.9/key, instructions: ( 48.0/key, 0.59/cycle) cache misses:  3.06/key branch misses: 0.00/key
-                            Xor8  106.79    0.00   25.92   25.88   25.86   25.94   25.98  0.3892%       9.84       8.01   22.9%    10.0
-
+./bulk-insert-and-query.exe 10000000
+                                                    find    find    find    find    find  1*add+                       optimal  wasted million
+                                     add  remove      0%     25%     50%     75%    100%  3*find      ε%  bits/item  bits/item  space%    keys
+
+add    cycles: 351.3/key, instructions: (332.4/key, 0.95/cycle) cache misses: 13.99/key branch misses: 1.23/key
+0.00%  cycles:  87.8/key, instructions: ( 48.0/key, 0.55/cycle) cache misses:  2.89/key branch misses: 0.00/key
+0.25%  cycles:  87.8/key, instructions: ( 48.0/key, 0.55/cycle) cache misses:  2.89/key branch misses: 0.00/key
+0.50%  cycles:  87.8/key, instructions: ( 48.0/key, 0.55/cycle) cache misses:  2.90/key branch misses: 0.00/key
+0.75%  cycles:  87.9/key, instructions: ( 48.0/key, 0.55/cycle) cache misses:  2.90/key branch misses: 0.00/key
+1.00%  cycles:  87.8/key, instructions: ( 48.0/key, 0.55/cycle) cache misses:  2.89/key branch misses: 0.00/key
+                            Xor8  106.59    0.00   23.85   23.85   23.85   23.88   23.83  178.15  0.3908       9.84       8.00    23.0  10.000
 ... # many more lines omitted
 ```
 
@@ -66,50 +61,48 @@ As part of the benchmark, we check the correctness of the implementation.
 
 ## Benchmarking
 
-The shell script `benchmark/benchmark.sh` runs the benchmark 3 times for the most important algorithms,
-with entry sizes of 10 million and 100 million keys.
-It is much slower than the above, because each invocation runs only one algorithm
-(to ensure running one algorithm doesn't influence benchmark results of other algorithms).
-It stores the results in the file `benchmark-results.txt`.
-To futher analyze the results, use the java tool `AnalyzeResults.java`
-from the project https://github.com/FastFilter/fastfilter_java.
-Requires GCC and Java 8.
+A simple way to run the benchmark for the most important algorithms
+is to run `./bulk-insert-and-query.exe <number of entries>`.
 To get a low error, it is best run on a Linux machine that is not otherwise in use.
-Steps to run the tests and analyze the results:
+You might want to run the tests multiple times to verify it is working properly.
+The steps to run the tests are:
 
     git clone https://github.com/FastFilter/fastfilter_cpp.git
-    git clone https://github.com/FastFilter/fastfilter_java.git
     cd fastfilter_cpp/benchmarks
     make clean ; make
-    # this may take an hour to run
-    ./benchmark.sh
-
-    cd ../..
-    cd fastfilter_java/fastfilter
-    mvn clean install
-    java -cp target/test-classes org.fastfilter.analysis.AnalyzeResults ../../fastfilter_cpp/benchmarks/benchmark-results.txt
-
+    # 100 million entries
+    ./bulk-insert-and-query.exe 100000000
 
 ## Where is your code?
 
 The filter implementations are in `src/<type>/`. Most implementations depend on `src/hashutil.h`. Examples:
 
 * src/bloom/bloom.h
 * src/xorfilter/xorfilter.h
+* src/xorfilter/3wise_xor_binary_fuse_filter_lowmem.h
+
+
+## References
+
+- [Xor Filters: Faster and Smaller Than Bloom and Cuckoo Filters](https://arxiv.org/abs/1912.08258), Journal of Experimental Algorithmics 25 (1), 2020
+
 
 ## Credit
 
 The cuckoo filter and the benchmark are derived from https://github.com/efficient/cuckoofilter by Bin Fan et al.
 The SIMD blocked Bloom filter is from https://github.com/apache/impala (via the cuckoo filter).
 The Morton filter is from https://github.com/AMDComputeLibraries/morton_filter.
 The Counting Quotient Filter (CQF) is from https://github.com/splatlab/cqf.
+The ribbon filters are from https://github.com/pdillinger/fastfilter_cpp.
 
 
-# Implementations of xor filters in other programming languages
+# Implementations of xor and binary fuse filters in other programming languages
 
-* [Go](https://github.com/FastFilter/xorfilter)
+* [C](https://github.com/FastFilter/xor_singleheader)
+* [C99](https://github.com/skeeto/xf8)
 * [Erlang](https://github.com/mpope9/exor_filter)
-* Rust: [1](https://github.com/bnclabs/xorfilter), [2](https://github.com/codri/xorfilter-rs), [3](https://github.com/Polochon-street/rustxorfilter)
+* [Go](https://github.com/FastFilter/xorfilter)
 * [Java](https://github.com/FastFilter/fastfilter_java)
-* [C](https://github.com/FastFilter/xor_singleheader)
 * [Python](https://github.com/GreyDireWolf/pyxorfilter)
+* Rust: [1](https://github.com/bnclabs/xorfilter), [2](https://github.com/codri/xorfilter-rs), [3](https://github.com/Polochon-street/rustxorfilter)
+* [Zig](https://github.com/hexops/xorfilter)

benchmarks/benchmark-fast.sh

@@ -0,0 +1,17 @@
+#!/bin/sh
+# run the benchmark multiple times with all important algorithms
+# for algorithm ids and other parameters, see
+# bulk-insert-and-query.cc
+#
+# rnd: random number generators to use
+for rnd in `seq -1 -1`; do
+  # m: number of entries (in millions)
+  for m in `seq 100 100`; do
+    # test: test id
+    for test in `seq 1 10`; do
+      sleep 5;
+      # default algorithms
+      ./bulk-insert-and-query.exe ${m}000000;
+    done;
+  done;
+done > benchmark-fast-results.txt 2>&1

benchmarks/benchmark.sh

@@ -5,14 +5,15 @@
 #
 # rnd: random number generators to use
 for rnd in `seq -1 -1`; do
-  # alg: algorithms to test
-  for alg in 0 2 3 4 11 12 13 15 16 17 20 30 40 41 42 51 80 100; do
-    # m: number of entries
-    for m in `seq 10 90 100`; do
-      # test: test id
-      for test in `seq 1 3`; do
+  # m: number of entries (in millions)
+  for m in `seq 100 100`; do
+    # test: test id
+    for test in `seq 1 20`; do
+      # alg: algorithms to test
+      for alg in 0 2 11 12 44 45 51 116 117 118 119 1086 1156 3086 3156; do
         now=$(date +"%T");
-        echo ${now} alg ${alg} size ${m} ${rnd};
+        echo ${test} ${now} alg ${alg} size ${m} ${rnd};
+        sleep 10;
         ./bulk-insert-and-query.exe ${m}000000 ${alg} ${rnd};
       done;
     done;

benchmarks/bulk-insert-and-query.cc

@@ -289,12 +289,13 @@ Statistics FilterBenchmark(
 #ifdef __linux__
     unified.end(results);
     printf("remove ");
-    printf("cycles: %5.1f/key, instructions: (%5.1f/key, %4.2f/cycle) cache misses: %5.2f/key branch misses: %4.2f/key\n",
+    printf("cycles: %5.1f/key, instructions: (%5.1f/key, %4.2f/cycle) cache misses: %5.2f/key branch misses: %4.2f/key effective frequency %4.2f GHz\n",
       results[0]*1.0/add_count,
       results[1]*1.0/add_count ,
       results[1]*1.0/results[0],
       results[2]*1.0/add_count,
-      results[3]*1.0/add_count);
+      results[3]*1.0/add_count,
+      results[0]*1.0/time);
 #else
     std::cout << "." << std::flush;
 #endif
@@ -347,7 +348,7 @@ int main(int argc, char * argv[]) {
     {0, "Xor8"}, {1, "Xor12"}, {2, "Xor16"},
     {3, "Xor+8"}, {4, "Xor+16"},
     {5, "Xor10"}, {6, "Xor10.666"},
-    {7, "Xor10 (NBitArray)"}, {8, "Xor14 (NBitArray)"}, {9, "Xor8-2^n"},
+    {7, "Xor10-NBitArray"}, {8, "Xor14-NBitArray"}, {9, "Xor8-2^n"},
     // Cuckooo
     {10,"Cuckoo8"}, {11,"Cuckoo12"}, {12,"Cuckoo16"},
     {13,"CuckooSemiSort13"},
@@ -363,27 +364,27 @@ int main(int argc, char * argv[]) {
 #endif
     // Bloom
     {40, "Bloom8"}, {41, "Bloom12" }, {42, "Bloom16"},
-    {43, "Bloom8 (addall)"}, {44, "Bloom12 (addall)"}, {45, "Bloom16 (addall)"},
-    {46, "BranchlessBloom8 (addall)"},
-    {47, "BranchlessBloom12 (addall)"},
-    {48, "BranchlessBloom16 (addall)"},
+    {43, "Bloom8-addAll"}, {44, "Bloom12-addAll"}, {45, "Bloom16-addAll"},
+    {46, "BranchlessBloom8-addAll"},
+    {47, "BranchlessBloom12-addAll"},
+    {48, "BranchlessBloom16-addAll"},
     // Blocked Bloom
     {50, "SimpleBlockedBloom"},
 #ifdef __aarch64__
     {51, "BlockedBloom"},
-    {52, "BlockedBloom (addall)"},
+    {52, "BlockedBloom-addAll"},
 #elif defined( __AVX2__)
     {51, "BlockedBloom"},
-    {52, "BlockedBloom (addall)"},
+    {52, "BlockedBloom-addAll"},
     {53, "BlockedBloom64"},
 #endif
 #ifdef __SSE41__
     {54, "BlockedBloom16"},
 #endif
 
     // Counting Bloom
-    {60, "CountingBloom10 (addall)"},
-    {61, "SuccCountingBloom10 (addall)"},
+    {60, "CountingBloom10-addAll"},
+    {61, "SuccCountingBloom10-addAll"},
     {62, "SuccCountBlockBloom10"},
     {63, "SuccCountBlockBloomRank10"},
 
@@ -393,8 +394,8 @@ int main(int argc, char * argv[]) {
 
     {80, "Morton"},
 
-    {96, "XorBinaryFuse8"},
-    {97, "XorBinaryFuse16"},
+    {96, "XorBinaryFuse8-Naive"},
+    {97, "XorBinaryFuse16-Naive"},
     {98, "XorBinaryFuse8-4Wise-Prefetch"},
     {99, "XorBinaryFuse16-4Wise-Prefetch"},
     {100, "XorBinaryFuse8-Prefetched"},
@@ -411,10 +412,10 @@ int main(int argc, char * argv[]) {
     {113, "XorBinaryFuse16-4Wise-PSorted"},
     {114, "XorBinaryFuse8-OneHash"},
     {115, "XorBinaryFuse16-OneHash"},
-    {116, "XorBinaryFuse8-LowMem"},
-    {117, "XorBinaryFuse16-LowMem"},
-    {118, "XorBinaryFuse8-4Wise-LowMem"},
-    {119, "XorBinaryFuse16-4Wise-LowMem"},
+    {116, "XorBinaryFuse8"},
+    {117, "XorBinaryFuse16"},
+    {118, "XorBinaryFuse8-4Wise"},
+    {119, "XorBinaryFuse16-4Wise"},
     {1056, "HomogRibbon64_5"},
     {1076, "HomogRibbon64_7"}, // interesting
     {1086, "HomogRibbon64_8"},
@@ -801,7 +802,7 @@ int main(int argc, char * argv[]) {
       cout << setw(NAME_WIDTH) << names[a] << cf << endl;
   }
   a = 41;
-  if (algorithmId == a  || (algos.find(a) != algos.end())) {
+  if (algorithmId == a || (algos.find(a) != algos.end())) {
       auto cf = FilterBenchmark<
           BloomFilter<uint64_t, 12, false, SimpleMixSplit>>(
           add_count, to_add, intersectionsize, mixed_sets);