Merge branch 'master' of github.com:FastFilter/FilterPassword_private

Author: lemire

README.md

@@ -3,7 +3,7 @@
 
 Suppose that you have a database made of half a billion passwords. You want to check quickly whether a given password is in this database. We allow a small probability of false positives (that can be later checked against the full database) but we otherwise do not want any false negatives (if the password is in the set, we absolutely want to know about it).
 
-The typical approach to this problem is to apply a Bloom filters. We test here an Xor Filter. The goal is for the filter to use very little memory.
+The typical approach to this problem is to apply a Bloom filters. We test here a binary fuse filter. The goal is for the filter to use very little memory.
 
 
 ## Requirement
@@ -15,11 +15,11 @@ Though the constructed filter may use only about a byte per set entry, the const
 
 ## Limitations
 
-The expectation is that the filter is built once. To build the filter over the full 550 million passwords, you currently need a machine with a sizeable amount of free RAM. It will almost surely fail on a laptop with 4GB or 8GB of RAM; 64 GB of RAM or more is recommended. We could further partition the problem (by dividing up the set) for lower memory usage or better parallelization.
+The expectation is that the filter is built once. To build the filter over the full 550 million passwords, you currently need a machine with a sizeable amount of free RAM. It will almost surely fail on a laptop with 4GB  of RAM; 64 GB of RAM or more is recommended. We could further partition the problem (by dividing up the set) for lower memory usage or better parallelization.
 
 Queries are very fast, however.
 
-We support up to 4 billion entries, if you have the available memory.
+We support hundreds of millions of entries and more, if you have the available memory.
 
 
 ## Preparing the data file
@@ -92,79 +92,54 @@ Expected number of queries per second: 17241.379
 ## Performance comparisons
 
 For a comparable false positive probability (about 0.3%), the Bloom filter uses more space
-and is slower. The main downside of the xor filter is a more expensive construction.
+and is slower. The binary fuse 8 uses less space, is constructed faster, and has faster queries.
 
 
 ```
-$ ./build_filter -m 10000000 -o xor8.bin -f xor8 pwned-passwords-sha1-ordered-by-count-v4.txt
+$ ./build_filter -m 10000000 -o binaryfuse8.bin -f binaryfuse8 pwned-passwords-sha1-ordered-by-count-v4.txt
 setting the max. number of entries to 10000000
 read 10000000 hashes.Reached the maximum number of lines 10000000.
-I read 10000000 hashes in total (0.902 seconds).
+I read 10000000 hashes in total (12.644 seconds).
 Bytes read = 452288199.
 Constructing the filter...
-Done in 1.303 seconds.
-filter data saved to xor8.bin. Total bytes = 12300054.
+Done in 0.420 seconds.
+filter data saved to binaryfuse8.bin. Total bytes = 11272228.
 
 
 $ ./build_filter -m 10000000 -o bloom12.bin -f bloom12 pwned-passwords-sha1-ordered-by-count-v4.txt
 setting the max. number of entries to 10000000
 read 10000000 hashes.Reached the maximum number of lines 10000000.
-I read 10000000 hashes in total (0.914 seconds).
+I read 10000000 hashes in total (12.696 seconds).
 Bytes read = 452288199.
 Constructing the filter...
-Done in 0.448 seconds.
+Done in 0.613 seconds.
 filter data saved to bloom12.bin. Total bytes = 15000024.
 
 
 
-$ for i in {1..3} ; do ./query_filter xor8.bin 7C4A8D09CA3762AF6 ; done
+$./query_filter binaryfuse8.bin 7C4A8D09CA3762AF6
+using database: binaryfuse8.bin
 hexval = 0x7c4a8d09ca3762af
-Xor filter detected.
-I expect the file to span 12300054 bytes.
-memory mapping is a success.
-Probably in the set.
-Processing time 88.000 microseconds.
-Expected number of queries per second: 11363.637
-hexval = 0x7c4a8d09ca3762af
-Xor filter detected.
-I expect the file to span 12300054 bytes.
-memory mapping is a success.
-Probably in the set.
-Processing time 59.000 microseconds.
-Expected number of queries per second: 16949.152
-hexval = 0x7c4a8d09ca3762af
-Xor filter detected.
-I expect the file to span 12300054 bytes.
-memory mapping is a success.
-Probably in the set.
-Processing time 68.000 microseconds.
-Expected number of queries per second: 14705.883
-
-$ for i in {1..3} ; do ./query_filter bloom12.bin 7C4A8D09CA3762AF6 ; done
-hexval = 0x7c4a8d09ca3762af
-Bloom filter detected.
-I expect the file to span 15000024 bytes.
-memory mapping is a success.
-Surely not in the set.
-Processing time 99.000 microseconds.
-Expected number of queries per second: 10101.010
-hexval = 0x7c4a8d09ca3762af
-Bloom filter detected.
-I expect the file to span 15000024 bytes.
+Binary fuse filter detected.
+I expect the file to span 11206692 bytes.
 memory mapping is a success.
 Surely not in the set.
-Processing time 88.000 microseconds.
-Expected number of queries per second: 11363.637
+Processing time 241.000 microseconds.
+Expected number of queries per second: 4149.377
+
+$ ./query_filter bloom12.bin 7C4A8D09CA3762AF6 ; done
+using database: bloom12.bin
 hexval = 0x7c4a8d09ca3762af
 Bloom filter detected.
 I expect the file to span 15000024 bytes.
 memory mapping is a success.
-Surely not in the set.
-Processing time 86.000 microseconds.
-Expected number of queries per second: 11627.907
+Probably in the set.
+Processing time 344.000 microseconds.
+Expected number of queries per second: 2906.977
 ```
 
 ## Reference
 
+* Thomas Mueller Graf,  Daniel Lemire, Binary Fuse Filters: Fast and Smaller Than Xor Filters
 * Thomas Mueller Graf,  Daniel Lemire, [Xor Filters: Faster and Smaller Than Bloom and Cuckoo Filters](https://arxiv.org/abs/1912.08258), Journal of Experimental Algorithmics 25 (1), 2020. DOI: 10.1145/3376122
 

src/build_filter.cpp

@@ -11,13 +11,12 @@
 
 #include "bloom/bloom.h"
 #include "hexutil.h"
-#include "xorfilter/xorfilter.h"
-#include "xor_singleheader/include/xorfilter.h"
 #include "xor_singleheader/include/binaryfusefilter.h"
+#include "xor_singleheader/include/xorfilter.h"
+#include "xorfilter/xorfilter.h"
 
 #include "mappeablebloomfilter.h"
 
-
 static void printusage(char *command) {
   printf(" Try %s -f xor8 -o filter.bin mydatabase \n", command);
   ;
@@ -26,9 +25,8 @@ static void printusage(char *command) {
   printf("The -V flag verifies the resulting filter.\n");
 }
 
-
-
-uint64_t * read_data(const char *filename, size_t & array_size, size_t maxline, bool printall) {
+uint64_t *read_data(const char *filename, size_t &array_size, size_t maxline,
+                    bool printall) {
   char *line = NULL;
   size_t line_capacity = 0;
   int read;
@@ -140,11 +138,11 @@ int main(int argc, char **argv) {
     }
 
   size_t array_size;
-  uint64_t * array;
-  if(synthetic) {
+  uint64_t *array;
+  if (synthetic) {
     array_size = synthetic_size;
     array = (uint64_t *)malloc(array_size * sizeof(uint64_t));
-    for(size_t i = 0; i < array_size; i++) {
+    for (size_t i = 0; i < array_size; i++) {
       array[i] = i;
     }
   } else {
@@ -154,13 +152,12 @@ int main(int argc, char **argv) {
     }
     const char *filename = argv[optind];
     array = read_data(filename, array_size, maxline, printall);
-    if(array == nullptr) {
+    if (array == nullptr) {
       return EXIT_FAILURE;
     }
   }
   clock_t start, end;
 
-
   printf("Constructing the filter...\n");
   fflush(NULL);
   if (strcmp("binaryfuse8", filtername) == 0) {
@@ -170,23 +167,25 @@ int main(int argc, char **argv) {
     binary_fuse8_populate(array, array_size, &filter);
     end = clock();
     printf("Done in %.3f seconds.\n", (float)(end - start) / CLOCKS_PER_SEC);
-    if(verify) {
+    if (verify) {
       printf("Checking for false negatives\n");
-      for(size_t i = 0; i < array_size; i++) {
-        if(!binary_fuse8_contain(array[i],&filter)) {
-          printf("Detected a false negative. You probably have a bug. Aborting.\n");
+      for (size_t i = 0; i < array_size; i++) {
+        if (!binary_fuse8_contain(array[i], &filter)) {
+          printf("Detected a false negative. You probably have a bug. "
+                 "Aborting.\n");
           return EXIT_FAILURE;
         }
       }
       printf("Verified with success: no false negatives\n");
       size_t matches = 0;
       size_t volume = 100000;
-      for(size_t t = 0; t < volume; t++) {
-        if(binary_fuse8_contain( t * 10001 + 13 + array_size,&filter)) {
+      for (size_t t = 0; t < volume; t++) {
+        if (binary_fuse8_contain(t * 10001 + 13 + array_size, &filter)) {
           matches++;
         }
       }
-      printf("estimated false positive rate: %.3f percent\n", matches * 100.0 / volume);
+      printf("estimated false positive rate: %.3f percent\n",
+             matches * 100.0 / volume);
     }
     free(array);
 
@@ -198,21 +197,24 @@ int main(int argc, char **argv) {
     }
     uint64_t cookie = 1234569;
     bool isok = true;
-    size_t total_bytes = sizeof(cookie) + sizeof(filter.Seed) + sizeof(filter.SegmentLength)
-                         + sizeof(filter.SegmentLengthMask) + sizeof(filter.SegmentCount)
-                         + sizeof(filter.SegmentCountLength) + sizeof(filter.ArrayLength)
-                         + sizeof(uint8_t) * filter.ArrayLength;
-
+    size_t total_bytes =
+        sizeof(cookie) + sizeof(filter.Seed) + sizeof(filter.SegmentLength) +
+        sizeof(filter.SegmentLengthMask) + sizeof(filter.SegmentCount) +
+        sizeof(filter.SegmentCountLength) + sizeof(filter.ArrayLength) +
+        sizeof(uint8_t) * filter.ArrayLength;
 
     isok &= fwrite(&cookie, sizeof(cookie), 1, write_ptr);
     isok &= fwrite(&filter.Seed, sizeof(filter.Seed), 1, write_ptr);
-    isok &= fwrite(&filter.SegmentLength, sizeof(filter.SegmentLength), 1, write_ptr);
-    isok &= fwrite(&filter.SegmentLengthMask, sizeof(filter.SegmentLengthMask), 1, write_ptr);
-    isok &= fwrite(&filter.SegmentCount, sizeof(filter.SegmentCount), 1, write_ptr);
-    isok &= fwrite(&filter.SegmentCountLength, sizeof(filter.SegmentCountLength), 1, write_ptr);
-    isok &= fwrite(&filter.ArrayLength, sizeof(filter.ArrayLength), 1, write_ptr);
-
-
+    isok &= fwrite(&filter.SegmentLength, sizeof(filter.SegmentLength), 1,
+                   write_ptr);
+    isok &= fwrite(&filter.SegmentLengthMask, sizeof(filter.SegmentLengthMask),
+                   1, write_ptr);
+    isok &=
+        fwrite(&filter.SegmentCount, sizeof(filter.SegmentCount), 1, write_ptr);
+    isok &= fwrite(&filter.SegmentCountLength,
+                   sizeof(filter.SegmentCountLength), 1, write_ptr);
+    isok &=
+        fwrite(&filter.ArrayLength, sizeof(filter.ArrayLength), 1, write_ptr);
     isok &= fwrite(filter.Fingerprints, sizeof(uint8_t) * filter.ArrayLength, 1,
                    write_ptr);
     isok &= (fclose(write_ptr) == 0);
@@ -231,23 +233,25 @@ int main(int argc, char **argv) {
     xor8_buffered_populate(array, array_size, &filter);
     end = clock();
     printf("Done in %.3f seconds.\n", (float)(end - start) / CLOCKS_PER_SEC);
-    if(verify) {
+    if (verify) {
       printf("Checking for false negatives\n");
-      for(size_t i = 0; i < array_size; i++) {
-        if(!xor8_contain(array[i],&filter)) {
-          printf("Detected a false negative. You probably have a bug. Aborting.\n");
+      for (size_t i = 0; i < array_size; i++) {
+        if (!xor8_contain(array[i], &filter)) {
+          printf("Detected a false negative. You probably have a bug. "
+                 "Aborting.\n");
           return EXIT_FAILURE;
         }
       }
       printf("Verified with success: no false negatives\n");
       size_t matches = 0;
       size_t volume = 100000;
-      for(size_t t = 0; t < volume; t++) {
-        if(xor8_contain( t * 10001 + 13 + array_size,&filter)) {
+      for (size_t t = 0; t < volume; t++) {
+        if (xor8_contain(t * 10001 + 13 + array_size, &filter)) {
           matches++;
         }
       }
-      printf("estimated false positive rate: %.3f percent\n", matches * 100.0 / volume);
+      printf("estimated false positive rate: %.3f percent\n",
+             matches * 100.0 / volume);
     }
     free(array);
 
@@ -281,36 +285,39 @@ int main(int argc, char **argv) {
     start = clock();
     using Table = bloomfilter::BloomFilter<uint64_t, 12, false, SimpleMixSplit>;
     Table table(array_size);
-    for(size_t i = 0; i < array_size; i++) {
+    for (size_t i = 0; i < array_size; i++) {
       table.Add(array[i]);
     }
     end = clock();
     printf("Done in %.3f seconds.\n", (float)(end - start) / CLOCKS_PER_SEC);
-    if(verify) {
+    if (verify) {
       printf("Checking for false negatives\n");
-      for(size_t i = 0; i < array_size; i++) {
-        if(table.Contain(array[i]) != bloomfilter::Ok) {
-          printf("Detected a false negative. You probably have a bug. Aborting.\n");
+      for (size_t i = 0; i < array_size; i++) {
+        if (table.Contain(array[i]) != bloomfilter::Ok) {
+          printf("Detected a false negative. You probably have a bug. "
+                 "Aborting.\n");
           return EXIT_FAILURE;
         }
       }
-      MappeableBloomFilter<12> filter(
-        table.SizeInBytes() / 8, table.hasher.seed, table.data);
-      for(size_t i = 0; i < array_size; i++) {
-        if(!filter.Contain(array[i])) {
-          printf("Detected a false negative. You probably have a bug. Aborting.\n");
+      MappeableBloomFilter<12> filter(table.SizeInBytes() / 8,
+                                      table.hasher.seed, table.data);
+      for (size_t i = 0; i < array_size; i++) {
+        if (!filter.Contain(array[i])) {
+          printf("Detected a false negative. You probably have a bug. "
+                 "Aborting.\n");
           return EXIT_FAILURE;
         }
       }
       printf("Verified with success: no false negatives\n");
       size_t matches = 0;
       size_t volume = 100000;
-      for(size_t t = 0; t < volume; t++) {
-        if(filter.Contain( t * 10001 + 13 + array_size)) {
+      for (size_t t = 0; t < volume; t++) {
+        if (filter.Contain(t * 10001 + 13 + array_size)) {
           matches++;
         }
       }
-      printf("estimated false positive rate: %.3f percent\n", matches * 100.0 / volume);
+      printf("estimated false positive rate: %.3f percent\n",
+             matches * 100.0 / volume);
     }
     free(array);
     FILE *write_ptr;