'Random Binary Projection' algorithm added (#228)

Author: gyrdym

CHANGELOG.md

@@ -1,5 +1,8 @@
 # Changelog
 
+## 16.12.0
+- `RandomBinaryProjectionSearcher` class added
+
 ## 16.11.4
 - `getPimaIndiansDiabetesDataFrame`, `getIrisDataFrame` used
 

README.md

@@ -35,44 +35,48 @@ it in web applications.
 ## The library content
 
 - #### Model selection
-    - [CrossValidator](https://github.com/gyrdym/ml_algo/blob/master/lib/src/model_selection/cross_validator/cross_validator.dart). 
+    - [CrossValidator](https://pub.dev/documentation/ml_algo/latest/ml_algo/CrossValidator-class.html). 
     A factory that creates instances of cross validators. Cross-validation allows researchers to fit different 
     [hyperparameters](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)) of machine learning algorithms 
     assessing prediction quality on different parts of a dataset. 
 
 - #### Classification algorithms
-    - [LogisticRegressor](https://github.com/gyrdym/ml_algo/blob/master/lib/src/classifier/logistic_regressor/logistic_regressor.dart). 
+    - [LogisticRegressor](https://pub.dev/documentation/ml_algo/latest/ml_algo/LogisticRegressor-class.html). 
     A class that performs linear binary classification of data. To use this kind of classifier your data has to be 
     [linearly separable](https://en.wikipedia.org/wiki/Linear_separability).
 
-    - [SoftmaxRegressor](https://github.com/gyrdym/ml_algo/blob/master/lib/src/classifier/softmax_regressor/softmax_regressor.dart). 
+    - [SoftmaxRegressor](https://pub.dev/documentation/ml_algo/latest/ml_algo/SoftmaxRegressor-class.html). 
     A class that performs linear multiclass classification of data. To use this kind of classifier your data has to be 
     [linearly separable](https://en.wikipedia.org/wiki/Linear_separability).
 
-    - [DecisionTreeClassifier](https://github.com/gyrdym/ml_algo/blob/master/lib/src/classifier/decision_tree_classifier/decision_tree_classifier.dart)
+    - [DecisionTreeClassifier](https://pub.dev/documentation/ml_algo/latest/ml_algo/DecisionTreeClassifier-class.html)
     A class that performs classification using decision trees. May work with data with non-linear patterns.
 
-    - [KnnClassifier](https://github.com/gyrdym/ml_algo/blob/master/lib/src/classifier/knn_classifier/knn_classifier.dart)
+    - [KnnClassifier](https://pub.dev/documentation/ml_algo/latest/ml_algo/KnnClassifier-class.html)
     A class that performs classification using `k nearest neighbours algorithm` - it makes predictions based on 
     the first `k` closest observations to the given one.
 
 - #### Regression algorithms
-    - [LinearRegressor](https://github.com/gyrdym/ml_algo/blob/master/lib/src/regressor/linear_regressor/linear_regressor.dart). 
+    - [LinearRegressor](https://pub.dev/documentation/ml_algo/latest/ml_algo/LinearRegressor-class.html). 
     A general class for finding a linear pattern in training data and predicting outcomes as real numbers.
 
-    - [LinearRegressor.lasso](https://github.com/gyrdym/ml_algo/blob/85f1e2f19b946beb2b594a62e0e3c999d1c31608/lib/src/regressor/linear_regressor/linear_regressor.dart#L219)
+    - [LinearRegressor.lasso](https://pub.dev/documentation/ml_algo/latest/ml_algo/LinearRegressor/LinearRegressor.lasso.html)
     Implementation of the linear regression algorithm based on coordinate descent with lasso regularisation
 
-    - [LinearRegressor.SGD](https://github.com/gyrdym/ml_algo/blob/c0ffc71676c1ad14927448fe9bbf984a425ce27a/lib/src/regressor/linear_regressor/linear_regressor.dart#L322)
+    - [LinearRegressor.SGD](https://pub.dev/documentation/ml_algo/latest/ml_algo/LinearRegressor/LinearRegressor.SGD.html)
     Implementation of the linear regression algorithm based on stochastic gradient descent with L2 regularisation
 
-    - [KnnRegressor](https://github.com/gyrdym/ml_algo/blob/master/lib/src/regressor/knn_regressor/knn_regressor.dart)
+    - [KnnRegressor](https://pub.dev/documentation/ml_algo/latest/ml_algo/KnnRegressor-class.html)
     A class that makes predictions for each new observation based on the first `k` closest observations from 
     training data. It may catch non-linear patterns of the data.
 
 - #### Clustering and retrieval algorithms
-    - [KDTree](https://github.com/gyrdym/ml_algo/blob/master/lib/src/retrieval/kd_tree/kd_tree.dart) An algorithm for
+    - [KDTree](https://pub.dev/documentation/ml_algo/latest/kd_tree/KDTree-class.html) An algorithm for
     efficient data retrieval.
+    - **Locality sensitive hashing.** A family of algorithms that randomly partition all reference data points into 
+    different bins, which makes it possible to perform efficient K Nearest Neighbours search, since there is no need 
+    to search for the neighbours through the entire data. The family is represented by the following classes:
+        - [RandomBinaryProjectionSearcher](https://pub.dev/documentation/ml_algo/latest/random_binary_projection_searcher/RandomBinaryProjectionSearcher-class.html)
 
 For more information on the library's API, please visit the [API reference](https://pub.dev/documentation/ml_algo/latest/ml_algo/ml_algo-library.html) 
 

benchmark/random_binary_projection_searcher/searcher_building.dart

@@ -0,0 +1,34 @@
+// 0.03 sec (MacBook Air mid 2017)
+import 'package:benchmark_harness/benchmark_harness.dart';
+import 'package:ml_algo/ml_algo.dart';
+import 'package:ml_dataframe/ml_dataframe.dart';
+import 'package:ml_linalg/matrix.dart';
+
+late DataFrame trainData;
+
+class RandomBinaryProjectionSearcherBuildingBenchmark extends BenchmarkBase {
+  RandomBinaryProjectionSearcherBuildingBenchmark()
+      : super('RandomBinaryProjectionSearcher building benchmark');
+
+  static void main() {
+    RandomBinaryProjectionSearcherBuildingBenchmark().report();
+  }
+
+  @override
+  void run() {
+    RandomBinaryProjectionSearcher(trainData, 4, seed: 10);
+  }
+
+  void tearDown() {}
+}
+
+Future main() async {
+  final points = Matrix.random(1000, 10, seed: 1, min: -5000, max: 5000);
+
+  trainData = DataFrame.fromMatrix(points);
+
+  print(
+      'Data dimension: ${trainData.rows.length}x${trainData.rows.first.length}');
+
+  RandomBinaryProjectionSearcherBuildingBenchmark.main();
+}

benchmark/random_binary_projection_searcher/searcher_querying.dart

@@ -0,0 +1,48 @@
+// 0.04 sec (MacBook Air mid 2017)
+import 'package:benchmark_harness/benchmark_harness.dart';
+import 'package:ml_algo/ml_algo.dart';
+import 'package:ml_algo/src/retrieval/random_binary_projection_searcher/random_binary_projection_searcher_impl.dart';
+import 'package:ml_dataframe/ml_dataframe.dart';
+import 'package:ml_linalg/linalg.dart';
+
+final k = 10;
+final digitCapacity = 10;
+final searchRadius = 3;
+
+late DataFrame trainData;
+late RandomBinaryProjectionSearcher searcher;
+late Vector point;
+
+class RandomBinaryProjectionSearcherQueryingBenchmark extends BenchmarkBase {
+  RandomBinaryProjectionSearcherQueryingBenchmark()
+      : super('RandomBinaryProjectionSearcher querying benchmark');
+
+  static void main() {
+    RandomBinaryProjectionSearcherQueryingBenchmark().report();
+  }
+
+  @override
+  void run() {
+    searcher.query(point, k, searchRadius);
+  }
+
+  void tearDown() {}
+}
+
+Future main() async {
+  final points = Matrix.random(20000, 10, seed: 1, min: -5000, max: 5000);
+
+  trainData = DataFrame.fromMatrix(points);
+  searcher = RandomBinaryProjectionSearcher(trainData, digitCapacity, seed: 10);
+  point = Vector.randomFilled(trainData.rows.first.length,
+      seed: 10, min: -5000, max: 5000);
+
+  print(
+      'Data dimension: ${trainData.rows.length}x${trainData.rows.first.length}');
+  print('Number of neighbours: $k');
+
+  RandomBinaryProjectionSearcherQueryingBenchmark.main();
+
+  print(
+      'Amount of search iterations: ${(searcher as RandomBinaryProjectionSearcherImpl).searchIterationCount}');
+}

e2e/random_binary_projection_searcher/random_binary_projection_searcher_32_v1.json

@@ -0,0 +1,75 @@
+import 'dart:io';
+
+import 'package:ml_algo/ml_algo.dart';
+import 'package:ml_linalg/dtype.dart';
+import 'package:ml_linalg/vector.dart';
+import 'package:test/test.dart';
+
+void main() async {
+  group('RandomBinaryProjectionSearcher', () {
+    test('should restore from a JSON file, dtype=DType.float32', () async {
+      final jsonSource = await File(
+              'e2e/random_binary_projection_searcher/random_binary_projection_searcher_32_v1.json')
+          .readAsString();
+      final searcher = RandomBinaryProjectionSearcher.fromJson(jsonSource);
+      final k = 5;
+      final searchRadius = 3;
+      final neighbours = searcher.query(
+          Vector.fromList(
+              [6.5, 3.01, 4.5, 1.5, 2.3, 22.3, 14.09, 2.9, 22.0, 11.22]),
+          k,
+          searchRadius);
+
+      expect(searcher.columns, [
+        'feature_1',
+        'feature_2',
+        'feature_3',
+        'feature_4',
+        'feature_5',
+        'feature_6',
+        'feature_7',
+        'feature_8',
+        'feature_9',
+        'feature_10'
+      ]);
+      expect(searcher.digitCapacity, 6);
+      expect(searcher.seed, 15);
+      expect(neighbours, hasLength(5));
+      expect(neighbours.toString(),
+          '((Index: 160, Distance: 4646.84383479798), (Index: 562, Distance: 5358.518638579137), (Index: 648, Distance: 5403.194564329513), (Index: 591, Distance: 5522.45033929686), (Index: 938, Distance: 6083.799141983568))');
+    });
+
+    test('should restore from a JSON file, dtype=DType.float64', () async {
+      final jsonSource = await File(
+              'e2e/random_binary_projection_searcher/random_binary_projection_searcher_64_v1.json')
+          .readAsString();
+      final searcher = RandomBinaryProjectionSearcher.fromJson(jsonSource);
+      final k = 5;
+      final searchRadius = 3;
+      final neighbours = searcher.query(
+          Vector.fromList(
+              [6.5, 3.01, 4.5, 1.5, 2.3, 22.3, 14.09, 2.9, 22.0, 11.22],
+              dtype: DType.float64),
+          k,
+          searchRadius);
+
+      expect(searcher.columns, [
+        'feature_1',
+        'feature_2',
+        'feature_3',
+        'feature_4',
+        'feature_5',
+        'feature_6',
+        'feature_7',
+        'feature_8',
+        'feature_9',
+        'feature_10'
+      ]);
+      expect(searcher.digitCapacity, 6);
+      expect(searcher.seed, 15);
+      expect(neighbours, hasLength(5));
+      expect(neighbours.toString(),
+          '((Index: 160, Distance: 4646.843905472263), (Index: 562, Distance: 5358.51853463377), (Index: 648, Distance: 5403.194350179125), (Index: 591, Distance: 5522.45040389316), (Index: 938, Distance: 6083.79924961192))');
+    });
+  });
+}

e2e/random_binary_projection_searcher/random_binary_projection_searcher_64_v1.json

@@ -14,4 +14,5 @@ export 'package:ml_algo/src/regressor/knn_regressor/knn_regressor.dart';
 export 'package:ml_algo/src/regressor/linear_regressor/linear_regressor.dart';
 export 'package:ml_algo/src/retrieval/kd_tree/kd_tree.dart';
 export 'package:ml_algo/src/retrieval/kd_tree/kd_tree_split_strategy.dart';
+export 'package:ml_algo/src/retrieval/random_binary_projection_searcher/random_binary_projection_searcher.dart';
 export 'package:ml_algo/src/tree_trainer/tree_assessor/tree_assessor_type.dart';

e2e/random_binary_projection_searcher/random_binary_projection_searcher_serialization_test.dart

@@ -1,11 +1,7 @@
 import 'package:ml_algo/src/model_selection/split_indices_provider/split_indices_provider.dart';
 
 class LpoIndicesProvider implements SplitIndicesProvider {
-  LpoIndicesProvider([this._p = 2]) {
-    if (_p == 0) {
-      throw UnsupportedError('Value `$_p` for parameter `p` is unsupported');
-    }
-  }
+  LpoIndicesProvider([this._p = 2]);
 
   final int _p;
 

lib/ml_algo.dart

@@ -2,8 +2,8 @@ import 'package:ml_algo/src/common/serializable/serializable.dart';
 import 'package:ml_algo/src/retrieval/kd_tree/helpers/create_kd_tree.dart';
 import 'package:ml_algo/src/retrieval/kd_tree/helpers/create_kd_tree_from_iterable.dart';
 import 'package:ml_algo/src/retrieval/kd_tree/kd_tree_impl.dart';
-import 'package:ml_algo/src/retrieval/kd_tree/kd_tree_neighbour.dart';
 import 'package:ml_algo/src/retrieval/kd_tree/kd_tree_split_strategy.dart';
+import 'package:ml_algo/src/retrieval/neighbour.dart';
 import 'package:ml_dataframe/ml_dataframe.dart';
 import 'package:ml_linalg/distance.dart';
 import 'package:ml_linalg/dtype.dart';
@@ -151,7 +151,7 @@ abstract class KDTree implements Serializable {
   ///   print(neighbours[0].index); // let's say, it outputs `3` which means that the nearest neighbour is kdTree.points[3]
   /// }
   /// ```
-  Iterable<KDTreeNeighbour> query(Vector point, int k,
+  Iterable<Neighbour> query(Vector point, int k,
       [Distance distance = Distance.euclidean]);
 
   /// Returns [k] nearest neighbours for [point], [point] is [Iterable] unlike
@@ -179,6 +179,6 @@ abstract class KDTree implements Serializable {
   ///   print(neighbours[0].index); // let's say, it outputs `3` which means that the nearest neighbour is kdTree.points[3]
   /// }
   /// ```
-  Iterable<KDTreeNeighbour> queryIterable(Iterable<num> point, int k,
+  Iterable<Neighbour> queryIterable(Iterable<num> point, int k,
       [Distance distance = Distance.euclidean]);
 }