Logistic Regression: newton method added (#239)

Author: gyrdym

CHANGELOG.md

@@ -1,8 +1,12 @@
 # Changelog
 
+## 16.17.0
+- LogisticRegressor:
+  - Newton method added
+
 ## 16.16.0
 - LinearRegressor:
-    - Newton method added
+  - Newton method added
 
 ## 16.15.2
 - LinearRegressor, LogisticRegressor, SoftmaxRegressor:

README.md

@@ -51,6 +51,10 @@ it in web applications.
 
         - [LogisticRegressor.BGD](https://pub.dev/documentation/ml_algo/latest/ml_algo/LogisticRegressor/LogisticRegressor.BGD.html). 
     Implementation of the logistic regression algorithm based on batch gradient descent with L2 regularisation. 
+    To use this kind of classifier your data has to be [linearly separable](https://en.wikipedia.org/wiki/Linear_separability).
+        
+        - [LogisticRegressor.newton](https://pub.dev/documentation/ml_algo/latest/ml_algo/LogisticRegressor/LogisticRegressor.newton.html). 
+    Implementation of the logistic regression algorithm based on Newton-Raphson method with L2 regularisation. 
     To use this kind of classifier your data has to be [linearly separable](https://en.wikipedia.org/wiki/Linear_separability).
 
     - [SoftmaxRegressor](https://pub.dev/documentation/ml_algo/latest/ml_algo/SoftmaxRegressor-class.html). 
@@ -78,7 +82,7 @@ it in web applications.
         Implementation of the linear regression algorithm based on batch gradient descent with L2 regularisation
 
         - [LinearRegressor.newton](https://pub.dev/documentation/ml_algo/latest/ml_algo/LinearRegressor/LinearRegressor.newton.html)
-        Implementation of the linear regression algorithm based on Newton method with L2 regularisation
+        Implementation of the linear regression algorithm based on Newton-Raphson method with L2 regularisation
 
     - [KnnRegressor](https://pub.dev/documentation/ml_algo/latest/ml_algo/KnnRegressor-class.html)
     A class that makes predictions for each new observation based on the first `k` closest observations from 
@@ -231,30 +235,18 @@ if the selected hyperparameters are good enough or not:
 
 ```dart
 final createClassifier = (DataFrame samples) =>
-  // BGD stands for "Batch Gradient Descent" that's meaning that the classifier will use the whole dataset on every 
-  // training iteration 
-  LogisticRegressor.BGD(
+  LogisticRegressor(
     samples
     targetColumnName,
-    iterationsLimit: 90,
-    learningRateType: LearningRateType.timeBased,
-    probabilityThreshold: 0.7,
   );
 ```
 
-Let's describe our hyperparameters:
-- `iterationsLimit` - number of learning iterations. The selected optimization algorithm (gradient ascent in our case) will 
-be cyclically run this amount of times
-- `learningRateType` - a strategy for learning rate update. In our case, the learning rate will decrease after every 
-iteration
-- `probabilityThreshold` - lower bound for positive label probability
-
 If we want to evaluate the learning process more thoroughly, we may pass `collectLearningData` argument to the classifier
 constructor:
 
 ```dart
 final createClassifier = (DataFrame samples) =>
-  LogisticRegressor.BGD(
+  LogisticRegressor(
     ...,
     collectLearningData: true,
   );
@@ -265,7 +257,7 @@ the model creation.
 
 ### Evaluate the performance of the model
 
-Assume, we chose really good hyperparameters. In order to validate this hypothesis let's use CrossValidator instance 
+Assume, we chose perfect hyperparameters. In order to validate this hypothesis, let's use CrossValidator instance 
 created before:
 
 ````dart
@@ -287,7 +279,7 @@ print('accuracy on k fold validation: ${accuracy.toStringAsFixed(2)}');
 We can see something like this:
 
 ````
-accuracy on k fold validation: 0.65
+accuracy on k fold validation: 0.75
 ````
 
 Let's assess our hyperparameters on the test set in order to evaluate the model's generalization error:
@@ -374,12 +366,9 @@ void main() async {
   final testData = splits[1];
   final validator = CrossValidator.kFold(validationData, numberOfFolds: 5);
   final createClassifier = (DataFrame samples) =>
-    LogisticRegressor.BGD(
+    LogisticRegressor(
       samples
       targetColumnName,
-      iterationsLimit: 90,
-      learningRateType: LearningRateType.timeBased,
-      probabilityThreshold: 0.7,
     );
   final scores = await validator.evaluate(createClassifier, MetricType.accuracy);
   final accuracy = scores.mean();
@@ -417,12 +406,9 @@ void main() async {
   final testData = splits[1];
   final validator = CrossValidator.kFold(validationData, numberOfFolds: 5);
   final createClassifier = (DataFrame samples) =>
-    LogisticRegressor.BGD(
+    LogisticRegressor(
       samples
       targetColumnName,
-      iterationsLimit: 90,
-      learningRateType: LearningRateType.timeBased,
-      probabilityThreshold: 0.7,
     );
   final scores = await validator.evaluate(createClassifier, MetricType.accuracy);
   final accuracy = scores.mean();

analysis_options.yaml

@@ -11,4 +11,3 @@ analyzer:
     dead_code: error
     duplicate_import: error
     unused_import: error
-    deprecated_member_use_from_same_package: warning

e2e/decision_tree_classifier/iris_tree.svg

@@ -40,7 +40,7 @@ Future main() async {
         'should return adequate score on pima indians diabetes dataset using '
         'accuracy metric, dtype=DType.float64', () async {
       final scores =
-          await evaluateLogisticRegressor(MetricType.accuracy, DType.float32);
+          await evaluateLogisticRegressor(MetricType.accuracy, DType.float64);
 
       expect(scores.mean(), greaterThan(0.5));
     });
@@ -58,7 +58,7 @@ Future main() async {
         'should return adequate score on pima indians diabetes dataset using '
         'precision metric, dtype=DType.float64', () async {
       final scores =
-          await evaluateLogisticRegressor(MetricType.precision, DType.float32);
+          await evaluateLogisticRegressor(MetricType.precision, DType.float64);
 
       expect(scores.mean(), greaterThan(0.5));
     });
@@ -76,7 +76,7 @@ Future main() async {
         'should return adequate score on pima indians diabetes dataset using '
         'recall metric, dtype=DType.float64', () async {
       final scores =
-          await evaluateLogisticRegressor(MetricType.recall, DType.float32);
+          await evaluateLogisticRegressor(MetricType.recall, DType.float64);
 
       expect(scores.mean(), greaterThan(0.5));
     });

e2e/decision_tree_classifier/pima_indians_tree.svg

@@ -0,0 +1,85 @@
+import 'package:ml_algo/ml_algo.dart';
+import 'package:ml_dataframe/ml_dataframe.dart';
+import 'package:test/test.dart';
+
+num evaluateLogisticRegressor(MetricType metric, DType dtype) {
+  final data = getPimaIndiansDiabetesDataFrame().shuffle();
+  final samples = splitData(data, [0.8]);
+  final trainSamples = samples.first;
+  final testSamples = samples.last;
+  final model = LogisticRegressor.newton(
+    trainSamples,
+    'Outcome',
+    dtype: dtype,
+  );
+
+  return model.assess(testSamples, metric);
+}
+
+Future main() async {
+  group('LogisticRegressor.newton', () {
+    test(
+        'should return adequate score on pima indians diabetes dataset using '
+        'accuracy metric, dtype=DType.float32', () {
+      final score =
+          evaluateLogisticRegressor(MetricType.accuracy, DType.float32);
+
+      print('float32, accuracy is $score');
+
+      expect(score, greaterThan(0.7));
+    });
+
+    test(
+        'should return adequate score on pima indians diabetes dataset using '
+        'accuracy metric, dtype=DType.float64', () {
+      final score =
+          evaluateLogisticRegressor(MetricType.accuracy, DType.float64);
+
+      print('float64, accuracy is $score');
+
+      expect(score, greaterThan(0.7));
+    });
+
+    test(
+        'should return adequate score on pima indians diabetes dataset using '
+        'precision metric, dtype=DType.float32', () {
+      final score =
+          evaluateLogisticRegressor(MetricType.precision, DType.float32);
+
+      print('float32, precision is $score');
+
+      expect(score, greaterThan(0.65));
+    });
+
+    test(
+        'should return adequate score on pima indians diabetes dataset using '
+        'precision metric, dtype=DType.float64', () {
+      final score =
+          evaluateLogisticRegressor(MetricType.precision, DType.float64);
+
+      print('float64, precision is $score');
+
+      expect(score, greaterThan(0.65));
+    });
+
+    test(
+        'should return adequate score on pima indians diabetes dataset using '
+        'recall metric, dtype=DType.float32', () {
+      final score = evaluateLogisticRegressor(MetricType.recall, DType.float32);
+
+      print('float32, recall is $score');
+
+      expect(score, greaterThan(0.65));
+    });
+
+    test(
+        'should return adequate score on pima indians diabetes dataset using '
+        'recall metric, dtype=DType.float64', () {
+      final score = evaluateLogisticRegressor(MetricType.recall, DType.float64);
+
+      print('float64, recall is $score');
+
+      expect(score, greaterThan(0.65));
+    });
+  });
+}

e2e/logistic_regressor/logistic_regressor_bgd_test.dart

@@ -41,7 +41,7 @@ Future main() async {
         'should return adequate score on pima indians diabetes dataset using '
         'accuracy metric, dtype=DType.float64', () async {
       final scores =
-          await evaluateLogisticRegressor(MetricType.accuracy, DType.float32);
+          await evaluateLogisticRegressor(MetricType.accuracy, DType.float64);
 
       expect(scores.mean(), greaterThan(0.5));
     });
@@ -59,7 +59,7 @@ Future main() async {
         'should return adequate score on pima indians diabetes dataset using '
         'precision metric, dtype=DType.float64', () async {
       final scores =
-          await evaluateLogisticRegressor(MetricType.precision, DType.float32);
+          await evaluateLogisticRegressor(MetricType.precision, DType.float64);
 
       expect(scores.mean(), greaterThan(0.5));
     });
@@ -77,7 +77,7 @@ Future main() async {
         'should return adequate score on pima indians diabetes dataset using '
         'recall metric, dtype=DType.float64', () async {
       final scores =
-          await evaluateLogisticRegressor(MetricType.recall, DType.float32);
+          await evaluateLogisticRegressor(MetricType.recall, DType.float64);
 
       expect(scores.mean(), greaterThan(0.5));
     });

e2e/logistic_regressor/logistic_regressor_newton_test.dart

@@ -7,10 +7,6 @@ void main() {
   final model = LogisticRegressor(
     splits.first,
     'Outcome',
-    batchSize: splits.first.rows.length,
-    learningRateType: LearningRateType.exponential,
-    decay: 0.7,
-    collectLearningData: true,
   );
 
   print('ACURACY:');
@@ -22,8 +18,10 @@ void main() {
   print('PRECISION:');
   print(model.assess(splits.last, MetricType.precision));
 
-  print('LD: ');
-  print(splits.last['Outcome'].data.take(10));
+  print('Results (first row - actual values, second row - predicted values):');
+  print(splits.last['Outcome'].data
+      .take(10)
+      .map((val) => num.parse(val.toString()).toDouble()));
   print(model
       .predict(splits.last.dropSeries(names: ['Outcome']))
       .series

e2e/logistic_regressor/logistic_regressor_sgd_test.dart

@@ -2,4 +2,5 @@ import 'package:ml_algo/src/linear_optimizer/linear_optimizer_type.dart';
 
 const supportedLinearOptimizerTypes = [
   LinearOptimizerType.gradient,
+  LinearOptimizerType.newton,
 ];