[53] Update to sklearn 0.18.1

Currently, spark-sklearn gives deprecation warnings when used with sklearn version .18 because several classes in grid_search and cross_validation were refactored into a new module called model_selection. The changes here make spark-sklearn compatible with the changes introduced in sklearn .18.

The most critical changes reflected in the new version of sklearn is that:

sklearn.model_selection.GridSearchCV now has:

cv_results_ : dict of numpy (masked) ndarrays - A dict with keys as column headers and values as columns, that can be imported into a pandas DataFrame.
best_estimator_ : estimator - Estimator that was chosen by the search, i.e. estimator which gave highest score (or smallest loss if specified) on the left out data. Not available if refit=False.
best_score_ : float - Score of best_estimator on the left out data.
best_params_ : dict - Parameter setting that gave the best results on the hold out data.
best_index_ : int - The index (of the cv_results_ arrays) which corresponds to the best candidate parameter setting.
scorer_ : function - Scorer function used on the held out data to choose the best parameters for the model.
n_splits_ : int - The number of cross-validation splits (folds/iterations).
While spark-sklearn.GridSearchCV has:

grid_scores_ : list of named tuples
best_estimator_ : estimator - Estimator that was chosen by the search, i.e. estimator which gave highest score (or smallest loss if specified) on the left out data. Not available if refit=False.
best_score_ : float - Score of best_estimator on the left out data.
best_params_ : dict - Parameter setting that gave the best results on the hold out data.
scorer_ : function - Scorer function used on the held out data to choose the best parameters for the model.
The biggest is that sklearn added the more comprehensive cv_results_ which adds data that the formerly compatible grid_scores_ is lacking.

Note: This version of spark-sklearn is not compatible with sklearn <= .17.

Author: jkbradley

.travis.yml

@@ -32,7 +32,7 @@ install:
   - conda info -a
 
   # Replace dep1 dep2 ... with your dependencies
-  - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION scikit-learn==0.17.0 nose=1.3.7 pandas=0.18
+  - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION scikit-learn==0.18.1 nose=1.3.7 pandas=0.18
   - source activate test-environment
 
 script:

python/README.md

@@ -20,7 +20,8 @@ This package is released under the Apache 2.0 license. See the LICENSE file.
 ## Installation
 
 This package has the following requirements:
- - a recent version of scikit-learn. Version 0.17 has been tested, older versions may work too.
+This package has the following requirements:
+ - Sklearn version >= 0.18.1
  - Spark >= 2.1.1 Spark may be downloaded from the
  [Spark official website](http://spark.apache.org/). In order to use spark-sklearn, you need to use the pyspark interpreter or another Spark-compliant python interpreter. See the [Spark guide](https://spark.apache.org/docs/latest/programming-guide.html#overview) for more details.
  - [nose](https://nose.readthedocs.org) (testing dependency only)

python/requirements.txt

@@ -1,2 +1,2 @@
 # This file should list any python package dependencies.
-scikit-learn==0.17.1
+scikit-learn==0.18.1

python/setup.py

@@ -19,7 +19,7 @@
     "Programming Language :: Python",
     "Topic :: Scientific/Engineering",
 ]
-INSTALL_REQUIRES = ["scikit-learn >= 0.17"]
+INSTALL_REQUIRES = ["scikit-learn >= 0.18.1"]
 
 # Project root
 ROOT = os.path.abspath(os.getcwd() + "/")