pzmm updates (#87)

* Merge 4075/4083 commits from OMMR

* Adjust pzmm zipping method to output iofile.

* Remove now-redundant pzmm model upload process.

* Edit notebook example to leverage sasctl framework.

* Adjust dmcas writers doc_strings to state acceptance of dataframe, array, or list

* First attempt to change dmcas_fitstat writer to accept more data types.

* added Session.as_swat()

* updated docs

* test as_swat()

* Combine Lift/ROC chart generation into one call.

* Spell transpose correctly and remove unused functions.

* Remove uploadData from init statement.

* Include empty ROC and Lift dmcas JSON files in installation.

* Lowercase all references to pzmm.

* Add empty dmcas lift/roc JSON files.

* Add missing null roc json file.

* Final dev updates to Lift/ROC chart fixes.

* custom get_repository() method to ignore 403 errors.  Improved efficiency for default_repository()

* Adjust score code for invalid column names/values.

* Fix missing numpy package in writeScoreCode

* Fix isidentifer error with improper type.

* Add error for no data provided in calculateFitStat

* Fix dataframe input for generateROCLiftStat()

* Update example notebook to include new features

* Adjust protocol for Session object.

* Fix error in generating ROC/LIFT charts.

* Update example notebook and outputs.

* Fix overwritten model object on repeat runs.

* Adjust text to documentation standards.

* Remove sas server names from example before GitHub release.

* include docs and examples

* additional example

* Merge from Github

* Fix missing argument in pickle and zipping calls.

* refreshed cassettes

* refreshed cassettes

* refreshed cassettes

* temporary disable regression test

* temporary disable regression test

* correctly mock default repo

* fix tests

* fixed cli command for get_repository

* Update changelog.md

* prep for release

* Adjust score code for invalid column names/values.

* Fix missing numpy package in writeScoreCode

* Fix isidentifer error with improper type.

* Merge 4075/4083 commits from OMMR

* Adjust dmcas writers doc_strings to state acceptance of dataframe, array, or list

* First attempt to change dmcas_fitstat writer to accept more data types.

* custom get_repository() method to ignore 403 errors.  Improved efficiency for default_repository()

* Adjust protocol for Session object.

* Update example notebook and outputs.

* Fix overwritten model object on repeat runs.

* Adjust text to documentation standards.

* Adjust score code for invalid column names/values.

* Fix missing numpy package in writeScoreCode

* Fix isidentifer error with improper type.

* Adjust score code for invalid column names/values.

* Fix missing numpy package in writeScoreCode

* Fix isidentifer error with improper type.

* Change writeScoreCode intercept default behavior

* writeScoreCode should handle non-binary targets

* Add software version checking API call and task.

* Added MM version check to writeScoreCode().

* Add DS2 wrapper function to writeScoreCode.py

* Include future API builds for Viya 4.0

* Add API call to move Python score resources.

* Fix doc_string errors.

* Add f(x) to upload/copy pzmm Python/DS2 score code

* Include upload/copy score code in writeScore()

* Fix indentation error

* First run of testing new copy/upload task.

* Fix incorrect role assignment of pickle file.

* Add headers to PUT scoreResources API call.

* Adjust version to differentiate between main/dev versions.

* Handle case of Viya 3.5 & no model provided.

* General fixes to DS2 converter.

* Fix dtype vs string error.

* Call ds2 wrapper endpoint and adjust writeScoreCode.

* Add mojo model file handling to PickelModel.

* Add H2O option to writeFileMetaData.

* Add mojo file to zippable files

* Fix astore vs Python typo

* Add new import file to handle model imports and adjust init file

* Adjust examples and create new simple/h2o notebooks

* Include single row csv for testing

* Major update to writeScoreCode

* Final modification to writeScoreCode & new notebooks

* Edit notebook to remove unused cells

* Fix pathing and .zip errors in H2O example

* Fix trailing .zip

* Include default method for scoring/validating in CAS for Viya 3.5 users.

* Required *.sas files for scoring in CAS/MAS

* Remove reference to uncommitted feature

* Bug fixes for CAS wrapper and documentation

* Delete FleetManagementSimple.ipynb

* Update FleetManagementSimpleH2O.ipynb

* Update H2OModelFleetMaintenance.ipynb

* Adjust to new handling of model importing

* Delete fleet_maintenance.csv

* Update change to CSV files

* Update H2OModelFleetMaintenance.ipynb

* Update README.md

* moved get_software_version() to platform_version()

* removed f strings for compatibility

* bugfix

* remove f strings

* removed duplicate method

Co-authored-by: Scott Lindauer <[email protected]>
Co-authored-by: Scott Lindauer <[email protected]>

Author: jlwalke2

CHANGELOG.md

@@ -2,6 +2,10 @@ Unreleased
 ----------
  **Improvements**
  - `PagedList` handles situations where the server over-estimates the number of items available for paging.
+ - The version of SAS Viya on the server can now be determined using `sasctl.platform_version()`.
+ 
+ **Bugfixes**
+ - Reworked the `model_repository.get_repository()` to prevent HTTP 403 errors that could occur with some Viya environments.
 
  v1.5.5 (2021-03-26)
  -------------------

examples/FleetManagement.ipynb

@@ -20,7 +20,7 @@
     "\n",
     "This notebook provides an example of how to build and train a Python model and then import the model into SAS Model Manager using the fleet maintenance data set. Lines of code that must be modified by the user, such as directory paths are noted with the comment \"_Changes required by user._\".\n",
     "\n",
-    "_**Note:** If you download only this notebook and not the rest of the repository, you must also download the fleet_maintenance.csv from the data folder in the examples directory. These files are used when executing this notebook example._\n",
+    "_**Note:** If you download only this notebook and not the rest of the repository, you must also download the fleet maintenance CSV file from the data folder in the examples directory. These files are used when executing this notebook example._\n",
     "\n",
     "Here are the steps shown in this notebook:\n",
     "\n",
@@ -989,20 +989,16 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": 15,
    "metadata": {
     "Collapsed": "false"
    },
    "outputs": [],
    "source": [
-    "S = pzmm.ScoreCode()\n",
+    "I = pzmm.ImportModel()\n",
     "for (prefix, path) in zip(modelPrefix, zipFolder):\n",
-    "    S.writeScoreCode(x, y, prefix,\n",
-    "                     '{}.predict({})', prefix + '.pickle',\n",
-    "                     pyPath=path)\n",
-    "    zipIOFile = pzmm.ZipModel.zipFiles(_, path, prefix)\n",
     "    with sess:\n",
-    "        modelReponse = modelRepo.import_model_from_zip(prefix, 'Fleet Management Report', zipIOFile)"
+    "        I.pzmmImportModel(path, prefix, 'Fleet Management Report', x, y, '{}.predict({})')\n"
    ]
   },
   {
@@ -1030,6 +1026,53 @@
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
    "version": "3.8.3"
+  },
+  "latex_envs": {
+   "LaTeX_envs_menu_present": true,
+   "autoclose": true,
+   "autocomplete": true,
+   "bibliofile": "biblio.bib",
+   "cite_by": "apalike",
+   "current_citInitial": 1,
+   "eqLabelWithNumbers": true,
+   "eqNumInitial": 1,
+   "hotkeys": {
+    "equation": "Ctrl-E",
+    "itemize": "Ctrl-I"
+   },
+   "labels_anchors": false,
+   "latex_user_defs": false,
+   "report_style_numbering": false,
+   "user_envs_cfg": false
+  },
+  "varInspector": {
+   "cols": {
+    "lenName": 16,
+    "lenType": 16,
+    "lenVar": 40
+   },
+   "kernels_config": {
+    "python": {
+     "delete_cmd_postfix": "",
+     "delete_cmd_prefix": "del ",
+     "library": "var_list.py",
+     "varRefreshCmd": "print(var_dic_list())"
+    },
+    "r": {
+     "delete_cmd_postfix": ") ",
+     "delete_cmd_prefix": "rm(",
+     "library": "var_list.r",
+     "varRefreshCmd": "cat(var_dic_list()) "
+    }
+   },
+   "types_to_exclude": [
+    "module",
+    "function",
+    "builtin_function_or_method",
+    "instance",
+    "_Feature"
+   ],
+   "window_display": false
   }
  },
  "nbformat": 4,

examples/H2OModelFleetMaintenance.ipynb

@@ -0,0 +1,49 @@
+
+
+import math
+import pickle
+import pandas as pd
+import numpy as np
+
+
+global _thisModelFit
+
+with open('/models/resources/viya/fc809ba9-11ec-45ad-bc49-ce2cd2a02fd4/DecisionTreeClassifier.pickle', 'rb') as _pFile:
+    _thisModelFit = pickle.load(_pFile)
+
+def scoreDecisionTreeClassifier(Speed_sensor, Vibration, Engine_Load, Coolant_Temp, Intake_Pressure, Engine_RPM, Speed_OBD, Intake_Air, Flow_Rate, Throttle_Pos, Voltage, Ambient, Accel, Engine_Oil_Temp, Speed_GPS, GPS_Longitude, GPS_Latitude, GPS_Bearing, GPS_Altitude, Turbo_Boost, Trip_Distance, Litres_Per_km, Accel_Ssor_Total, CO2, Trip_Time, CO_emission, HC_emission, PM_emission, NOx_emission, CO2_emission, Fuel_level, Oil_life, Vibration_alert, VibrationAlert_Total, Vibration_Recent, Turbo_alert, Emission_alert, Fog_control, Engine_control):
+    "Output: EM_EVENTPROBABILITY, EM_CLASSIFICATION"
+
+    try:
+        _thisModelFit
+    except NameError:
+
+        with open('/models/resources/viya/fc809ba9-11ec-45ad-bc49-ce2cd2a02fd4/DecisionTreeClassifier.pickle', 'rb') as _pFile:
+            _thisModelFit = pickle.load(_pFile)
+
+    try:
+        inputArray = pd.DataFrame([[Speed_sensor, Vibration, Engine_Load, Coolant_Temp, Intake_Pressure, Engine_RPM, Speed_OBD, Intake_Air, Flow_Rate, Throttle_Pos, Voltage, Ambient, Accel, Engine_Oil_Temp, Speed_GPS, GPS_Longitude, GPS_Latitude, GPS_Bearing, GPS_Altitude, Turbo_Boost, Trip_Distance, Litres_Per_km, Accel_Ssor_Total, CO2, Trip_Time, CO_emission, HC_emission, PM_emission, NOx_emission, CO2_emission, Fuel_level, Oil_life, Vibration_alert, VibrationAlert_Total, Vibration_Recent, Turbo_alert, Emission_alert, Fog_control, Engine_control]],
+                                  columns=['Speed_sensor', 'Vibration', 'Engine_Load', 'Coolant_Temp', 'Intake_Pressure', 'Engine_RPM', 'Speed_OBD', 'Intake_Air', 'Flow_Rate', 'Throttle_Pos', 'Voltage', 'Ambient', 'Accel', 'Engine_Oil_Temp', 'Speed_GPS', 'GPS_Longitude', 'GPS_Latitude', 'GPS_Bearing', 'GPS_Altitude', 'Turbo_Boost', 'Trip_Distance', 'Litres_Per_km', 'Accel_Ssor_Total', 'CO2', 'Trip_Time', 'CO_emission', 'HC_emission', 'PM_emission', 'NOx_emission', 'CO2_emission', 'Fuel_level', 'Oil_life', 'Vibration_alert', 'VibrationAlert_Total', 'Vibration_Recent', 'Turbo_alert', 'Emission_alert', 'Fog_control', 'Engine_control'],
+                                  dtype=float)
+        prediction = _thisModelFit.predict(inputArray)
+    except ValueError:
+    # For models requiring or including an intercept value, a 'const' column is required
+    # For example, many statsmodels models include an intercept value that must be included for the model prediction
+        inputArray = pd.DataFrame([[1.0, Speed_sensor, Vibration, Engine_Load, Coolant_Temp, Intake_Pressure, Engine_RPM, Speed_OBD, Intake_Air, Flow_Rate, Throttle_Pos, Voltage, Ambient, Accel, Engine_Oil_Temp, Speed_GPS, GPS_Longitude, GPS_Latitude, GPS_Bearing, GPS_Altitude, Turbo_Boost, Trip_Distance, Litres_Per_km, Accel_Ssor_Total, CO2, Trip_Time, CO_emission, HC_emission, PM_emission, NOx_emission, CO2_emission, Fuel_level, Oil_life, Vibration_alert, VibrationAlert_Total, Vibration_Recent, Turbo_alert, Emission_alert, Fog_control, Engine_control]],
+                                columns=['const', 'Speed_sensor', 'Vibration', 'Engine_Load', 'Coolant_Temp', 'Intake_Pressure', 'Engine_RPM', 'Speed_OBD', 'Intake_Air', 'Flow_Rate', 'Throttle_Pos', 'Voltage', 'Ambient', 'Accel', 'Engine_Oil_Temp', 'Speed_GPS', 'GPS_Longitude', 'GPS_Latitude', 'GPS_Bearing', 'GPS_Altitude', 'Turbo_Boost', 'Trip_Distance', 'Litres_Per_km', 'Accel_Ssor_Total', 'CO2', 'Trip_Time', 'CO_emission', 'HC_emission', 'PM_emission', 'NOx_emission', 'CO2_emission', 'Fuel_level', 'Oil_life', 'Vibration_alert', 'VibrationAlert_Total', 'Vibration_Recent', 'Turbo_alert', 'Emission_alert', 'Fog_control', 'Engine_control'],
+                                dtype=float)
+        prediction = _thisModelFit.predict(inputArray)
+
+    try:
+        EM_EVENTPROBABILITY = float(prediction)
+    except TypeError:
+    # If the model expects non-binary responses, a TypeError will be raised.
+    # The except block shifts the prediction to accept a non-binary response.
+        EM_EVENTPROBABILITY = float(prediction[:,1])
+
+    if (EM_EVENTPROBABILITY >= 0.2269170579029734):
+        EM_CLASSIFICATION = '1'
+    else:
+        EM_CLASSIFICATION = '0' 
+
+    return(EM_EVENTPROBABILITY, EM_CLASSIFICATION)

examples/data/FleetMaintenanceModels/DecisionTreeClassifierSimple/DecisionTreeClassifier.pickle

@@ -0,0 +1,16 @@
+{
+    "name": "DecisionTreeClassifier",
+    "description": " : Maintenance_flag = Speed_sensor + Vibration + Engine_Load + Coolant_Temp + Intake_Pressure + Engine_RPM + Speed_OBD + Intake_Air + Flow_Rate + Throttle_Pos + Voltage + Ambient + Accel + Engine_Oil_Temp + Speed_GPS + GPS_Longitude + GPS_Latitude + GPS_Bearing + GPS_Altitude + Turbo_Boost + Trip_Distance + Litres_Per_km + Accel_Ssor_Total + CO2 + Trip_Time + CO_emission + HC_emission + PM_emission + NOx_emission + CO2_emission + Fuel_level + Oil_life + Vibration_alert + VibrationAlert_Total + Vibration_Recent + Turbo_alert + Emission_alert + Fog_control + Engine_control",
+    "function": "classification",
+    "scoreCodeType": "python",
+    "trainTable": " ",
+    "trainCodeType": "Python",
+    "algorithm": "",
+    "targetVariable": "Maintenance_flag",
+    "targetEvent": 1,
+    "targetLevel": "BINARY",
+    "eventProbVar": "EM_EVENTPROBABILITY",
+    "modeler": "sasdemo",
+    "tool": "Python 3",
+    "toolVersion": "3.8.3"
+}