Skip to content

Commit 491fa01

Browse files
cmccarthy1DianeoddmorgankxConor McCarthyConor McCarthy
authored
AutoML Refactor Version 0.3.0 (#14)
* Version 0.3.0 update (#13) * addition of latex support and torch functionality * merged nlp into new version * update to import and checking functionality * removal of hack for save paths * commenting to init,no longer defining the .automml.p namespace, functions won't be callable unless they're all available anyway * removal of unnecessary type check, more readable choice of first element, checknlp -> validnlp, or not and for check * space between separate columns, rename of util to prep. ... * refactoring of nlp preprocessing execution * splitting of preprocessing functions into sub folders, models folder now splits into sections * null and constant drop, simplification of percentage calculation in stop tab function * First pass commit at automl code structure with new graphing mechanism * full graph in new format which can run 'basic' .automl.run * Addition of stub files for AutoML graph testing * update to travis test code * first pass at data ingestion, configuration creation and data checking * addition of save path to config, additional checking for NLP * update to config retrieval to support flat files, coinciding refactor of function * renaming of nlp checks to be clearer * removal of overwritten date/time and update to structure/commenting * change to camelCase, addition of image graphs, change to structure * First pass update to include new coding standard definitions * addition of a common location for general use utilities * removal of unnecessary hidden files * Addition of tests for target data functionality * variable -> variant * tests for process based retrieval of feature data * addition of appropriate tests for the dataCheck node * minor updates * update to graph, inclusion of label encodeing symbol mapping to graph both code and images * addition of tests for remaining function in dataCheck node * review of targetData, featureData and dataCheck * added labelEncode functionality and corresponding tests * Initial addition of now renamed featureDescription node, update to graph images * change from modification to description in node naming * removal of unneeded param to dataDescription function, update to tests to cover all expected behaviour * update to automl graph to use new label encode function from the toolkit * added node for modelGeneration. Added customization folder for models, scoring funcs etc * addition of funcs.q * updated comments from review of code * Minor improvements to code * changes to keras functions to make it more adaptable for the addition of other models * windows fix for updateConfig (#11) * windows fix for updateConfig - no longer overwrites dir * code tidy up * Addition of dataPreprocessing node (#12) * addition of dataPreprocessing node * cleaned up commenting * updated review changes * minor updates to dataPreprocessing functionality, models definitions updated for keras.q Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> * New commenting style required for featureDescription node (#19) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * new commenting style * Feature creation node (#16) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * addition of dataPreprocessing node * cleaned up commenting * updated review changes * minor updates to dataPreprocessing functionality, models definitions updated for keras.q * added featurecreation functionality * cleaned up nlp functions * cleaned up code * updated graph for feat create model, added test print statements. Added travis/appveyor PYTHONHASHSEED * updated appveyor build scripts to install embedpy via conda * pythonhashseed env * code review and test changes * removal of old testing data * added tests and error trap for NLP and pulled down review * changed NLP tests * updated NLP tests to use spacy 2.3.2 * updated code in line with comments and added tests for ml.df2tab addition Co-authored-by: Deanna Morgan <dmorgan1@kx.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> * Selectmodels node (#18) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * added selectModels node * included funcs.q * test fixes * updated any comments in PR. Pulled down latest version Co-authored-by: Deanna Morgan <dmorgan1@kx.com> * addition of predictParams node (#21) * addition of predictParams node * graph updates Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> * Created pathConstruct node (#23) * addition of paramConsolidate node * created pathConstruct node * Automl graph tts (#15) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * addition of featureSignificance node * addition of trainTestSplit node * addition of featureSignificance node * sigFeat fixes * sigFeat error trapping * sigFeat tests * change of train test split output type * train test split tests * correction to featSig tests * correction to featSig tests * correction to featSig tests * test updates * test updates * correlated columns * review of tts * review of sigfeat * correction to sigFeats functions to include one of correlation columns * addition of tests for funcs.q * addition of q/python func check * review of tts, moved qpyFuncSearch to dataCheck * reviewed featSig tests * utils moved to funcs.q for TTS + sz check added * utils moved to funcs.q for TTS + sz check added * removed pythonTTS.p - already in dataCheck * PR changes Co-authored-by: Dianeod <dodonoghue@kx.com> * Addition of saveGraph node (#24) * addition of saveGraph node * addition of saveGraph node * addition of extra plots * removed folders created in tests * review of comments made * updated Graph * moved plt to utils, changed marker size Co-authored-by: cmccarthy1 <38653604+cmccarthy1@users.noreply.github.com> * Graph runmodels (#17) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * addition of featureSignificance node * addition of trainTestSplit node * addition of featureSignificance node * sigFeat fixes * sigFeat error trapping * sigFeat tests * change of train test split output type * train test split tests * correction to featSig tests * correction to featSig tests * correction to featSig tests * test updates * test updates * correlated columns * review of tts * review of sigfeat * updated * addition of runmodels node * updated graph * runModels review * added number of reps for gs/xv * updated dataCheck test * updated comments made in PR * addition of information for metadata * resolved all comments Co-authored-by: Deanna Morgan <dmorgan1@kx.com> Co-authored-by: dmorgankx <44678213+dmorgankx@users.noreply.github.com> * Automl graph preproc params (#26) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * Update Automl_Graph.drawio * graph, test and code format updates * test print statements and updated graph * Update Automl_Graph.drawio * graph updates * Connection abd workflow clarity for graph images Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> * Addition of saveMeta node (#25) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * new commenting style * addition of saveMeta node * addition of saveopt check and tests * node review - moved mdlMeta to funcs, removed repeated code * added print statements. Removed pathDict created in pathConstruct * updated modelMeta lib * Addition of tests to check paths/metadata is created Co-authored-by: Deanna Morgan <dmorgan1@kx.com> * OptimizeModels Node (#20) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * new commenting style * added optimization node * update optimization node * updated graph * Update to include confusion matrix and impact dictionary * added regression calculation * node review Co-authored-by: Deanna Morgan <dmorgan1@kx.com> * Fixed any bugs found (#27) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * new commenting style * added optimization node * update optimization node * updated graph * Update to include confusion matrix and impact dictionary * added regression calculation * node review * fixed any bugs found, nor runs through for all nodes Co-authored-by: Deanna Morgan <dmorgan1@kx.com> * Moved testing functions to separate file (#29) * moved passing/failing test to seperate file * added test/utils.q * Addition of saveModel node (#28) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * new commenting style * addition of saveModels node * addition of savemodels node * addition of saveModels node * clearned up if statement * addition of saveModels node * added tests for NLP Co-authored-by: Deanna Morgan <dmorgan1@kx.com> * Updating any bugs so that .`automl.run` works (#30) * removed duplicated function * fixed any errors to make sure automl.run runs through * updated travis to include spacy english model and keras * Automl graph savereport (#31) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * new commenting style * addition of saveModels node * addition of savemodels node * addition of saveModels node * clearned up if statement * minor code changes * addition of fpdf report gen * addition of fpdf report gen * updated saveGraph to run latex * report tests * report tests * updated tests * updated image size * latex formatting * Updated latex checking and change to code organization * pdflatex naming typo * Fix to force absolute location of generated reports * update to reportlab generation, new page removes custom font for headers Co-authored-by: Dianeod <dodonoghue@kx.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> Co-authored-by: Conor McCarthy <cmccarthy1@kx.com> * removed duplicated function (#33) * move hyperparams to json files (#34) * removed duplicated function * move hyperparams to json files * Update to some descriptions of functions and indenting of json Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> * Add functionality to add custom save path (#35) * removed duplicated function * added functionality to add custom model save path * move hyperparams to json files (#34) * removed duplicated function * move hyperparams to json files * Update to some descriptions of functions and indenting of json Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> * Update to be more strongly typed Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> * Introduction of command line interface api for automl (#36) * Initial pass at json driven command line interfacce * Major update to command line interface to support new input naming and allow first pass at fire and forget * update to allow data retrieval via ipc/csv in command line case * update to json format and command line input structure * addition of code commenting for new command line version * Final change to facilitate appropriate model naming conventioj * typo fix * Review of code (#37) * removed duplicated function * review of code * refactor default layout * Reintroduction of prediction mechanism for automl (#38) * first pass at addition of prediction functionality * Working pass at retrieval of models from disk * update to remove multiple paths to generate predict function * revert to pre cli_testing merge * Minor updates to clean up NLP and correctly retrieve saved model. Update to feature creation for FRESH to support tabular input * minor fixes to issues with retrieving named models and using the correct save option name * Update tests (#40) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * new commenting style * addition of saveModels node * addition of savemodels node * addition of saveModels node * clearned up if statement * minor code changes * update to tests to be in line with new dictionary input structure * updates in line with requirements for tests with dataPreprocessing node * featureExtractionType across the board change * Update to tests for featureCreation node * update to featureExtractionType name * update to featureSignificance node tests to account for new config structure * update to configuration retrieval to ensure full config retrieved * renaming of config parameters for model optimization and update to feature extraction naming for preprocParams node * update to runModels test config * updates to configurations for train test split node * Fix to bug in data split function and renaming of configuration in testing in line with new functionality * update to saveMeta testing to align with revised structure for prediction functionality * update to saveoption and feature extraction naming in line with new config for saveModels node * Update to configuration for testing of saveReport node * removal of old config definition * Fix to bug introduced with change to hyperparameter function retrieval, update to configuration keys * path error fix and model meta check * reintroduction of test utilities needed for passing/failing test logic * Review of updateTests branch (#42) * removed duplicated function * review of code * refactor default layout * review of testing code * Reintroduction of travis testing (#43) * initial update to reintroduce tests * reintroduction of tensorflow install requirement * Change to number of features and minor change to model paths * update to FRESH data to align with correct representation Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> Co-authored-by: Deanna Morgan <dmorgan1@kx.com> Co-authored-by: Dianeod <dodonoghue@kx.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> Co-authored-by: Dianeod <40861871+Dianeod@users.noreply.github.com> * Automl scoringmodels (#46) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * new commenting style * addition of saveModels node * addition of savemodels node * addition of saveModels node * clearned up if statement * minor code changes * update to tests to be in line with new dictionary input structure * updates in line with requirements for tests with dataPreprocessing node * featureExtractionType across the board change * Update to tests for featureCreation node * update to featureExtractionType name * update to featureSignificance node tests to account for new config structure * update to configuration retrieval to ensure full config retrieved * renaming of config parameters for model optimization and update to feature extraction naming for preprocParams node * update to runModels test config * updates to configurations for train test split node * Fix to bug in data split function and renaming of configuration in testing in line with new functionality * update to saveMeta testing to align with revised structure for prediction functionality * update to saveoption and feature extraction naming in line with new config for saveModels node * Update to configuration for testing of saveReport node * removal of old config definition * Fix to bug introduced with change to hyperparameter function retrieval, update to configuration keys * move to json structure for models * model text files not needed * json additions for models and scoring * scoring json file * apply flag/boolean seed/scoring fixes/docs link * test fixes * test fixes Co-authored-by: Dianeod <dodonoghue@kx.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> * review of predict node (#44) * removed duplicated function * review of code * refactor default layout * review of predict function * addition of warning if model comes from an unsupported library Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> * reintroduction of load for test utils in saveModels * Adding printing Functionality (#45) * removed duplicated function * added functionality to add custom model save path * Update to be more strongly typed * added print statements * removed file * moved remaining print statements to new format. Added print python warning option * cleaned up printing dict * fixed naming convention * updated naming convention. Adding additional logging parameter * moved api functionality to utils * Updates to clean up ordering of printing, allow logging directories/files to be modified in json definitions, update graph as required and add print for graph file locations Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> Co-authored-by: Deanna Morgan <dmorgan1@kx.com> Co-authored-by: Dianeod <dodonoghue@kx.com> Co-authored-by: Dianeod <40861871+Dianeod@users.noreply.github.com> Co-authored-by: dmorgankx <44678213+dmorgankx@users.noreply.github.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> Co-authored-by: Dianeod <40861871+Dianeod@users.noreply.github.com> Co-authored-by: Deanna Morgan <dmorgan1@kx.com> Co-authored-by: Dianeod <dodonoghue@kx.com> Co-authored-by: dmorgankx <44678213+dmorgankx@users.noreply.github.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> * Graph warning (#47) * removed duplicated function * Initial pass at json driven command line interfacce * Major update to command line interface to support new input naming and allow first pass at fire and forget * update to allow data retrieval via ipc/csv in command line case * update to json format and command line input structure * addition of code commenting for new command line version * Final change to facilitate appropriate model naming conventioj * typo fix * review of code * refactor default layout * first pass at addition of prediction functionality * Review of code (#37) * removed duplicated function * review of code * refactor default layout * Working pass at retrieval of models from disk * update to remove multiple paths to generate predict function * revert to pre cli_testing merge * Minor updates to clean up NLP and correctly retrieve saved model. Update to feature creation for FRESH to support tabular input * minor fixes to issues with retrieving named models and using the correct save option name * review of predict function * Update tests (#40) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * new commenting style * addition of saveModels node * addition of savemodels node * addition of saveModels node * clearned up if statement * minor code changes * update to tests to be in line with new dictionary input structure * updates in line with requirements for tests with dataPreprocessing node * featureExtractionType across the board change * Update to tests for featureCreation node * update to featureExtractionType name * update to featureSignificance node tests to account for new config structure * update to configuration retrieval to ensure full config retrieved * renaming of config parameters for model optimization and update to feature extraction naming for preprocParams node * update to runModels test config * updates to configurations for train test split node * Fix to bug in data split function and renaming of configuration in testing in line with new functionality * update to saveMeta testing to align with revised structure for prediction functionality * update to saveoption and feature extraction naming in line with new config for saveModels node * Update to configuration for testing of saveReport node * removal of old config definition * Fix to bug introduced with change to hyperparameter function retrieval, update to configuration keys * path error fix and model meta check * reintroduction of test utilities needed for passing/failing test logic * Review of updateTests branch (#42) * removed duplicated function * review of code * refactor default layout * review of testing code * Reintroduction of travis testing (#43) * initial update to reintroduce tests * reintroduction of tensorflow install requirement * Change to number of features and minor change to model paths * update to FRESH data to align with correct representation Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> Co-authored-by: Deanna Morgan <dmorgan1@kx.com> Co-authored-by: Dianeod <dodonoghue@kx.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> Co-authored-by: Dianeod <40861871+Dianeod@users.noreply.github.com> * add capability to ignore warnings/error statements * Automl scoringmodels (#46) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * new commenting style * addition of saveModels node * addition of savemodels node * addition of saveModels node * clearned up if statement * minor code changes * update to tests to be in line with new dictionary input structure * updates in line with requirements for tests with dataPreprocessing node * featureExtractionType across the board change * Update to tests for featureCreation node * update to featureExtractionType name * update to featureSignificance node tests to account for new config structure * update to configuration retrieval to ensure full config retrieved * renaming of config parameters for model optimization and update to feature extraction naming for preprocParams node * update to runModels test config * updates to configurations for train test split node * Fix to bug in data split function and renaming of configuration in testing in line with new functionality * update to saveMeta testing to align with revised structure for prediction functionality * update to saveoption and feature extraction naming in line with new config for saveModels node * Update to configuration for testing of saveReport node * removal of old config definition * Fix to bug introduced with change to hyperparameter function retrieval, update to configuration keys * move to json structure for models * model text files not needed * json additions for models and scoring * scoring json file * apply flag/boolean seed/scoring fixes/docs link * test fixes * test fixes Co-authored-by: Dianeod <dodonoghue@kx.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> * review of predict node (#44) * removed duplicated function * review of code * refactor default layout * review of predict function * addition of warning if model comes from an unsupported library Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> * cleaned up code * added more verbose warnings. Changed the location of removal of previous savePaths * Update to reverse ordering of warning levels, fixes to deletion logic for tests, cfg->config Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> Co-authored-by: cmccarthy1 <38653604+cmccarthy1@users.noreply.github.com> Co-authored-by: Deanna Morgan <dmorgan1@kx.com> Co-authored-by: dmorgankx <44678213+dmorgankx@users.noreply.github.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> * addition of fit/predict functionality and fix to retrieval of models based on name * Revert "addition of fit/predict functionality and fix to retrieval of models based on name" This reverts commit 0e70c17. * Addition of Theano capability (#48) * removed duplicated function * added capability for adding a Theano model * Update to Theano model support to remove models and allow run to continue if theano not installed * added theano model check. Cleaned up printWarnings dict. Fixed print to screen check if saveOpt is 0 Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> * Reintroduction of fit-predict tests and fix to named model retrieval (#49) * addition of fit-predict tests and fix to retrieval of named models * Graph testing upd (#50) * removed duplicated function * added print statements, included all test files in a bat file * Changed txt file to bat file in travis Co-authored-by: cmccarthy1 <38653604+cmccarthy1@users.noreply.github.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> Co-authored-by: Dianeod <40861871+Dianeod@users.noreply.github.com> * Overall code review (#51) * windows fix for updateConfig - no longer overwrites dir * code tidy up * variable declared as global broke tests - changed to local * new commenting style * addition of saveModels node * addition of savemodels node * addition of saveModels node * clearned up if statement * minor code changes * code review * code review * code review * code review * code review * code review * code review * code review * fixes to dataCheck tests * test updates for windows * test fixes * conflict fixes * review of changes to overall codebase * minor change to selectModels test Co-authored-by: Dianeod <dodonoghue@kx.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> * Graph log warning tests (#52) * removed duplicated function * added logging tests * addition of warning/theano/torch tests * fix for appveyor and travis tests * fixed appveyor build * removed swp file * updated ignorewarnings print statement * minor updates, torch change required for non gpu install torch Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> * Addition of retrieval logic to get nearest model based on start date (#53) * removed duplicated function * Initial pass at retrieval of closest model * added capability for adding a Theano model * Update to Theano model support to remove models and allow run to continue if theano not installed * Addition of model deletion functionality * removal of code duplication * Graph delete models (#55) * added logging tests * addition of warning/theano/torch tests * fix for appveyor and travis tests * fixed appveyor build * removed swp file * updated ignorewarnings print statement * minor updates, torch change required for non gpu install torch * review of code * fix delete models * fix for getModels using time Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> * addition of command line interface test and addition of test flag for running cli automl (#54) Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> * Graph fix misc (#57) * fixed logging andw warning tests;Updated README; Check for wrong input * revert changed to TF print * changed date/time to original format * cleaned up code * cleaned up code * Graph tests (#58) * reduced tests for appveyor timeout * reduced number of iterations for Theano Co-authored-by: Dianeod <dodonoghue@kx.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> Co-authored-by: Dianeod <40861871+Dianeod@users.noreply.github.com> Co-authored-by: Dianeod <dodonoghue@kx.com> Co-authored-by: Dianeod <40861871+Dianeod@users.noreply.github.com> Co-authored-by: dmorgankx <44678213+dmorgankx@users.noreply.github.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> Co-authored-by: Deanna Morgan <dmorgan1@kx.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local> * Update requirements.txt Co-authored-by: Dianeod <dodonoghue@kx.com> Co-authored-by: Dianeod <40861871+Dianeod@users.noreply.github.com> Co-authored-by: dmorgankx <44678213+dmorgankx@users.noreply.github.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.mynet> Co-authored-by: Deanna Morgan <dmorgan1@kx.com> Co-authored-by: Conor McCarthy <conormccarthy@brainpool1.local>
1 parent d8e4bdf commit 491fa01

File tree

186 files changed

+8351
-2196
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

186 files changed

+8351
-2196
lines changed

.travis.yml

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,12 @@ install:
2828
- if [[ "x$QLIC_KC" != "x" ]]; then
2929
echo -n $QLIC_KC |base64 --decode > q/kc.lic;
3030
pip -q install -r requirements.txt;
31+
pip install tensorflow;
32+
pip install keras;
3133
pip install torch;
34+
pip install theano;
3235
pip install gensim;
3336
pip install sobol-seq;
34-
3537
fi
3638
beforescript:
3739
- IMPLEMENTATION=$(if [[ "x$TRAVIS_TAG" == "x" ]]; then echo $TRAVIS_BRANCH-$TRAVIS_COMMIT; else echo $TRAVIS_TAG; fi;)
@@ -47,13 +49,12 @@ script:
4749
- echo "Packaged as automl_$TRAVIS_OS_NAME-$TRAVIS_BRANCH.zip"
4850
- if [[ "x$QLIC_KC" != "x" ]]; then
4951
curl -fsSL -o test.q https://github.com/KxSystems/embedpy/raw/master/test.q;
50-
q test.q -q;
52+
PYTHONHASHSEED=0 bash code/tests/testFiles.bat
5153

5254
else
5355
echo No kdb+, no tests;
5456
fi
5557

56-
5758
deploy:
5859
provider: releases
5960
api_key: "$GITHUB_APIKEY"

README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ At present the machine learning frameworks supported for this are based on:
1818

1919
1. One-to-one feature to target non time-series
2020
2. FRESH based feature extraction and model production
21+
3. NLP-based feature creation and word2vec transformation.
2122

2223
The problems which can be solved by this framework will be expanded over time as will the available functionality.
2324

@@ -65,6 +66,11 @@ tensorflow
6566
torch
6667
```
6768

69+
**Theano** - via conda or pip:
70+
```bash
71+
theano
72+
```
73+
6874
**NLP**
6975

7076
The NLP functionality contained within AutoML requires the [Kx NLP library](https://github.com/KxSystems/nlp) along with `gensim` which can be installed using conda or pip.

automl.q

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,26 @@
1-
// Fix for windows argv matplotlib conflict
21
\l p.q
3-
.p.import[`sys;:;`:argv;enlist""]
42

53
\d .automl
64
version:@[{AUTOMLVERSION};`;`development]
75
path:{string`automl^`$@[{"/"sv -1_"/"vs ssr[;"\\";"/"](-3#get .z.s)0};`;""]}`
86
loadfile:{$[.z.q;;-1]"Loading ",x:_[":"=x 0]x:$[10=type x;;string]x;system"l ",path,"/",x;}
7+
8+
// @kind description
9+
// @name commandLineParameters
10+
// @desc Retrieve command line parameters and convert to a kdb+ dictionary
11+
commandLineInput:first each .Q.opt .z.x
12+
13+
// @kind description
14+
// @name commandLineExecution
15+
// @desc If a user has defined both config and run command line arguments, the
16+
// interface will attempt to run the fully automated version of AutoML. The
17+
// content of the JSON file provided will be parsed to retrieve data
18+
// appropriately via ipc/from disk, then the q session will exit.
19+
commandLineArguments:lower key commandLineInput
20+
if[all`config`run in commandLineArguments;
21+
loadfile`:init.q;
22+
.ml.updDebug[];
23+
testRun:`test in commandLineArguments;
24+
runCommandLine[testRun];
25+
exit 0]
26+

build/buildscript.bat

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@ if not defined QLIC_KC (
1111
goto :nokdb
1212
)
1313

14-
1514
set PATH=C:\Miniconda3-x64;C:\Miniconda3-x64\Scripts;%PATH%
1615
conda config --set always_yes yes --set changeps1 no
1716
call "build\getkdb.bat" || goto :error

build/test.bat

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,18 @@
11
if defined QLIC_KC (
22
pip -q install -r requirements.txt
3-
echo getting test.q from embedpy
4-
git clone https://github.com/KxSystems/ml.git
3+
git clone https://github.com/KxSystems/ml.git
54
git clone https://github.com/KxSystems/nlp.git
65
pip -q install -r nlp/requirements.txt
76
python -m spacy download en
87
pip install gensim
98
pip install sobol-seq
10-
pip install pytorch
9+
pip install keras
10+
pip install tensorflow
11+
pip install torch
12+
pip install theano
13+
echo getting test.q from embedpy
1114
curl -fsSL -o test.q https://github.com/KxSystems/embedpy/raw/master/test.q
12-
q test.q -q
15+
env:PYTHONHASHSEED=0
16+
call "code\tests\testFiles.bat"
17+
1318
)

0 commit comments

Comments
 (0)