diff --git a/source/widgets/evaluate/confusionmatrix.md b/source/widgets/evaluate/confusionmatrix.md index 873ff42..e62049c 100644 --- a/source/widgets/evaluate/confusionmatrix.md +++ b/source/widgets/evaluate/confusionmatrix.md @@ -12,39 +12,38 @@ Shows proportions between the predicted and actual class. - Selected Data: data subset selected from confusion matrix - Data: data with the additional information on whether a data instance was selected -The [Confusion Matrix](https://en.wikipedia.org/wiki/Confusion_matrix) gives the number/proportion of instances between the predicted and actual class. The selection of the elements in the matrix feeds the corresponding instances into the output signal. This way, one can observe which specific instances were misclassified and how. +The [Confusion Matrix](https://en.wikipedia.org/wiki/Confusion_matrix) gives the number/proportion of instances between the predicted and actual class. Each row corresponds to a correct class, while columns represent the predicted classes. The rightmost column gives the number of instances from each class and the bottom row gives the number of instances classified into each class. -The widget usually gets the evaluation results from [Test & Score](../evaluate/testandscore.md); an example of the schema is shown below. +The selection of the elements in the matrix feeds the corresponding instances into the output signal. This way, one can observe which specific instances were misclassified and how. + +The widget usually gets the evaluation results from [Test and Score](../evaluate/testandscore.md); an example of the schema is shown below. ![](images/ConfusionMatrix-stamped.png) -1. When evaluation results contain data on multiple learning algorithms, we have to choose one in the *Learners* box. - The snapshot shows the confusion matrix for [Tree](../model/tree.md) and [Naive Bayesian](../model/naivebayes.md) models trained and tested on the *iris* data. The right-hand side of the widget contains the matrix for the naive Bayesian model (since this model is selected on the left). Each row corresponds to a correct class, while columns represent the predicted classes. For instance, four instances of *Iris-versicolor* were misclassified as *Iris-virginica*. The rightmost column gives the number of instances from each class (there are 50 irises of each of the three classes) and the bottom row gives the number of instances classified into each class (e.g., 48 instances were classified into virginica). -2. In *Show*, we select what data we would like to see in the matrix. +1. *Learners*: Choose a learning algorithm to display. +2. *Output*: define what is sent to the output, namely predicted classes (*Predictions*) or their probabilities (*Probabilities*). +3. The widget outputs every change if *Send Automatically* is ticked. If not, the user will need to click *Send Selected* to commit the changes. +4. *Show*: select what data to see in the matrix. - **Number of instances** shows correctly and incorrectly classified instances numerically. - - **Proportions of predicted** shows how many instances classified as, say, *Iris-versicolor* are in which true class; in the table we can read the 0% of them are actually setosae, 88.5% of those classified as versicolor are versicolors, and 7.7% are virginicae. - - **Proportions of actual** shows the opposite relation: of all true versicolors, 92% were classified as versicolors and 8% as virginicae. - ![](images/ConfusionMatrix-propTrue.png) -3. In *Select*, you can choose the desired output. - - **Correct** sends all correctly classified instances to the output by selecting the diagonal of the matrix. - - **Misclassified** selects the misclassified instances. - - **None** annuls the selection. - As mentioned before, one can also select individual cells of the table to select specific kinds of misclassified instances (e.g. the versicolors classified as virginicae). -4. When sending selected instances, the widget can add new attributes, such as predicted classes or their probabilities, if the corresponding options *Predictions* and/or *Probabilities* are checked. -5. The widget outputs every change if *Send Automatically* is ticked. If not, the user will need to click *Send Selected* to commit the changes. -6. Produce a report. + - **Proportions of predicted** shows how many instances classified as, say, *Iris-versicolor* are in which true class; in the table we can read the 0% of them are actually setosae, 95.9% of those classified as versicolor are versicolors, and 4.1% are virginicae. + ![](images/ConfusionMatrix-propPred.png) + - **Proportions of actual** shows the opposite relation: of all true versicolors, 94% were classified as versicolors and 6% as virginicae. + ![](images/ConfusionMatrix-propAct.png) + - **Sum of probabilities** sums the probabilities for each class. For example, there are 50 iris-setosas. This option sums probabilities for all *actual* versicolors to belong to iris-setosa (1.4), to iris-versicolor (41.4), and to iris-virginica (7.2). + ![](images/ConfusionMatrix-sumProb.png) +5. Select the desired output: + - **Select Correct** sends all correctly classified instances to the output by selecting the diagonal of the matrix. + - **Select Misclassified** selects the misclassified instances. + - **Clear Selection** annuls the selection. + As mentioned before, one can also select individual cells of the table to select specific kinds of misclassified instances. Example ------- -The following workflow demonstrates what this widget can be used for. - -![](images/ConfusionMatrix-Schema.png) - -[Test & Score](../evaluate/testandscore.md) gets the data from [File](../data/file.md) and two learning algorithms from [Naive Bayes](../model/naivebayes.md) and [Tree](../model/tree.md). It performs cross-validation or some other train-and-test procedures to get class predictions by both algorithms for all (or some) data instances. The test results are fed into the **Confusion Matrix**, where we can observe how many instances were misclassified and in which way. +The following workflow demonstrates what this widget can be used for. We are using the *iris* dataset for demonstration. -In the output, we used [Data Table](../data/datatable.md) to show the instances we selected in the confusion matrix. If we, for instance, click *Misclassified*, the table will contain all instances which were misclassified by the selected method. +![](images/ConfusionMatrix-Example.png) -The [Scatter Plot](../visualize/scatterplot.md) gets two sets of data. From the [File](../data/file.md) widget it gets the complete data, while the confusion matrix sends only the selected data, misclassifications for instance. The scatter plot will show all the data, with bold symbols representing the selected data. +[Test and Score](../evaluate/testandscore.md) gets the data from [File](../data/file.md) and a learning algorithm, [Naive Bayes](../model/naivebayes.md). It performs cross-validation or some other train-and-test procedures to get class predictions for all (or some) data instances. The test results are fed into the **Confusion Matrix**, where we can observe how many instances were misclassified and in which way. -![](images/ConfusionMatrix-Example.png) +In the output, we used [Scatter Plot](../visualize/scatterplot.md), which gets two sets of data, the whole dataset from the File widget, and a subset from Confusion Matrix. Now, we wish to show where the misclassifications lie in the Scatter Plot. We click *Misclassified*, which selects all the off-diagonal (misclassified) instances from the confusion matrix. The scatter plot will show all the data in a graph, with bold symbols representing the selected (misclassified) data points. diff --git a/source/widgets/evaluate/icons/calibration-plot.png b/source/widgets/evaluate/icons/calibration-plot.png deleted file mode 100644 index bbce5c6..0000000 Binary files a/source/widgets/evaluate/icons/calibration-plot.png and /dev/null differ diff --git a/source/widgets/evaluate/icons/confusion-matrix.png b/source/widgets/evaluate/icons/confusion-matrix.png deleted file mode 100644 index ace561e..0000000 Binary files a/source/widgets/evaluate/icons/confusion-matrix.png and /dev/null differ diff --git a/source/widgets/evaluate/icons/lift-curve.png b/source/widgets/evaluate/icons/lift-curve.png deleted file mode 100644 index 134fc14..0000000 Binary files a/source/widgets/evaluate/icons/lift-curve.png and /dev/null differ diff --git a/source/widgets/evaluate/icons/predictions.png b/source/widgets/evaluate/icons/predictions.png deleted file mode 100644 index 3ef99a4..0000000 Binary files a/source/widgets/evaluate/icons/predictions.png and /dev/null differ diff --git a/source/widgets/evaluate/icons/roc-analysis.png b/source/widgets/evaluate/icons/roc-analysis.png deleted file mode 100644 index bf1af55..0000000 Binary files a/source/widgets/evaluate/icons/roc-analysis.png and /dev/null differ diff --git a/source/widgets/evaluate/icons/test-and-score.png b/source/widgets/evaluate/icons/test-and-score.png deleted file mode 100644 index 0885531..0000000 Binary files a/source/widgets/evaluate/icons/test-and-score.png and /dev/null differ diff --git a/source/widgets/evaluate/images/ConfusionMatrix-Example.png b/source/widgets/evaluate/images/ConfusionMatrix-Example.png index 2f6515d..39e508a 100644 Binary files a/source/widgets/evaluate/images/ConfusionMatrix-Example.png and b/source/widgets/evaluate/images/ConfusionMatrix-Example.png differ diff --git a/source/widgets/evaluate/images/ConfusionMatrix-propAct.png b/source/widgets/evaluate/images/ConfusionMatrix-propAct.png new file mode 100644 index 0000000..754af6f Binary files /dev/null and b/source/widgets/evaluate/images/ConfusionMatrix-propAct.png differ diff --git a/source/widgets/evaluate/images/ConfusionMatrix-propPred.png b/source/widgets/evaluate/images/ConfusionMatrix-propPred.png new file mode 100644 index 0000000..4e56b5f Binary files /dev/null and b/source/widgets/evaluate/images/ConfusionMatrix-propPred.png differ diff --git a/source/widgets/evaluate/images/ConfusionMatrix-propTrue.png b/source/widgets/evaluate/images/ConfusionMatrix-propTrue.png deleted file mode 100644 index 8f4edff..0000000 Binary files a/source/widgets/evaluate/images/ConfusionMatrix-propTrue.png and /dev/null differ diff --git a/source/widgets/evaluate/images/ConfusionMatrix-stamped.png b/source/widgets/evaluate/images/ConfusionMatrix-stamped.png index c473132..7a1b9ff 100644 Binary files a/source/widgets/evaluate/images/ConfusionMatrix-stamped.png and b/source/widgets/evaluate/images/ConfusionMatrix-stamped.png differ diff --git a/source/widgets/evaluate/images/ConfusionMatrix-sumProb.png b/source/widgets/evaluate/images/ConfusionMatrix-sumProb.png new file mode 100644 index 0000000..7d40816 Binary files /dev/null and b/source/widgets/evaluate/images/ConfusionMatrix-sumProb.png differ diff --git a/source/widgets/evaluate/images/ParameterFitter-Example.png b/source/widgets/evaluate/images/ParameterFitter-Example.png new file mode 100644 index 0000000..607f639 Binary files /dev/null and b/source/widgets/evaluate/images/ParameterFitter-Example.png differ diff --git a/source/widgets/evaluate/images/ParameterFitter-stamped.png b/source/widgets/evaluate/images/ParameterFitter-stamped.png new file mode 100644 index 0000000..e3a0836 Binary files /dev/null and b/source/widgets/evaluate/images/ParameterFitter-stamped.png differ diff --git a/source/widgets/evaluate/images/ParameterFitter.png b/source/widgets/evaluate/images/ParameterFitter.png deleted file mode 100644 index 9450571..0000000 Binary files a/source/widgets/evaluate/images/ParameterFitter.png and /dev/null differ diff --git a/source/widgets/evaluate/images/Predictions-Example1.png b/source/widgets/evaluate/images/Predictions-Example1.png index 5531805..d6d42ea 100644 Binary files a/source/widgets/evaluate/images/Predictions-Example1.png and b/source/widgets/evaluate/images/Predictions-Example1.png differ diff --git a/source/widgets/evaluate/images/Predictions-Example2.png b/source/widgets/evaluate/images/Predictions-Example2.png index 6626a3d..7b534c4 100644 Binary files a/source/widgets/evaluate/images/Predictions-Example2.png and b/source/widgets/evaluate/images/Predictions-Example2.png differ diff --git a/source/widgets/evaluate/images/Predictions-Regression.png b/source/widgets/evaluate/images/Predictions-Regression.png new file mode 100644 index 0000000..2416cad Binary files /dev/null and b/source/widgets/evaluate/images/Predictions-Regression.png differ diff --git a/source/widgets/evaluate/images/Predictions-stamped.png b/source/widgets/evaluate/images/Predictions-stamped.png index 318a1ed..847afcd 100644 Binary files a/source/widgets/evaluate/images/Predictions-stamped.png and b/source/widgets/evaluate/images/Predictions-stamped.png differ diff --git a/source/widgets/evaluate/images/Preprocess-Models1.png b/source/widgets/evaluate/images/Preprocess-Models1.png new file mode 100644 index 0000000..fd4a8cc Binary files /dev/null and b/source/widgets/evaluate/images/Preprocess-Models1.png differ diff --git a/source/widgets/evaluate/images/Preprocess-Models2.png b/source/widgets/evaluate/images/Preprocess-Models2.png new file mode 100644 index 0000000..c65be6c Binary files /dev/null and b/source/widgets/evaluate/images/Preprocess-Models2.png differ diff --git a/source/widgets/evaluate/images/Preprocess-Models3.png b/source/widgets/evaluate/images/Preprocess-Models3.png new file mode 100644 index 0000000..42f2fb6 Binary files /dev/null and b/source/widgets/evaluate/images/Preprocess-Models3.png differ diff --git a/source/widgets/evaluate/images/TestAndScore-Classification.png b/source/widgets/evaluate/images/TestAndScore-Classification.png index 5e7b89c..ade81ac 100644 Binary files a/source/widgets/evaluate/images/TestAndScore-Classification.png and b/source/widgets/evaluate/images/TestAndScore-Classification.png differ diff --git a/source/widgets/evaluate/images/TestAndScore-Example.png b/source/widgets/evaluate/images/TestAndScore-Example.png index 41de29d..fcd95bc 100644 Binary files a/source/widgets/evaluate/images/TestAndScore-Example.png and b/source/widgets/evaluate/images/TestAndScore-Example.png differ diff --git a/source/widgets/evaluate/images/TestAndScore-Regression.png b/source/widgets/evaluate/images/TestAndScore-Regression.png index 11e8673..309f37d 100644 Binary files a/source/widgets/evaluate/images/TestAndScore-Regression.png and b/source/widgets/evaluate/images/TestAndScore-Regression.png differ diff --git a/source/widgets/evaluate/images/TestAndScore-stamped.png b/source/widgets/evaluate/images/TestAndScore-stamped.png index b685993..343e9a3 100644 Binary files a/source/widgets/evaluate/images/TestAndScore-stamped.png and b/source/widgets/evaluate/images/TestAndScore-stamped.png differ diff --git a/source/widgets/evaluate/parameterfitter.md b/source/widgets/evaluate/parameterfitter.md index cef2c0c..fb4f2f0 100644 --- a/source/widgets/evaluate/parameterfitter.md +++ b/source/widgets/evaluate/parameterfitter.md @@ -8,11 +8,22 @@ Find the best hyper-parameters for a model. - Data: input data - Learner: learning algorithm -Parameter fitter shows performance of a learning algorithms with different settings of a hyper-parameter. The widget is currently limited to a single integer parameter. Not all learning algorithms support hyper-parameter tuning. +Parameter fitter shows performance of a learning algorithms with different settings of a hyper-parameter. The widget is currently limited to a single integer parameter. Not all learning algorithms support hyper-parameter tuning (currently, only [Random Forest](../model/randomforest.md) and [PLS](../model/pls.md)). The widget shows a plot of the model's performance at different values of the parameter. The graph shows AUC for classification problems and R2 for regression. -![](images/ParameterFitter.png) +![](images/ParameterFitter-stamped.png) 1. Choose the parameter to fit. -2. Define the lower and the upper limit; step size is determined automatically. -3. Alternatively, specifies the values for the parameter. The widget also accepts `...`, e.g. `1, 2, 3, ..., 10` or `40, 60, ..., 100`. When the parameter has a minimal value (e.g. the number of components cannot be negative), one can also omit the lower bound, e.g. `..., 80, 100`; and if the parameter has a maximal value, one can omit the upper bound, e.g. `2, 4, 6, ...,`. -4. A plot showing the performance at different values of the parameter. The graph shows AUC for classification problems and R2 for regression. \ No newline at end of file + *Range*: Define the lower and the upper limit; step size is determined automatically. + *Manual*: Alternatively, specify the values for the parameter. The widget also accepts `...`, e.g. `1, 2, 3, ..., 10` or `40, 60, ..., 100`. When the parameter has a minimal value (e.g. the number of components cannot be negative), one can also omit the lower bound, e.g. `..., 80, 100`; and if the parameter has a maximal value, one can omit the upper bound, e.g. `2, 4, 6, ...,`. +2. If *Apply Automatically* is ticked, changes are communicated automatically. Alternatively, click *Apply*. + +Example +------- + +Here is a simple example on how to fit parameters using the **Parameter Fitter** widget. We are using the *heart-disease* data for this example and loading it with the [File](../data/file.md) widget. We pass the data to Parameter Fitter. The widget also needs a learner to fit, the [Random Forest](../model/randomforest.md) in this case. + +Parameter Fitter enables observing performance for a varying number of trees. We set the range from 1 to 10, namely we will observe performance for every number of trees up to 10. + +We see there's a slight peak in AUC value for cross-validation at 3 trees, while 8 trees seem to be optimal overall. (Note that this is just a toy example!) + +![](images/ParameterFitter-Example.png) \ No newline at end of file diff --git a/source/widgets/evaluate/predictions.md b/source/widgets/evaluate/predictions.md index f23240e..fdb3d9e 100644 --- a/source/widgets/evaluate/predictions.md +++ b/source/widgets/evaluate/predictions.md @@ -10,6 +10,7 @@ Shows models' predictions on the data. **Outputs** +- Selected Predictions: a subset of predictions - Predictions: data with added predictions - Evaluation Results: results of testing classification algorithms @@ -17,17 +18,24 @@ The widget receives a dataset and one or more predictors (predictive models, not ![](images/Predictions-stamped.png) -1. Information on the input, namely the number of instances to predict, the number of predictors and the task (classification or regression). If you have sorted the data table by attribute and you wish to see the original view, press *Restore Original Order*. -2. You can select the options for classification. If *Predicted class* is ticked, the view provides information on predicted class. If *Predicted probabilities for* is ticked, the view provides information on probabilities predicted by the classifier(s). You can also select the predicted class displayed in the view. The option *Draw distribution bars* provides a visualization of probabilities. -3. By ticking the *Show full dataset*, you can view the entire data table (otherwise only class variable will be shown). -4. Select the desired output. -5. Predictions. +1. *Show probabilities for* enables showing different kinds of prediction probabilities, namely for classes in the data, for classes known to the model, or classes in data and model. + If *Show classification errors* is ticked, an additional error column is shown, displaying 1 - probability assigned to the correct class. + If you have sorted the data table by attribute and you wish to see the original view, press *Restore Original Order*. +2. *Show performance scores* shows scores for model comparison. For classification, one can set the *Target class* to compare scores on different classes. *(Average over classes)* averages individual class scores. -The widget show the probabilities and final decisions of [predictive models](https://en.wikipedia.org/wiki/Predictive_modelling). The output of the widget is another dataset, where predictions are appended as new meta attributes. You can select which features you wish to output (original data, predictions, probabilities). The result can be observed in a [Data Table](../data/datatable.md). If the predicted data includes true class values, the result of prediction can also be observed in a [Confusion Matrix](../evaluate/confusionmatrix.md). +The widget show the probabilities and final decisions of [predictive models](https://en.wikipedia.org/wiki/Predictive_modelling). There are three widget outputs, namely another dataset, where predictions are appended as new meta attributes, a selected subset of enhanced dataset, or evaluation results (predicted data icnluding a true class values) to be used with [Confusion Matrix](../evaluate/confusionmatrix.md), [ROC Analysis](../evaluate/rocanalysis.md), [Performance Curve](../evaluate/performancecurve.md) or [Calibration Plot](../evaluate/calibrationplot.md). Predictions can be observed in a [Data Table](../data/datatable.md) or analyzed downstream. + +![](images/Predictions-Regression.png) + +For regression, the widget shows different types of regression errors, namely *Difference* (to true value), *Absolute difference*, *Relative difference* (in percentage), and *Absolute relative*. Note that "positive" differences (overestimations) are shown in red, while "negative" differences (underestimations) are shown in blue. + +On the left side, a pink line indicates the forecast value, while the dot indicates a true value. This, along with the error column, allows for a quick visual estimation of predictive errors. Examples -------- +#### Predict on new data + In the first example, we will use *Attrition - Train* data from the [Datasets](../data/datasets.md) widget. This is a data on attrition of employees. In other words, we wish to know whether a certain employee will resign from the job or not. We will construct a predictive model with the [Tree](../model/tree.md) widget and observe probabilities in **Predictions**. For predictions we need both the training data, which we have loaded in the first **Datasets** widget and the data to predict, which we will load in another [Datasets](../data/datasets.md) widget. We will use *Attrition - Predict* data this time. Connect the second data set to **Predictions**. Now we can see predictions for the three data instances from the second data set. @@ -36,16 +44,18 @@ The [Tree](../model/tree.md) model predicts none of the employees will leave the ![](images/Predictions-Example1.png) -In the second example, we will see how to properly use [Preprocess](../data/preprocess.md) with **Predictions** or [Test & Score](../evaluate/testandscore.md). +#### How to use preprocessing + +In the second example, we will see how to properly use [Preprocess](../data/preprocess.md) with **Predictions** or [Test &andScore](../evaluate/testandscore.md). This time we are using the *heart disease.tab* data from the [File](../data/file.md) widget. You can access the data through the dropdown menu. This is a dataset with 303 patients that came to the doctor suffering from a chest pain. After the tests were done, some patients were found to have diameter narrowing and others did not (this is our class variable). -The heart disease data have some missing values and we wish to account for that. First, we will split the data set into train and test data with [Data Sampler](../data/datasampler.md). +The heart disease data have some missing values and we wish to account for that. First, we will split the data set into train and test data with [Data Sampler](../data/datasampler.md), say 70% for training and 30% for testing. -Then we will send the *Data Sample* into [Preprocess](../data/preprocess.md). We will use *Impute Missing Values*, but you can try any combination of preprocessors on your data. We will send preprocessed data to [Logistic Regression](../model/logisticregression.md) and the constructed model to **Predictions**. +Then we will send the *Data Sample* into [Preprocess](../data/preprocess.md). We will use *Impute Missing Values*, but you can try any combination of preprocessors on your data. We will send *Preprocessed Data* to [Logistic Regression](../model/logisticregression.md) and the constructed model to **Predictions**. Finally, **Predictions** also needs the data to predict on. We will use the output of [Data Sampler](../data/datasampler.md) for prediction, but this time not the *Data Sample*, but the *Remaining Data*, this is the data that wasn't used for training the model. -Notice how we send the remaining data directly to **Predictions** without applying any preprocessing. This is because Orange handles preprocessing on new data internally to prevent any errors in the model construction. The exact same preprocessor that was used on the training data will be used for predictions. The same process applies to [Test & Score](../evaluate/testandscore.md). +Notice how we send the remaining data directly to **Predictions** without applying any preprocessing. This is because Orange handles preprocessing on new data internally to prevent any errors in the model construction. The exact same preprocessor that was used on the training data will be used for predictions. The same process applies to [Test and Score](../evaluate/testandscore.md). ![](images/Predictions-Example2.png) diff --git a/source/widgets/evaluate/testandscore.md b/source/widgets/evaluate/testandscore.md index d33a92a..2c44fc8 100644 --- a/source/widgets/evaluate/testandscore.md +++ b/source/widgets/evaluate/testandscore.md @@ -8,28 +8,31 @@ Tests learning algorithms on data. - Data: input dataset - Test Data: separate data for testing - Learner: learning algorithm(s) +- Preprocessor: preprocessing method(s) **Outputs** +- Predictions: data with added predictions - Evaluation Results: results of testing classification algorithms -The widget tests learning algorithms. Different sampling schemes are available, including using separate test data. The widget does two things. First, it shows a table with different classifier performance measures, such as [classification accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precision) and [area under the curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve). Second, it outputs evaluation results, which can be used by other widgets for analyzing the performance of classifiers, such as [ROC Analysis](../evaluate/rocanalysis.md) or [Confusion Matrix](../evaluate/confusionmatrix.md). +The widget tests learning algorithms. Different sampling schemes are available, including using separate test data. The widget does four things. First, it shows a table with different classifier performance measures, such as [classification accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precision) and [area under the curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve). Second, it shows model comparison below evaluation results to determine how the model results truly vary. Third, it outputs evaluation results, which can be used by other widgets for analyzing the performance of classifiers, such as [ROC Analysis](../evaluate/rocanalysis.md) or [Confusion Matrix](../evaluate/confusionmatrix.md). Fourth, it outputs predictions, which can be observed directly in a [Data Table](../data/datatable.md) or analyzed downstream. The *Learner* signal has an uncommon property: it can be connected to more than one widget to test multiple learners with the same procedures. ![](images/TestAndScore-stamped.png) 1. The widget supports various sampling methods. - - [Cross-validation](https://en.wikipedia.org/wiki/Cross-validation_\(statistics\)) splits the data into a given number of folds (usually 5 or 10). The algorithm is tested by holding out examples from one fold at a time; the model is induced from other folds and examples from the held out fold are classified. This is repeated for all the folds. + - [Cross-validation](https://en.wikipedia.org/wiki/Cross-validation_\(statistics\)) splits the data into a given *number of folds* (usually 5 or 10). The algorithm is tested by holding out examples from one fold at a time; the model is induced from other folds and examples from the held out fold are classified. This is repeated for all the folds. The *Stratified* option ensures the folds are similar in terms of class distribution. - **Cross validation by feature** performs cross-validation but folds are defined by the selected categorical feature from meta-features. - - **Random sampling** randomly splits the data into the training and testing set in the given proportion (e.g. 70:30); the whole procedure is repeated for a specified number of times. + - **Random sampling** randomly splits the data into the training and testing set in the given proportion (e.g. 70:30, see *Training set size*); the whole procedure is repeated for a specified number of times (*Repeat train/test*). *Statified* option ensures the folds are similar in terms of class distribution. - **Leave-one-out** is similar, but it holds out one instance at a time, inducing the model from all others and then classifying the held out instances. This method is obviously very stable, reliable... and very slow. - **Test on train data** uses the whole dataset for training and then for testing. This method practically always gives wrong results. - **Test on test data**: the above methods use the data from *Data* signal only. To input another dataset with testing examples (for instance from another file or some data selected in another widget), we select *Separate Test Data* signal in the communication channel and select Test on test data. -2. For classification, *Target class* can be selected at the bottom of the widget. When *Target class* is (Average over classes), methods return scores that are weighted averages over all classes. For example, in case of the classifier with 3 classes, scores are computed for class 1 as a target class, class 2 as a target class, and class 3 as a target class. Those scores are averaged with weights based on the class size to retrieve the final score. -3. The widget will compute a number of performance statistics. A few are shown by default. To see others, right-click on the header and select the desired statistic. - - Classification +2. The widget will compute a number of performance statistics. A few are shown by default. To see others, right-click on the header and select the desired statistic. + - **Classification**: ![](images/TestAndScore-Classification.png) + *Evaluation results for target* can be selected at the top of the evaluation box. When *(None, show average over classes)* is selected, methods return scores that are weighted averages over all classes. For example, in case of the classifier with 3 classes, scores are computed for class 1 as a target class, class 2 as a target class, and class 3 as a target class. Those scores are averaged with weights based on the class size to retrieve the final score. + Performance statistics for classification: - [Area under ROC](http://gim.unmc.edu/dxtests/roc3.htm) is the area under the receiver-operating curve. - [Classification accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precision) is the proportion of correctly classified examples. - [F-1](https://en.wikipedia.org/wiki/F1_score) is a weighted harmonic mean of precision and recall (see below). @@ -44,14 +47,14 @@ The *Learner* signal has an uncommon property: it can be connected to more than ![](images/TestAndScore-Regression.png) - [MSE](https://en.wikipedia.org/wiki/Mean_squared_error) measures the average of the squares of the errors or deviations (the difference between the estimator and what is estimated). - [RMSE](https://en.wikipedia.org/wiki/Root_mean_square) is the square root of the arithmetic mean of the squares of a set of numbers (a measure of imperfection of the fit of the estimator to the data) - - [MAE]() is used to measure how close forecasts or predictions are to eventual outcomes. - - [R2]() is interpreted as the proportion of the variance in the dependent variable that is predictable from the independent variable. + - [MAE](https://en.wikipedia.org/wiki/Mean_absolute_error) is used to measure how close forecasts or predictions are to eventual outcomes. + - [MAPE](https://en.wikipedia.org/wiki/Mean_absolute_percentage_error) or mean absolute percentage error measures how far off your predictions are, on average, in percentage terms. + - [R2](https://en.wikipedia.org/wiki/Coefficient_of_determination) is interpreted as the proportion of the variance in the dependent variable that is predictable from the independent variable. - [CVRMSE](https://en.wikipedia.org/wiki/Root-mean-square_deviation) is RMSE normalized by the mean value of actual values. - Train time - cumulative time in seconds used for training models. - Test time - cumulative time in seconds used for testing models. -4. Choose the score for pairwise comparison of models and the region of practical equivalence (ROPE), in which differences are considered negligible. -5. Pairwise comparison of models using the selected score (available only for cross-validation). The number in the table gives the probability that the model corresponding to the row has a higher score than the model corresponding to the column. What the higher score means depends on the metric: a higher score can either mean a model is better (for example, CA or AUC) or the opposite (for example, RMSE). If negligible difference is enabled, the smaller number below shows the probability that the difference between the pair is negligible. The test is based on the [Bayesian interpretation of the t-test](https://link.springer.com/article/10.1007/s10994-015-5486-z) ([shorter introduction](https://baycomp.readthedocs.io/en/latest/introduction.html)). -6. Get help and produce a report. +3. Choose the score for pairwise comparison of models and the region of practical equivalence (ROPE), in which differences are considered negligible. + Pairwise comparison of models using the selected score (available only for cross-validation). The number in the table gives the probability that the model corresponding to the row has a higher score than the model corresponding to the column. What the higher score means depends on the metric: a higher score can either mean a model is better (for example, CA or AUC) or the opposite (for example, RMSE). If negligible difference is enabled, the smaller number below shows the probability that the difference between the pair is negligible. The test is based on the [Bayesian interpretation of the t-test](https://link.springer.com/article/10.1007/s10994-015-5486-z) ([shorter introduction](https://baycomp.readthedocs.io/en/latest/introduction.html)). Preprocessing for predictive modeling -------------------------------------- @@ -60,22 +63,22 @@ When building predictive models, one has to be careful about how to preprocess t 1. Connect [Preprocess](../data/preprocess.md) to the learner. This will override the default preprocessing pipeline for the learner and apply only custom preprocessing pipeline (default preprocessing steps are described in each learner's documentation). The procedure might lead to errors within the learner. - ![](../data/images/Preprocess-Models1.png) + ![](images/Preprocess-Models1.png){width=400px} 2. Connect **Preprocess** to Test and Score. This will apply the preprocessors to each batch within cross-validation. Then the learner's preprocessors will be applied to the preprocessed subset. - ![](../data/images/Preprocess-Models2.png) + ![](images/Preprocess-Models2.png){width=400px} Finally, there's a wrong way to do it. Connecting **Preprocess** directly to the original data and outputting preprocessed data set will likely overfit the model. Don't do it. - ![](../data/images/Preprocess-Models3.png) + ![](images/Preprocess-Models3.png){width=400px} Example ------- -In a typical use of the widget, we give it a dataset and a few learning algorithms and we observe their performance in the table inside the **Test & Score** widget and in the [ROC](../evaluate/rocanalysis.md). The data is often preprocessed before testing; in this case we did some manual feature selection ([Select Columns](../data/selectcolumns.md) widget) on *Titanic* dataset, where we want to know only the sex and status of the survived and omit the age. +In a typical use of the widget, we give it a dataset, say *titanic*, and a few learning algorithms ([Naive Bayes](../model/naivebayes.md), [Tree](../model/tree.md), [Logistic Regression](../model/logisticregression.md), and [SVM](../model/svm.md)) and we observe their performance in the table inside the **Test & Score** widget and in the [ROC](../evaluate/rocanalysis.md). The data is often preprocessed before testing; in this case we did some manual feature selection ([Select Columns](../data/selectcolumns.md) widget) on *Titanic* dataset, where we kept sex and status in the features and removed age. -In the bottom table, we have a pairwise comparison of models. We selected that comparison is based on the _area under ROC curve_ statistic. The number in the table gives the probability that the model corresponding to the row is better than the model corresponding to the column. We can, for example, see that probability for the tree to be better than SVM is almost one, and the probability that tree is better than Naive Bayes is 0.001. Smaller numbers in the table are probabilities that the difference between the pair is negligible based on the negligible threshold 0.1. +In the bottom table of the Test and Score widget, we have a pairwise comparison of models. We selected that comparison is based on the _area under ROC curve_ statistic. The number in the table gives the probability that the model corresponding to the row is better than the model corresponding to the column. We can, for example, see that probability for the tree to be better than SVM is almost one, and the probability that Naive Bayes is better than the tree is 0.085. ![](images/TestAndScore-Example.png) diff --git a/source/widgets/evaluate/workflows/confusionmatrix.ows b/source/widgets/evaluate/workflows/confusionmatrix.ows new file mode 100644 index 0000000..d108f31 --- /dev/null +++ b/source/widgets/evaluate/workflows/confusionmatrix.ows @@ -0,0 +1,135 @@ + + + + + + + + + + + + + + + + + + + + gASVzwsAAAAAAAB9lCiMEmNvbnRyb2xBcmVhVmlzaWJsZZSIjAxyZWNlbnRfcGF0aHOUXZQojB5v +cmFuZ2V3aWRnZXQudXRpbHMuZmlsZWRpYWxvZ3OUjApSZWNlbnRQYXRolJOUKYGUfZQojAdhYnNw +YXRolIwzL1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL2lyaXMudGFi +lIwGcHJlZml4lIwPc2FtcGxlLWRhdGFzZXRzlIwHcmVscGF0aJSMCGlyaXMudGFilIwFdGl0bGWU +jACUjAVzaGVldJRoEIwLZmlsZV9mb3JtYXSUTnViaAYpgZR9lChoCYw8L1VzZXJzL2FqZGEvb3Jh +bmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL2hlYXJ0X2Rpc2Vhc2UudGFilGgLaAxoDYwRaGVh +cnRfZGlzZWFzZS50YWKUaA9oEGgRaBBoEk51YmgGKYGUfZQoaAmMNi9Vc2Vycy9hamRhL29yYW5n +ZS9vcmFuZ2UzL09yYW5nZS9kYXRhc2V0cy9ob3VzaW5nLnRhYpRoC2gMaA2MC2hvdXNpbmcudGFi +lGgPaBBoEWgQaBJOdWJoBimBlH2UKGgJjDYvVXNlcnMvYWpkYS9vcmFuZ2Uvb3JhbmdlMy9PcmFu +Z2UvZGF0YXNldHMvdGl0YW5pYy50YWKUaAtoDGgNjAt0aXRhbmljLnRhYpRoD2gQaBFoEGgSTnVi +aAYpgZR9lChoCYw9L1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL2Jy +b3duLXNlbGVjdGVkLnRhYpRoC2gMaA2MEmJyb3duLXNlbGVjdGVkLnRhYpRoD2gQaBFoEGgSTnVi +aAYpgZR9lChoCYwyL1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL3pv +by50YWKUaAtoDGgNjAd6b28udGFilGgPaBBoEWgQaBJOdWJljAtyZWNlbnRfdXJsc5RdlIwTc2F2 +ZWRXaWRnZXRHZW9tZXRyeZRDQgHZ0MsAAwAAAAACigAAAFAAAAThAAADCAAAAooAAABsAAAE4QAA +AwgAAAAAAAAAAAXoAAACigAAAGwAAAThAAADCJSMC3NoZWV0X25hbWVzlH2UjAZzb3VyY2WUSwCM +A3VybJRoEIwNZG9tYWluX2VkaXRvcpR9lIwLX192ZXJzaW9uX1+USwGMEGNvbnRleHRfc2V0dGlu +Z3OUXZQojBVvcmFuZ2V3aWRnZXQuc2V0dGluZ3OUjAdDb250ZXh0lJOUKYGUfZQojAZ2YWx1ZXOU +fZQojAl2YXJpYWJsZXOUXZRoL32UaDtdlChdlCiMDHNlcGFsIGxlbmd0aJSMFE9yYW5nZS5kYXRh +LnZhcmlhYmxllIwSQ29udGludW91c1ZhcmlhYmxllJOUSwBoEIhlXZQojAtzZXBhbCB3aWR0aJRo +Q0sAaBCIZV2UKIwMcGV0YWwgbGVuZ3RolGhDSwBoEIhlXZQojAtwZXRhbCB3aWR0aJRoQ0sAaBCI +ZV2UKIwEaXJpc5RoQYwQRGlzY3JldGVWYXJpYWJsZZSTlEsBjCxJcmlzLXNldG9zYSwgSXJpcy12 +ZXJzaWNvbG9yLCBJcmlzLXZpcmdpbmljYZSJZWVzaDFLAXWMCmF0dHJpYnV0ZXOUKIwMc2VwYWwg +bGVuZ3RolEsChpSMC3NlcGFsIHdpZHRolEsChpSMDHBldGFsIGxlbmd0aJRLAoaUjAtwZXRhbCB3 +aWR0aJRLAoaUdJSMBW1ldGFzlCmMCmNsYXNzX3ZhcnOUjARpcmlzlF2UKIwLSXJpcy1zZXRvc2GU +jA9JcmlzLXZlcnNpY29sb3KUjA5JcmlzLXZpcmdpbmljYZRlhpSFlIwSbW9kaWZpZWRfdmFyaWFi +bGVzlF2UdWJoNimBlH2UKGg5fZQoaDtdlGgvfZRoO12UKF2UKIwDYWdllGhDSwBoEIhlXZQojAZn +ZW5kZXKUaE1LAIwMZmVtYWxlLCBtYWxllIllXZQojApjaGVzdCBwYWlulGhNSwCMNGFzeW1wdG9t +YXRpYywgYXR5cGljYWwgYW5nLCBub24tYW5naW5hbCwgdHlwaWNhbCBhbmeUiWVdlCiMCHJlc3Qg +U0JQlGhDSwBoEIhlXZQojAtjaG9sZXN0ZXJvbJRoQ0sAaBCIZV2UKIwZZmFzdGluZyBibG9vZCBz +dWdhciA+IDEyMJRoTUsAjAQwLCAxlIhlXZQojAhyZXN0IEVDR5RoTUsAjCxub3JtYWwsIGxlZnQg +dmVudCBoeXBlcnRyb3BoeSwgU1QtVCBhYm5vcm1hbJSJZV2UKIwGbWF4IEhSlGhDSwBoEIhlXZQo +jA1leGVyYyBpbmQgYW5nlGhNSwCMBDAsIDGUiGVdlCiMDlNUIGJ5IGV4ZXJjaXNllGhDSwBoEIhl +XZQojBFzbG9wZSBwZWFrIGV4YyBTVJRoTUsAjBx1cHNsb3BpbmcsIGZsYXQsIGRvd25zbG9waW5n +lIllXZQojBVtYWpvciB2ZXNzZWxzIGNvbG9yZWSUaENLAGgQiGVdlCiMBHRoYWyUaE1LAIwnbm9y +bWFsLCByZXZlcnNhYmxlIGRlZmVjdCwgZml4ZWQgZGVmZWN0lIllXZQojBJkaWFtZXRlciBuYXJy +b3dpbmeUaE1LAYwEMCwgMZSIZWVzaDFLAXVoTyiMA2FnZZRLAoaUjAZnZW5kZXKUXZQojAZmZW1h +bGWUjARtYWxllGWGlIwKY2hlc3QgcGFpbpRdlCiMDGFzeW1wdG9tYXRpY5SMDGF0eXBpY2FsIGFu +Z5SMC25vbi1hbmdpbmFslIwLdHlwaWNhbCBhbmeUZYaUjAhyZXN0IFNCUJRLAoaUjAtjaG9sZXN0 +ZXJvbJRLAoaUjBlmYXN0aW5nIGJsb29kIHN1Z2FyID4gMTIwlF2UKIwBMJSMATGUZYaUjAhyZXN0 +IEVDR5RdlCiMBm5vcm1hbJSMFWxlZnQgdmVudCBoeXBlcnRyb3BoeZSMDVNULVQgYWJub3JtYWyU +ZYaUjAZtYXggSFKUSwKGlIwNZXhlcmMgaW5kIGFuZ5RdlChoomijZYaUjA5TVCBieSBleGVyY2lz +ZZRLAoaUjBFzbG9wZSBwZWFrIGV4YyBTVJRdlCiMCXVwc2xvcGluZ5SMBGZsYXSUjAtkb3duc2xv +cGluZ5RlhpSMFW1ham9yIHZlc3NlbHMgY29sb3JlZJRLAoaUjAR0aGFslF2UKIwGbm9ybWFslIwR +cmV2ZXJzYWJsZSBkZWZlY3SUjAxmaXhlZCBkZWZlY3SUZYaUdJRoWSloWowSZGlhbWV0ZXIgbmFy +cm93aW5nlF2UKGiiaKNlhpSFlGhiXZR1Ymg2KYGUfZQoaDl9lChoO12UaC99lGg7XZQoXZQojARD +UklNlGhDSwBoEIhlXZQojAJaTpRoQ0sAaBCIZV2UKIwFSU5EVVOUaENLAGgQiGVdlCiMBENIQVOU +aENLAGgQiGVdlCiMA05PWJRoQ0sAaBCIZV2UKIwCUk2UaENLAGgQiGVdlCiMA0FHRZRoQ0sAaBCI +ZV2UKIwDRElTlGhDSwBoEIhlXZQojANSQUSUaENLAGgQiGVdlCiMA1RBWJRoQ0sAaBCIZV2UKIwH +UFRSQVRJT5RoQ0sAaBCIZV2UKIwBQpRoQ0sAaBCIZV2UKIwFTFNUQVSUaENLAGgQiGVdlCiMBE1F +RFaUaENLAWgQiGVlc2gxSwF1aE8ojARDUklNlEsChpSMAlpOlEsChpSMBUlORFVTlEsChpSMBENI +QVOUSwKGlIwDTk9YlEsChpSMAlJNlEsChpSMA0FHRZRLAoaUjANESVOUSwKGlIwDUkFElEsChpSM +A1RBWJRLAoaUjAdQVFJBVElPlEsChpRo40sChpSMBUxTVEFUlEsChpR0lGhZKWhajARNRURWlEsC +hpSFlGhiXZR1YmV1Lg== + + gASVzwUAAAAAAAB9lCiMFGNvbXBhcmlzb25fY3JpdGVyaW9ulEsAjBJjb250cm9sQXJlYVZpc2li +bGWUiIwNY3Zfc3RyYXRpZmllZJSIjAduX2ZvbGRzlEsCjAluX3JlcGVhdHOUSwOMCnJlc2FtcGxp +bmeUSwCMBHJvcGWURz+5mZmZmZmajAtzYW1wbGVfc2l6ZZRLCYwTc2F2ZWRXaWRnZXRHZW9tZXRy +eZRDQgHZ0MsAAwAAAAAAZgAAAPsAAAKHAAADBwAAAGYAAAEXAAAChwAAAwcAAAAAAAAAAAeAAAAA +ZgAAARcAAAKHAAADB5SMEnNodWZmbGVfc3RyYXRpZmllZJSIjAh1c2Vfcm9wZZSJjAtzY29yZV90 +YWJsZZR9lIwQc2hvd19zY29yZV9oaW50c5R9lCiMBk1vZGVsX5SIjAZUcmFpbl+UiYwFVGVzdF+U +iYwCQ0GUiIwXUHJlY2lzaW9uUmVjYWxsRlN1cHBvcnSUiIwLVGFyZ2V0U2NvcmWUiIwJUHJlY2lz +aW9ulIiMBlJlY2FsbJSIjAJGMZSIjANBVUOUiIwHTG9nTG9zc5SJjAtTcGVjaWZpY2l0eZSJjBdN +YXR0aGV3c0NvcnJDb2VmZmljaWVudJSIjANNU0WUiIwEUk1TRZSIjANNQUWUiIwETUFQRZSIjAJS +MpSIjAZDVlJNU0WUiYwPQ2x1c3RlcmluZ1Njb3JllIiMClNpbGhvdWV0dGWUiIwXQWRqdXN0ZWRN +dXR1YWxJbmZvU2NvcmWUiIwQQ29uY29yZGFuY2VJbmRleJSIdXOMC19fdmVyc2lvbl9flEsEjBBj +b250ZXh0X3NldHRpbmdzlF2UKIwVb3Jhbmdld2lkZ2V0LnNldHRpbmdzlIwHQ29udGV4dJSTlCmB +lH2UKIwGdmFsdWVzlH2UKIwPY2xhc3Nfc2VsZWN0aW9ulIwhKE5vbmUsIHNob3cgYXZlcmFnZSBv +dmVyIGNsYXNzZXMplEr/////hpSMDGZvbGRfZmVhdHVyZZROSv7///+GlIwVZm9sZF9mZWF0dXJl +X3NlbGVjdGVklIlK/v///4aUaA19lGgoSwR1jAphdHRyaWJ1dGVzlCiMDHNlcGFsIGxlbmd0aJRL +AoaUjAtzZXBhbCB3aWR0aJRLAoaUjAxwZXRhbCBsZW5ndGiUSwKGlIwLcGV0YWwgd2lkdGiUSwKG +lHSUjAVtZXRhc5QpjApjbGFzc192YXJzlIwEaXJpc5RLAYaUhZR1YmgtKYGUfZQoaDB9lChoMmgz +Sv////+GlGg1Tkr+////hpRoN4lK/v///4aUaA19lGgoSwR1aDoojANhZ2WUSwKGlIwGZ2VuZGVy +lEsBhpSMCmNoZXN0IHBhaW6USwGGlIwIcmVzdCBTQlCUSwKGlIwLY2hvbGVzdGVyb2yUSwKGlIwZ +ZmFzdGluZyBibG9vZCBzdWdhciA+IDEyMJRLAYaUjAhyZXN0IEVDR5RLAYaUjAZtYXggSFKUSwKG +lIwNZXhlcmMgaW5kIGFuZ5RLAYaUjA5TVCBieSBleGVyY2lzZZRLAoaUjBFzbG9wZSBwZWFrIGV4 +YyBTVJRLAYaUjBVtYWpvciB2ZXNzZWxzIGNvbG9yZWSUSwKGlIwEdGhhbJRLAYaUdJRoRCloRYwS +ZGlhbWV0ZXIgbmFycm93aW5nlEsBhpSFlHViaC0pgZR9lChoMH2UKGgyaDNK/////4aUaDVOSv7/ +//+GlGg3iUr+////hpRoDX2UaChLBHVoOiiMBENSSU2USwKGlIwCWk6USwKGlIwFSU5EVVOUSwKG +lIwEQ0hBU5RLAoaUjANOT1iUSwKGlIwCUk2USwKGlIwDQUdFlEsChpSMA0RJU5RLAoaUjANSQUSU +SwKGlIwDVEFYlEsChpSMB1BUUkFUSU+USwKGlIwBQpRLAoaUjAVMU1RBVJRLAoaUdJRoRCloRYwE +TUVEVpRLAoaUhZR1YmV1Lg== + + gASV6gEAAAAAAAB9lCiMEmFwcGVuZF9wcmVkaWN0aW9uc5SIjBRhcHBlbmRfcHJvYmFiaWxpdGll +c5SJjAphdXRvY29tbWl0lIiMEmNvbnRyb2xBcmVhVmlzaWJsZZSIjBNzYXZlZFdpZGdldEdlb21l +dHJ5lENCAdnQywADAAAAAAAnAAABsgAAAzQAAAMmAAAAJwAAAc4AAAM0AAADJgAAAAAAAAAABegA +AAAnAAABzgAAAzQAAAMmlIwQc2VsZWN0ZWRfbGVhcm5lcpSMB2NvcHlyZWeUjA5fcmVjb25zdHJ1 +Y3RvcpSTlIwIYnVpbHRpbnOUjARsaXN0lJOUaA0ph5RSlEsAYYwRc2VsZWN0ZWRfcXVhbnRpdHmU +SwCMC19fdmVyc2lvbl9flEsBjBBjb250ZXh0X3NldHRpbmdzlF2UjBVvcmFuZ2V3aWRnZXQuc2V0 +dGluZ3OUjAdDb250ZXh0lJOUKYGUfZQojAZ2YWx1ZXOUfZQojAlzZWxlY3Rpb26Uj5QoSwBLAYaU +SwBLAoaUSwFLAoaUSwJLAYaUSwFLAIaUSwJLAIaUkGgRSwF1jAdjbGFzc2VzlIwLSXJpcy1zZXRv +c2GUjA9JcmlzLXZlcnNpY29sb3KUjA5JcmlzLXZpcmdpbmljYZSHlHViYXUu + + {'auto_apply': True, 'controlAreaVisible': True, 'learner_name': '', 'savedWidgetGeometry': None, '__version__': 1} + gASVVAMAAAAAAAB9lCiMDWF0dHJfeF9pc19hYnOUiYwNYXR0cl95X2lzX2Fic5SJjAthdXRvX2Nv +bW1pdJSIjAthdXRvX3NhbXBsZZSIjBJjb250cm9sQXJlYVZpc2libGWUiIwTc2F2ZWRXaWRnZXRH +ZW9tZXRyeZRDQgHZ0MsAAwAAAAACYAAAAQwAAAVxAAADFAAAAmAAAAEoAAAFcQAAAxQAAAAAAAAA +AAXoAAACYAAAASgAAAVxAAADFJSMCXNlbGVjdGlvbpROjBF0b29sdGlwX3Nob3dzX2FsbJSIjA92 +aXN1YWxfc2V0dGluZ3OUfZSMBWdyYXBolH2UKIwLYWxwaGFfdmFsdWWUS4CMDWNsYXNzX2RlbnNp +dHmUiYwRaml0dGVyX2NvbnRpbnVvdXOUiYwLaml0dGVyX3NpemWUSwCME2xhYmVsX29ubHlfc2Vs +ZWN0ZWSUiYwWb3J0aG9ub3JtYWxfcmVncmVzc2lvbpSJjAtwb2ludF93aWR0aJRLCowMc2hvd19l +bGxpcHNllImMCXNob3dfZ3JpZJSJjAtzaG93X2xlZ2VuZJSIjA1zaG93X3JlZ19saW5llIl1jAtf +X3ZlcnNpb25fX5RLBYwQY29udGV4dF9zZXR0aW5nc5RdlIwVb3Jhbmdld2lkZ2V0LnNldHRpbmdz +lIwHQ29udGV4dJSTlCmBlH2UKIwGdmFsdWVzlH2UKIwKYXR0cl9jb2xvcpSMBGlyaXOUS2WGlIwK +YXR0cl9sYWJlbJROSv7///+GlIwKYXR0cl9zaGFwZZROSv7///+GlIwJYXR0cl9zaXpllE5K/v// +/4aUjAZhdHRyX3iUjAtwZXRhbCB3aWR0aJRLZoaUjAxhdHRyX3hfbG93ZXKUTkr+////hpSMDGF0 +dHJfeF91cHBlcpROSv7///+GlIwGYXR0cl95lIwMcGV0YWwgbGVuZ3RolEtmhpSMDGF0dHJfeV9s +b3dlcpROSv7///+GlIwMYXR0cl95X3VwcGVylE5K/v///4aUaAx9lGgZSwV1jAphdHRyaWJ1dGVz +lH2UKIwMc2VwYWwgbGVuZ3RolEsCjAtzZXBhbCB3aWR0aJRLAmg0SwJoLUsCaCRLAXWMBW1ldGFz +lH2UdWJhdS4= + + + + + + diff --git a/source/widgets/evaluate/workflows/parameterfitter.ows b/source/widgets/evaluate/workflows/parameterfitter.ows new file mode 100644 index 0000000..ef333ee --- /dev/null +++ b/source/widgets/evaluate/workflows/parameterfitter.ows @@ -0,0 +1,76 @@ + + + + + + + + + + + + + + + {'auto_commit': True, 'controlAreaVisible': True, 'manual_steps': '', 'maximum': 10, 'minimum': 1, 'parameter_index': 0, 'savedWidgetGeometry': b'\x01\xd9\xd0\xcb\x00\x03\x00\x00\x00\x00\x00\xe7\x00\x00\x01?\x00\x00\x04W\x00\x00\x03O\x00\x00\x00\xe7\x00\x00\x01[\x00\x00\x04W\x00\x00\x03O\x00\x00\x00\x00\x00\x00\x00\x00\x05\xe8\x00\x00\x00\xe7\x00\x00\x01[\x00\x00\x04W\x00\x00\x03O', 'type': 0, 'visual_settings': {}, '__version__': 1} + gASVzwsAAAAAAAB9lCiMEmNvbnRyb2xBcmVhVmlzaWJsZZSIjAxyZWNlbnRfcGF0aHOUXZQojB5v +cmFuZ2V3aWRnZXQudXRpbHMuZmlsZWRpYWxvZ3OUjApSZWNlbnRQYXRolJOUKYGUfZQojAdhYnNw +YXRolIw8L1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL2hlYXJ0X2Rp +c2Vhc2UudGFilIwGcHJlZml4lIwPc2FtcGxlLWRhdGFzZXRzlIwHcmVscGF0aJSMEWhlYXJ0X2Rp +c2Vhc2UudGFilIwFdGl0bGWUjACUjAVzaGVldJRoEIwLZmlsZV9mb3JtYXSUTnViaAYpgZR9lCho +CYw2L1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL2hvdXNpbmcudGFi +lGgLaAxoDYwLaG91c2luZy50YWKUaA9oEGgRaBBoEk51YmgGKYGUfZQoaAmMMy9Vc2Vycy9hamRh +L29yYW5nZS9vcmFuZ2UzL09yYW5nZS9kYXRhc2V0cy9pcmlzLnRhYpRoC2gMaA2MCGlyaXMudGFi +lGgPaBBoEWgQaBJOdWJoBimBlH2UKGgJjDYvVXNlcnMvYWpkYS9vcmFuZ2Uvb3JhbmdlMy9PcmFu +Z2UvZGF0YXNldHMvdGl0YW5pYy50YWKUaAtoDGgNjAt0aXRhbmljLnRhYpRoD2gQaBFoEGgSTnVi +aAYpgZR9lChoCYw9L1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL2Jy +b3duLXNlbGVjdGVkLnRhYpRoC2gMaA2MEmJyb3duLXNlbGVjdGVkLnRhYpRoD2gQaBFoEGgSTnVi +aAYpgZR9lChoCYwyL1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL3pv +by50YWKUaAtoDGgNjAd6b28udGFilGgPaBBoEWgQaBJOdWJljAtyZWNlbnRfdXJsc5RdlIwTc2F2 +ZWRXaWRnZXRHZW9tZXRyeZRDQgHZ0MsAAwAAAAACigAAAFAAAAThAAADCAAAAooAAABsAAAE4QAA +AwgAAAAAAAAAAAXoAAACigAAAGwAAAThAAADCJSMC3NoZWV0X25hbWVzlH2UjAZzb3VyY2WUSwCM +A3VybJRoEIwNZG9tYWluX2VkaXRvcpR9lIwLX192ZXJzaW9uX1+USwGMEGNvbnRleHRfc2V0dGlu +Z3OUXZQojBVvcmFuZ2V3aWRnZXQuc2V0dGluZ3OUjAdDb250ZXh0lJOUKYGUfZQojAZ2YWx1ZXOU +fZQojAl2YXJpYWJsZXOUXZRoL32UaDtdlChdlCiMA2FnZZSMFE9yYW5nZS5kYXRhLnZhcmlhYmxl +lIwSQ29udGludW91c1ZhcmlhYmxllJOUSwBoEIhlXZQojAZnZW5kZXKUaEGMEERpc2NyZXRlVmFy +aWFibGWUk5RLAIwMZmVtYWxlLCBtYWxllIllXZQojApjaGVzdCBwYWlulGhHSwCMNGFzeW1wdG9t +YXRpYywgYXR5cGljYWwgYW5nLCBub24tYW5naW5hbCwgdHlwaWNhbCBhbmeUiWVdlCiMCHJlc3Qg +U0JQlGhDSwBoEIhlXZQojAtjaG9sZXN0ZXJvbJRoQ0sAaBCIZV2UKIwZZmFzdGluZyBibG9vZCBz +dWdhciA+IDEyMJRoR0sAjAQwLCAxlIhlXZQojAhyZXN0IEVDR5RoR0sAjCxub3JtYWwsIGxlZnQg +dmVudCBoeXBlcnRyb3BoeSwgU1QtVCBhYm5vcm1hbJSJZV2UKIwGbWF4IEhSlGhDSwBoEIhlXZQo +jA1leGVyYyBpbmQgYW5nlGhHSwCMBDAsIDGUiGVdlCiMDlNUIGJ5IGV4ZXJjaXNllGhDSwBoEIhl +XZQojBFzbG9wZSBwZWFrIGV4YyBTVJRoR0sAjBx1cHNsb3BpbmcsIGZsYXQsIGRvd25zbG9waW5n +lIllXZQojBVtYWpvciB2ZXNzZWxzIGNvbG9yZWSUaENLAGgQiGVdlCiMBHRoYWyUaEdLAIwnbm9y +bWFsLCByZXZlcnNhYmxlIGRlZmVjdCwgZml4ZWQgZGVmZWN0lIllXZQojBJkaWFtZXRlciBuYXJy +b3dpbmeUaEdLAYwEMCwgMZSIZWVzaDFLAXWMCmF0dHJpYnV0ZXOUKIwDYWdllEsChpSMBmdlbmRl +cpRdlCiMBmZlbWFsZZSMBG1hbGWUZYaUjApjaGVzdCBwYWlulF2UKIwMYXN5bXB0b21hdGljlIwM +YXR5cGljYWwgYW5nlIwLbm9uLWFuZ2luYWyUjAt0eXBpY2FsIGFuZ5RlhpSMCHJlc3QgU0JQlEsC +hpSMC2Nob2xlc3Rlcm9slEsChpSMGWZhc3RpbmcgYmxvb2Qgc3VnYXIgPiAxMjCUXZQojAEwlIwB +MZRlhpSMCHJlc3QgRUNHlF2UKIwGbm9ybWFslIwVbGVmdCB2ZW50IGh5cGVydHJvcGh5lIwNU1Qt +VCBhYm5vcm1hbJRlhpSMBm1heCBIUpRLAoaUjA1leGVyYyBpbmQgYW5nlF2UKGh9aH5lhpSMDlNU +IGJ5IGV4ZXJjaXNllEsChpSMEXNsb3BlIHBlYWsgZXhjIFNUlF2UKIwJdXBzbG9waW5nlIwEZmxh +dJSMC2Rvd25zbG9waW5nlGWGlIwVbWFqb3IgdmVzc2VscyBjb2xvcmVklEsChpSMBHRoYWyUXZQo +jAZub3JtYWyUjBFyZXZlcnNhYmxlIGRlZmVjdJSMDGZpeGVkIGRlZmVjdJRlhpR0lIwFbWV0YXOU +KYwKY2xhc3NfdmFyc5SMEmRpYW1ldGVyIG5hcnJvd2luZ5RdlChofWh+ZYaUhZSMEm1vZGlmaWVk +X3ZhcmlhYmxlc5RdlHViaDYpgZR9lChoOX2UKGg7XZRoL32UaDtdlChdlCiMBENSSU2UaENLAGgQ +iGVdlCiMAlpOlGhDSwBoEIhlXZQojAVJTkRVU5RoQ0sAaBCIZV2UKIwEQ0hBU5RoQ0sAaBCIZV2U +KIwDTk9YlGhDSwBoEIhlXZQojAJSTZRoQ0sAaBCIZV2UKIwDQUdFlGhDSwBoEIhlXZQojANESVOU +aENLAGgQiGVdlCiMA1JBRJRoQ0sAaBCIZV2UKIwDVEFYlGhDSwBoEIhlXZQojAdQVFJBVElPlGhD +SwBoEIhlXZQojAFClGhDSwBoEIhlXZQojAVMU1RBVJRoQ0sAaBCIZV2UKIwETUVEVpRoQ0sBaBCI +ZWVzaDFLAXVoaCiMBENSSU2USwKGlIwCWk6USwKGlIwFSU5EVVOUSwKGlIwEQ0hBU5RLAoaUjANO +T1iUSwKGlIwCUk2USwKGlIwDQUdFlEsChpSMA0RJU5RLAoaUjANSQUSUSwKGlIwDVEFYlEsChpSM +B1BUUkFUSU+USwKGlGjBSwKGlIwFTFNUQVSUSwKGlHSUaJwpaJ2MBE1FRFaUSwKGlIWUaKJdlHVi +aDYpgZR9lChoOX2UKGg7XZRoL32UaDtdlChdlCiMDHNlcGFsIGxlbmd0aJRoQ0sAaBCIZV2UKIwL +c2VwYWwgd2lkdGiUaENLAGgQiGVdlCiMDHBldGFsIGxlbmd0aJRoQ0sAaBCIZV2UKIwLcGV0YWwg +d2lkdGiUaENLAGgQiGVdlCiMBGlyaXOUaEdLAYwsSXJpcy1zZXRvc2EsIElyaXMtdmVyc2ljb2xv +ciwgSXJpcy12aXJnaW5pY2GUiWVlc2gxSwF1aGgojAxzZXBhbCBsZW5ndGiUSwKGlIwLc2VwYWwg +d2lkdGiUSwKGlIwMcGV0YWwgbGVuZ3RolEsChpSMC3BldGFsIHdpZHRolEsChpR0lGicKWidjARp +cmlzlF2UKIwLSXJpcy1zZXRvc2GUjA9JcmlzLXZlcnNpY29sb3KUjA5JcmlzLXZpcmdpbmljYZRl +hpSFlGiiXZR1YmV1Lg== + + {'auto_apply': True, 'class_weight': False, 'controlAreaVisible': True, 'index_output': 0, 'learner_name': '', 'max_depth': 3, 'max_features': 5, 'min_samples_split': 5, 'n_estimators': 10, 'savedWidgetGeometry': None, 'use_max_depth': False, 'use_max_features': False, 'use_min_samples_split': True, 'use_random_state': False, '__version__': 1} + + + + + diff --git a/source/widgets/evaluate/workflows/predictions-example1.ows b/source/widgets/evaluate/workflows/predictions-example1.ows new file mode 100644 index 0000000..68b402f --- /dev/null +++ b/source/widgets/evaluate/workflows/predictions-example1.ows @@ -0,0 +1,38 @@ + + + + + + + + + + + + + + + + + {'controlAreaVisible': True, 'domain': None, 'filter_hint': 'attr', 'header_state': b'\x00\x00\x00\xff\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\xe2\x00\x00\x00\x07\x01\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00d\xff\xff\xff\xff\x00\x00\x00\x81\x00\x00\x00\x00\x00\x00\x00\x07\x00\x00\x00(\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x01C\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00[\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00[\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00K\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00d\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x01\x12\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03\xe8\x00\x00\x00\x00d', 'language': 'English', 'savedWidgetGeometry': b'\x01\xd9\xd0\xcb\x00\x03\x00\x00\x00\x00\x00\xf2\x00\x00\x00\xdc\x00\x00\x04\xe1\x00\x00\x02\xeb\x00\x00\x00\xf2\x00\x00\x00\xf8\x00\x00\x04\xe1\x00\x00\x02\xeb\x00\x00\x00\x00\x00\x00\x00\x00\x05\xe8\x00\x00\x00\xf2\x00\x00\x00\xf8\x00\x00\x04\xe1\x00\x00\x02\xeb', 'selected_id': 'attrition-train.tab', 'splitter_state': b'\x00\x00\x00\xff\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x01,\x00\x00\x00\xc8\x01\xff\xff\xff\xff\x01\x00\x00\x00\x02\x00', '__version__': 2} + {'auto_apply': True, 'binary_trees': True, 'controlAreaVisible': True, 'learner_name': '', 'limit_depth': True, 'limit_majority': True, 'limit_min_internal': True, 'limit_min_leaf': True, 'max_depth': 100, 'min_internal': 5, 'min_leaf': 2, 'savedWidgetGeometry': None, 'sufficient_majority': 95, '__version__': 1} + gASVuAIAAAAAAAB9lCiMEmNvbnRyb2xBcmVhVmlzaWJsZZSIjBNzYXZlZFdpZGdldEdlb21ldHJ5 +lENCAdnQywADAAAAAAHpAAABOgAABKEAAAJUAAAB6QAAAVYAAAShAAACVAAAAAAAAAAABegAAAHp +AAABVgAABKEAAAJUlIwJc2VsZWN0aW9ulF2UjAtzaG93X3Njb3Jlc5SIjAtzY29yZV90YWJsZZR9 +lIwQc2hvd19zY29yZV9oaW50c5R9lCiMBk1vZGVsX5SIjAZUcmFpbl+UiYwFVGVzdF+UiYwCQ0GU +iIwXUHJlY2lzaW9uUmVjYWxsRlN1cHBvcnSUiIwLVGFyZ2V0U2NvcmWUiIwJUHJlY2lzaW9ulIiM +BlJlY2FsbJSIjAJGMZSIjANBVUOUiIwHTG9nTG9zc5SJjAtTcGVjaWZpY2l0eZSJjBdNYXR0aGV3 +c0NvcnJDb2VmZmljaWVudJSIjANNU0WUiIwEUk1TRZSIjANNQUWUiIwETUFQRZSIjAJSMpSIjAZD +VlJNU0WUiYwPQ2x1c3RlcmluZ1Njb3JllIiMClNpbGhvdWV0dGWUiIwXQWRqdXN0ZWRNdXR1YWxJ +bmZvU2NvcmWUiIwQQ29uY29yZGFuY2VJbmRleJSIdXOMC19fdmVyc2lvbl9flEsCjBBjb250ZXh0 +X3NldHRpbmdzlF2UjBVvcmFuZ2V3aWRnZXQuc2V0dGluZ3OUjAdDb250ZXh0lJOUKYGUfZQojAZ2 +YWx1ZXOUfZQojBdzaG93X3Byb2JhYmlsaXR5X2Vycm9yc5SIjA9zaG93X3JlZ19lcnJvcnOUSwGM +C3Nob3duX3Byb2JzlEsCjAx0YXJnZXRfY2xhc3OUjBYoQXZlcmFnZSBvdmVyIGNsYXNzZXMplGgH +fZRoIksCdYwHY2xhc3Nlc5QpdWJhdS4= + + {'controlAreaVisible': True, 'domain': None, 'filter_hint': 'att', 'header_state': b'\x00\x00\x00\xff\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02I\x00\x00\x00\x07\x01\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00d\xff\xff\xff\xff\x00\x00\x00\x81\x00\x00\x00\x00\x00\x00\x00\x07\x00\x00\x00(\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00~\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00N\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00H\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00E\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00d\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00d\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03\xe8\x00\x00\x00\x00d', 'language': 'English', 'savedWidgetGeometry': b'\x01\xd9\xd0\xcb\x00\x03\x00\x00\x00\x00\x00c\x00\x00\x01\xaa\x00\x00\x02T\x00\x00\x02\xf4\x00\x00\x00c\x00\x00\x01\xc6\x00\x00\x02T\x00\x00\x02\xf4\x00\x00\x00\x00\x00\x00\x00\x00\x05\xe8\x00\x00\x00c\x00\x00\x01\xc6\x00\x00\x02T\x00\x00\x02\xf4', 'selected_id': 'attrition-predict.tab', 'splitter_state': b'\x00\x00\x00\xff\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x01,\x00\x00\x00\xc8\x01\xff\xff\xff\xff\x01\x00\x00\x00\x02\x00', '__version__': 2} + + + + + diff --git a/source/widgets/evaluate/workflows/predictions-example2.ows b/source/widgets/evaluate/workflows/predictions-example2.ows new file mode 100644 index 0000000..286f360 --- /dev/null +++ b/source/widgets/evaluate/workflows/predictions-example2.ows @@ -0,0 +1,85 @@ + + + + + + + + + + + + + + + + + + + + gASVZAkAAAAAAAB9lCiMEmNvbnRyb2xBcmVhVmlzaWJsZZSIjAxyZWNlbnRfcGF0aHOUXZQojB5v +cmFuZ2V3aWRnZXQudXRpbHMuZmlsZWRpYWxvZ3OUjApSZWNlbnRQYXRolJOUKYGUfZQojAdhYnNw +YXRolIw8L1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL2hlYXJ0X2Rp +c2Vhc2UudGFilIwGcHJlZml4lIwPc2FtcGxlLWRhdGFzZXRzlIwHcmVscGF0aJSMEWhlYXJ0X2Rp +c2Vhc2UudGFilIwFdGl0bGWUjACUjAVzaGVldJRoEIwLZmlsZV9mb3JtYXSUTnViaAYpgZR9lCho +CYwzL1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL2lyaXMudGFilGgL +aAxoDYwIaXJpcy50YWKUaA9oEGgRaBBoEk51YmgGKYGUfZQoaAmMNi9Vc2Vycy9hamRhL29yYW5n +ZS9vcmFuZ2UzL09yYW5nZS9kYXRhc2V0cy90aXRhbmljLnRhYpRoC2gMaA2MC3RpdGFuaWMudGFi +lGgPaBBoEWgQaBJOdWJoBimBlH2UKGgJjDYvVXNlcnMvYWpkYS9vcmFuZ2Uvb3JhbmdlMy9PcmFu +Z2UvZGF0YXNldHMvaG91c2luZy50YWKUaAtoDGgNjAtob3VzaW5nLnRhYpRoD2gQaBFoEGgSTnVi +aAYpgZR9lChoCYw9L1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL2Jy +b3duLXNlbGVjdGVkLnRhYpRoC2gMaA2MEmJyb3duLXNlbGVjdGVkLnRhYpRoD2gQaBFoEGgSTnVi +aAYpgZR9lChoCYwyL1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL3pv +by50YWKUaAtoDGgNjAd6b28udGFilGgPaBBoEWgQaBJOdWJljAtyZWNlbnRfdXJsc5RdlIwTc2F2 +ZWRXaWRnZXRHZW9tZXRyeZRDQgHZ0MsAAwAAAAACigAAAFAAAAThAAADCAAAAooAAABsAAAE4QAA +AwgAAAAAAAAAAAXoAAACigAAAGwAAAThAAADCJSMC3NoZWV0X25hbWVzlH2UjAZzb3VyY2WUSwCM +A3VybJRoEIwNZG9tYWluX2VkaXRvcpR9lIwLX192ZXJzaW9uX1+USwGMEGNvbnRleHRfc2V0dGlu +Z3OUXZQojBVvcmFuZ2V3aWRnZXQuc2V0dGluZ3OUjAdDb250ZXh0lJOUKYGUfZQojAZ2YWx1ZXOU +fZQojAl2YXJpYWJsZXOUXZRoL32UaDtdlChdlCiMA2FnZZSMFE9yYW5nZS5kYXRhLnZhcmlhYmxl +lIwSQ29udGludW91c1ZhcmlhYmxllJOUSwBoEIhlXZQojAZnZW5kZXKUaEGMEERpc2NyZXRlVmFy +aWFibGWUk5RLAIwMZmVtYWxlLCBtYWxllIllXZQojApjaGVzdCBwYWlulGhHSwCMNGFzeW1wdG9t +YXRpYywgYXR5cGljYWwgYW5nLCBub24tYW5naW5hbCwgdHlwaWNhbCBhbmeUiWVdlCiMCHJlc3Qg +U0JQlGhDSwBoEIhlXZQojAtjaG9sZXN0ZXJvbJRoQ0sAaBCIZV2UKIwZZmFzdGluZyBibG9vZCBz +dWdhciA+IDEyMJRoR0sAjAQwLCAxlIhlXZQojAhyZXN0IEVDR5RoR0sAjCxub3JtYWwsIGxlZnQg +dmVudCBoeXBlcnRyb3BoeSwgU1QtVCBhYm5vcm1hbJSJZV2UKIwGbWF4IEhSlGhDSwBoEIhlXZQo +jA1leGVyYyBpbmQgYW5nlGhHSwCMBDAsIDGUiGVdlCiMDlNUIGJ5IGV4ZXJjaXNllGhDSwBoEIhl +XZQojBFzbG9wZSBwZWFrIGV4YyBTVJRoR0sAjBx1cHNsb3BpbmcsIGZsYXQsIGRvd25zbG9waW5n +lIllXZQojBVtYWpvciB2ZXNzZWxzIGNvbG9yZWSUaENLAGgQiGVdlCiMBHRoYWyUaEdLAIwnbm9y +bWFsLCByZXZlcnNhYmxlIGRlZmVjdCwgZml4ZWQgZGVmZWN0lIllXZQojBJkaWFtZXRlciBuYXJy +b3dpbmeUaEdLAYwEMCwgMZSIZWVzaDFLAXWMCmF0dHJpYnV0ZXOUKGhASwKGlGhFXZQojAZmZW1h +bGWUjARtYWxllGWGlGhKXZQojAxhc3ltcHRvbWF0aWOUjAxhdHlwaWNhbCBhbmeUjAtub24tYW5n +aW5hbJSMC3R5cGljYWwgYW5nlGWGlGhNSwKGlGhPSwKGlGhRXZQojAEwlIwBMZRlhpRoVF2UKIwG +bm9ybWFslIwVbGVmdCB2ZW50IGh5cGVydHJvcGh5lIwNU1QtVCBhYm5vcm1hbJRlhpRoV0sChpRo +WV2UKGh3aHhlhpRoXEsChpRoXl2UKIwJdXBzbG9waW5nlIwEZmxhdJSMC2Rvd25zbG9waW5nlGWG +lGhhSwKGlGhjXZQojAZub3JtYWyUjBFyZXZlcnNhYmxlIGRlZmVjdJSMDGZpeGVkIGRlZmVjdJRl +hpR0lIwFbWV0YXOUKYwKY2xhc3NfdmFyc5RoZl2UKGh3aHhlhpSFlIwSbW9kaWZpZWRfdmFyaWFi +bGVzlF2UdWJoNimBlH2UKGg5fZQoaDtdlGgvfZRoO12UKF2UKIwMc2VwYWwgbGVuZ3RolGhDSwBo +EIhlXZQojAtzZXBhbCB3aWR0aJRoQ0sAaBCIZV2UKIwMcGV0YWwgbGVuZ3RolGhDSwBoEIhlXZQo +jAtwZXRhbCB3aWR0aJRoQ0sAaBCIZV2UKIwEaXJpc5RoR0sBjCxJcmlzLXNldG9zYSwgSXJpcy12 +ZXJzaWNvbG9yLCBJcmlzLXZpcmdpbmljYZSJZWVzaDFLAXVoaCiMDHNlcGFsIGxlbmd0aJRLAoaU +jAtzZXBhbCB3aWR0aJRLAoaUjAxwZXRhbCBsZW5ndGiUSwKGlIwLcGV0YWwgd2lkdGiUSwKGlHSU +aI8paJCMBGlyaXOUXZQojAtJcmlzLXNldG9zYZSMD0lyaXMtdmVyc2ljb2xvcpSMDklyaXMtdmly +Z2luaWNhlGWGlIWUaJRdlHViZXUu + + {'compatibility_mode': False, 'controlAreaVisible': True, 'number_of_folds': 10, 'replacement': False, 'sampleSizeNumber': 1, 'sampleSizePercentage': 70, 'sampleSizeSqlPercentage': 0.1, 'sampleSizeSqlTime': 1, 'sampling_type': 0, 'savedWidgetGeometry': b"\x01\xd9\xd0\xcb\x00\x03\x00\x00\x00\x00\x00{\x00\x00\x01^\x00\x00\x01|\x00\x00\x03'\x00\x00\x02i\x00\x00\x01\x1b\x00\x00\x03j\x00\x00\x02\xc8\x00\x00\x00\x00\x02\x00\x00\x00\x05\xe8\x00\x00\x00{\x00\x00\x01z\x00\x00\x01|\x00\x00\x03'", 'selectedFold': 1, 'sql_dl': False, 'stratify': False, 'use_seed': True, '__version__': 2} + {'autocommit': True, 'controlAreaVisible': True, 'savedWidgetGeometry': b'\x01\xd9\xd0\xcb\x00\x03\x00\x00\x00\x00\x01\x95\x00\x00\x01K\x00\x00\x03\xdb\x00\x00\x02O\x00\x00\x01\x95\x00\x00\x01g\x00\x00\x03\xdb\x00\x00\x02O\x00\x00\x00\x00\x00\x00\x00\x00\x05\xe8\x00\x00\x01\x95\x00\x00\x01g\x00\x00\x03\xdb\x00\x00\x02O', 'storedsettings': {'name': '', 'preprocessors': [('orange.preprocess.impute', {})]}, '__version__': 2} + {'C_index': 61, 'auto_apply': True, 'class_weight': False, 'controlAreaVisible': True, 'learner_name': '', 'penalty_type': 1, 'savedWidgetGeometry': b'\x01\xd9\xd0\xcb\x00\x03\x00\x00\x00\x00\x03/\x00\x00\x01i\x00\x00\x04=\x00\x00\x02\xb2\x00\x00\x03/\x00\x00\x01\x85\x00\x00\x04=\x00\x00\x02\xb2\x00\x00\x00\x00\x02\x00\x00\x00\x07\x80\x00\x00\x03/\x00\x00\x01\x85\x00\x00\x04=\x00\x00\x02\xb2', '__version__': 2} + gASVwQIAAAAAAAB9lCiMEmNvbnRyb2xBcmVhVmlzaWJsZZSIjBNzYXZlZFdpZGdldEdlb21ldHJ5 +lENCAdnQywADAAAAAAGKAAACWAAABHsAAAONAAABigAAAnQAAAR7AAADjQAAAAAAAAAABegAAAGK +AAACdAAABHsAAAONlIwJc2VsZWN0aW9ulF2UjAtzaG93X3Njb3Jlc5SIjAtzY29yZV90YWJsZZR9 +lIwQc2hvd19zY29yZV9oaW50c5R9lCiMBk1vZGVsX5SIjAZUcmFpbl+UiYwFVGVzdF+UiYwCQ0GU +iIwXUHJlY2lzaW9uUmVjYWxsRlN1cHBvcnSUiIwLVGFyZ2V0U2NvcmWUiIwJUHJlY2lzaW9ulIiM +BlJlY2FsbJSIjAJGMZSIjANBVUOUiIwHTG9nTG9zc5SJjAtTcGVjaWZpY2l0eZSJjBdNYXR0aGV3 +c0NvcnJDb2VmZmljaWVudJSIjANNU0WUiIwEUk1TRZSIjANNQUWUiIwETUFQRZSIjAJSMpSIjAZD +VlJNU0WUiYwPQ2x1c3RlcmluZ1Njb3JllIiMClNpbGhvdWV0dGWUiIwXQWRqdXN0ZWRNdXR1YWxJ +bmZvU2NvcmWUiIwQQ29uY29yZGFuY2VJbmRleJSIdXOMC19fdmVyc2lvbl9flEsCjBBjb250ZXh0 +X3NldHRpbmdzlF2UjBVvcmFuZ2V3aWRnZXQuc2V0dGluZ3OUjAdDb250ZXh0lJOUKYGUfZQojAZ2 +YWx1ZXOUfZQojBdzaG93X3Byb2JhYmlsaXR5X2Vycm9yc5SIjA9zaG93X3JlZ19lcnJvcnOUSwGM +C3Nob3duX3Byb2JzlEsBjAx0YXJnZXRfY2xhc3OUjBYoQXZlcmFnZSBvdmVyIGNsYXNzZXMplGgH +fZRoIksCdYwHY2xhc3Nlc5SMATCUjAExlIaUdWJhdS4= + + + + + + diff --git a/source/widgets/evaluate/workflows/testandscore.ows b/source/widgets/evaluate/workflows/testandscore.ows new file mode 100644 index 0000000..7fe9606 --- /dev/null +++ b/source/widgets/evaluate/workflows/testandscore.ows @@ -0,0 +1,121 @@ + + + + + + + + + + + + + + + + + + + + + + + + + gASVNQgAAAAAAAB9lCiMEmNvbnRyb2xBcmVhVmlzaWJsZZSIjAxyZWNlbnRfcGF0aHOUXZQojB5v +cmFuZ2V3aWRnZXQudXRpbHMuZmlsZWRpYWxvZ3OUjApSZWNlbnRQYXRolJOUKYGUfZQojAdhYnNw +YXRolIw2L1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL3RpdGFuaWMu +dGFilIwGcHJlZml4lIwPc2FtcGxlLWRhdGFzZXRzlIwHcmVscGF0aJSMC3RpdGFuaWMudGFilIwF +dGl0bGWUjACUjAVzaGVldJRoEIwLZmlsZV9mb3JtYXSUTnViaAYpgZR9lChoCYw2L1VzZXJzL2Fq +ZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL2hvdXNpbmcudGFilGgLaAxoDYwLaG91 +c2luZy50YWKUaA9oEGgRaBBoEk51YmgGKYGUfZQoaAmMMy9Vc2Vycy9hamRhL29yYW5nZS9vcmFu +Z2UzL09yYW5nZS9kYXRhc2V0cy9pcmlzLnRhYpRoC2gMaA2MCGlyaXMudGFilGgPaBBoEWgQaBJO +dWJoBimBlH2UKGgJjDwvVXNlcnMvYWpkYS9vcmFuZ2Uvb3JhbmdlMy9PcmFuZ2UvZGF0YXNldHMv +aGVhcnRfZGlzZWFzZS50YWKUaAtoDGgNjBFoZWFydF9kaXNlYXNlLnRhYpRoD2gQaBFoEGgSTnVi +aAYpgZR9lChoCYw9L1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL2Jy +b3duLXNlbGVjdGVkLnRhYpRoC2gMaA2MEmJyb3duLXNlbGVjdGVkLnRhYpRoD2gQaBFoEGgSTnVi +aAYpgZR9lChoCYwyL1VzZXJzL2FqZGEvb3JhbmdlL29yYW5nZTMvT3JhbmdlL2RhdGFzZXRzL3pv +by50YWKUaAtoDGgNjAd6b28udGFilGgPaBBoEWgQaBJOdWJljAtyZWNlbnRfdXJsc5RdlIwTc2F2 +ZWRXaWRnZXRHZW9tZXRyeZRDQgHZ0MsAAwAAAAACigAAALIAAAThAAADagAAAooAAADOAAAE4QAA +A2oAAAAAAAAAAAeAAAACigAAAM4AAAThAAADapSMC3NoZWV0X25hbWVzlH2UjAZzb3VyY2WUSwCM +A3VybJRoEIwNZG9tYWluX2VkaXRvcpR9lIwLX192ZXJzaW9uX1+USwGMEGNvbnRleHRfc2V0dGlu +Z3OUXZQojBVvcmFuZ2V3aWRnZXQuc2V0dGluZ3OUjAdDb250ZXh0lJOUKYGUfZQojAZ2YWx1ZXOU +fZQojAl2YXJpYWJsZXOUXZRoL32UaDtdlChdlCiMBnN0YXR1c5SMFE9yYW5nZS5kYXRhLnZhcmlh +YmxllIwQRGlzY3JldGVWYXJpYWJsZZSTlEsAjBpjcmV3LCBmaXJzdCwgc2Vjb25kLCB0aGlyZJSJ +ZV2UKIwDYWdllGhDSwCMDGFkdWx0LCBjaGlsZJSJZV2UKIwDc2V4lGhDSwCMDGZlbWFsZSwgbWFs +ZZSJZV2UKIwIc3Vydml2ZWSUaENLAYwHbm8sIHllc5SJZWVzaDFLAXWMCmF0dHJpYnV0ZXOUaEBd +lCiMBGNyZXeUjAVmaXJzdJSMBnNlY29uZJSMBXRoaXJklGWGlGhGXZQojAVhZHVsdJSMBWNoaWxk +lGWGlGhJXZQojAZmZW1hbGWUjARtYWxllGWGlIeUjAVtZXRhc5QpjApjbGFzc192YXJzlGhMXZQo +jAJub5SMA3llc5RlhpSFlIwSbW9kaWZpZWRfdmFyaWFibGVzlF2UdWJoNimBlH2UKGg5fZQoaDtd +lGgvfZRoO12UKF2UKIwEQ1JJTZRoQYwSQ29udGludW91c1ZhcmlhYmxllJOUSwBoEIhlXZQojAJa +TpRocEsAaBCIZV2UKIwFSU5EVVOUaHBLAGgQiGVdlCiMBENIQVOUaHBLAGgQiGVdlCiMA05PWJRo +cEsAaBCIZV2UKIwCUk2UaHBLAGgQiGVdlCiMA0FHRZRocEsAaBCIZV2UKIwDRElTlGhwSwBoEIhl +XZQojANSQUSUaHBLAGgQiGVdlCiMA1RBWJRocEsAaBCIZV2UKIwHUFRSQVRJT5RocEsAaBCIZV2U +KIwBQpRocEsAaBCIZV2UKIwFTFNUQVSUaHBLAGgQiGVdlCiMBE1FRFaUaHBLAWgQiGVlc2gxSwF1 +aE4oaG5LAoaUaHJLAoaUaHRLAoaUaHZLAoaUaHhLAoaUaHpLAoaUaHxLAoaUaH5LAoaUaIBLAoaU +aIJLAoaUaIRLAoaUaIZLAoaUaIhLAoaUdJRoXiloX2iKSwKGlIWUaGVdlHViaDYpgZR9lChoOX2U +KGg7XZRoL32UaDtdlChdlCiMDHNlcGFsIGxlbmd0aJRocEsAaBCIZV2UKIwLc2VwYWwgd2lkdGiU +aHBLAGgQiGVdlCiMDHBldGFsIGxlbmd0aJRocEsAaBCIZV2UKIwLcGV0YWwgd2lkdGiUaHBLAGgQ +iGVdlCiMBGlyaXOUaENLAYwsSXJpcy1zZXRvc2EsIElyaXMtdmVyc2ljb2xvciwgSXJpcy12aXJn +aW5pY2GUiWVlc2gxSwF1aE4oaKNLAoaUaKVLAoaUaKdLAoaUaKlLAoaUdJRoXiloX2irXZQojAtJ +cmlzLXNldG9zYZSMD0lyaXMtdmVyc2ljb2xvcpSMDklyaXMtdmlyZ2luaWNhlGWGlIWUaGVdlHVi +ZXUu + + gASVYAUAAAAAAAB9lCiMFGNvbXBhcmlzb25fY3JpdGVyaW9ulEsAjBJjb250cm9sQXJlYVZpc2li +bGWUiYwNY3Zfc3RyYXRpZmllZJSIjAduX2ZvbGRzlEsCjAluX3JlcGVhdHOUSwOMCnJlc2FtcGxp +bmeUSwCMBHJvcGWURz+5mZmZmZmajAtzYW1wbGVfc2l6ZZRLCYwTc2F2ZWRXaWRnZXRHZW9tZXRy +eZRDQgHZ0MsAAwAAAAAAZgAAAPsAAAKHAAADBwAAAGYAAAEXAAAChwAAAwcAAAAAAAAAAAeAAAAA +ZgAAARcAAAKHAAADB5SMEnNodWZmbGVfc3RyYXRpZmllZJSIjAh1c2Vfcm9wZZSJjAtzY29yZV90 +YWJsZZR9lIwQc2hvd19zY29yZV9oaW50c5R9lCiMBk1vZGVsX5SIjAZUcmFpbl+UiYwFVGVzdF+U +iYwCQ0GUiIwXUHJlY2lzaW9uUmVjYWxsRlN1cHBvcnSUiIwLVGFyZ2V0U2NvcmWUiIwJUHJlY2lz +aW9ulIiMBlJlY2FsbJSIjAJGMZSIjANBVUOUiIwHTG9nTG9zc5SJjAtTcGVjaWZpY2l0eZSJjBdN +YXR0aGV3c0NvcnJDb2VmZmljaWVudJSIjANNU0WUiIwEUk1TRZSIjANNQUWUiIwETUFQRZSIjAJS +MpSIjAZDVlJNU0WUiYwPQ2x1c3RlcmluZ1Njb3JllIiMClNpbGhvdWV0dGWUiIwXQWRqdXN0ZWRN +dXR1YWxJbmZvU2NvcmWUiIwQQ29uY29yZGFuY2VJbmRleJSIdXOMC19fdmVyc2lvbl9flEsEjBBj +b250ZXh0X3NldHRpbmdzlF2UKIwVb3Jhbmdld2lkZ2V0LnNldHRpbmdzlIwHQ29udGV4dJSTlCmB +lH2UKIwGdmFsdWVzlH2UKIwPY2xhc3Nfc2VsZWN0aW9ulIwhKE5vbmUsIHNob3cgYXZlcmFnZSBv +dmVyIGNsYXNzZXMplEr/////hpSMDGZvbGRfZmVhdHVyZZROSv7///+GlIwVZm9sZF9mZWF0dXJl +X3NlbGVjdGVklIlK/v///4aUaA19lGgoSwR1jAphdHRyaWJ1dGVzlIwGc3RhdHVzlEsBhpSMA3Nl +eJRLAYaUhpSMBW1ldGFzlCmMCmNsYXNzX3ZhcnOUjAhzdXJ2aXZlZJRLAYaUhZR1YmgtKYGUfZQo +aDB9lChoMmgzSv////+GlGg1Tkr+////hpRoN4lK/v///4aUaA19lGgoSwR1aDqMBnN0YXR1c5RL +AYaUjANhZ2WUSwGGlIwDc2V4lEsBhpSHlGhAKWhBjAhzdXJ2aXZlZJRLAYaUhZR1YmgtKYGUfZQo +aDB9lChoMmgzSv////+GlGg1Tkr+////hpRoN4lK/v///4aUaA19lGgoSwR1aDoojARDUklNlEsC +hpSMAlpOlEsChpSMBUlORFVTlEsChpSMBENIQVOUSwKGlIwDTk9YlEsChpSMAlJNlEsChpSMA0FH +RZRLAoaUjANESVOUSwKGlIwDUkFElEsChpSMA1RBWJRLAoaUjAdQVFJBVElPlEsChpSMAUKUSwKG +lIwFTFNUQVSUSwKGlHSUaEApaEGMBE1FRFaUSwKGlIWUdWJoLSmBlH2UKGgwfZQoaDJoM0r///// +hpRoNU5K/v///4aUaDeJSv7///+GlGgNfZRoKEsEdWg6KIwMc2VwYWwgbGVuZ3RolEsChpSMC3Nl +cGFsIHdpZHRolEsChpSMDHBldGFsIGxlbmd0aJRLAoaUjAtwZXRhbCB3aWR0aJRLAoaUdJRoQClo +QYwEaXJpc5RLAYaUhZR1YmV1Lg== + + gASVwQEAAAAAAAB9lCiMC2F1dG9fY29tbWl0lIiMEmNvbnRyb2xBcmVhVmlzaWJsZZSIjBNpZ25v +cmVfbmV3X2ZlYXR1cmVzlImME3NhdmVkV2lkZ2V0R2VvbWV0cnmUQ0IB2dDLAAMAAAAAAHwAAAEj +AAACngAAArcAAAB8AAABPwAAAp4AAAK3AAAAAAAAAAAHgAAAAHwAAAE/AAACngAAAreUjBJ1c2Vf +aW5wdXRfZmVhdHVyZXOUiYwLX192ZXJzaW9uX1+USwGMEGNvbnRleHRfc2V0dGluZ3OUXZSMFW9y +YW5nZXdpZGdldC5zZXR0aW5nc5SMB0NvbnRleHSUk5QpgZR9lCiMBnZhbHVlc5R9lCiMEWRvbWFp +bl9yb2xlX2hpbnRzlH2UKIwDYWdllEsBhpSMCWF2YWlsYWJsZZRLAIaUjAZzdGF0dXOUSwGGlIwJ +YXR0cmlidXRllEsAhpSMA3NleJRLAYaUaBlLAYaUjAhzdXJ2aXZlZJRLAYaUjAVjbGFzc5RLAIaU +dUr+////hpRoB0sBdYwKYXR0cmlidXRlc5R9lChoF0sBaBNLAWgbSwFoHksBdYwFbWV0YXOUfZR1 +YmF1Lg== + + {'auto_apply': True, 'controlAreaVisible': True, 'learner_name': '', 'savedWidgetGeometry': None, '__version__': 1} + {'auto_apply': True, 'binary_trees': True, 'controlAreaVisible': True, 'learner_name': '', 'limit_depth': True, 'limit_majority': True, 'limit_min_internal': True, 'limit_min_leaf': True, 'max_depth': 100, 'min_internal': 5, 'min_leaf': 2, 'savedWidgetGeometry': None, 'sufficient_majority': 95, '__version__': 1} + {'C_index': 61, 'auto_apply': True, 'class_weight': False, 'controlAreaVisible': True, 'learner_name': '', 'penalty_type': 1, 'savedWidgetGeometry': b'\x01\xd9\xd0\xcb\x00\x03\x00\x00\x00\x00\x03/\x00\x00\x01i\x00\x00\x04=\x00\x00\x02\xb2\x00\x00\x03/\x00\x00\x01\x85\x00\x00\x04=\x00\x00\x02\xb2\x00\x00\x00\x00\x02\x00\x00\x00\x07\x80\x00\x00\x03/\x00\x00\x01\x85\x00\x00\x04=\x00\x00\x02\xb2', '__version__': 2} + {'C': 1.0, 'auto_apply': True, 'coef0': 1.0, 'controlAreaVisible': True, 'degree': 3, 'epsilon': 0.1, 'gamma': 0.0, 'kernel_type': 2, 'learner_name': '', 'limit_iter': True, 'max_iter': 100, 'nu': 0.5, 'nu_C': 1.0, 'savedWidgetGeometry': None, 'svm_type': 0, 'tol': 0.001, '__version__': 2} + gASVbAIAAAAAAAB9lCiMEmNvbnRyb2xBcmVhVmlzaWJsZZSJjBRkaXNwbGF5X2NvbnZleF9jdXJ2 +ZZSJjBNkaXNwbGF5X2NvbnZleF9odWxslImMFWRpc3BsYXlfZGVmX3RocmVzaG9sZJSIjBFkaXNw +bGF5X3BlcmZfbGluZZSIjAdmbl9jb3N0lE30AYwHZnBfY29zdJRN9AGMDXJvY19hdmVyYWdpbmeU +SwCME3NhdmVkV2lkZ2V0R2VvbWV0cnmUQ0IB2dDLAAMAAAAAA1UAAAEFAAAFkAAAAxQAAANVAAAB +IQAABZAAAAMUAAAAAAAAAAAHgAAAA1UAAAEhAAAFkAAAAxSUjAx0YXJnZXRfcHJpb3KUjBVudW1w +eS5jb3JlLm11bHRpYXJyYXmUjAZzY2FsYXKUk5SMBW51bXB5lIwFZHR5cGWUk5SMAmY4lImIh5RS +lChLA4wBPJROTk5K/////0r/////SwB0lGJDCAAAAAAAAFFAlIaUUpSMC19fdmVyc2lvbl9flEsB +jBBjb250ZXh0X3NldHRpbmdzlF2UjBVvcmFuZ2V3aWRnZXQuc2V0dGluZ3OUjAdDb250ZXh0lJOU +KYGUfZQojAZ2YWx1ZXOUfZQojBRzZWxlY3RlZF9jbGFzc2lmaWVyc5RdlChLAEsBSwJLA2WMDHRh +cmdldF9pbmRleJRLAGgaSwF1jAdjbGFzc2VzlIwCbm+UjAN5ZXOUhpSMEGNsYXNzaWZpZXJfbmFt +ZXOUXZQojAtOYWl2ZSBCYXllc5SMBFRyZWWUjBNMb2dpc3RpYyBSZWdyZXNzaW9ulIwDU1ZNlGV1 +YmF1Lg== + + + + + +