Evaluate: Updated documentation

source/widgets/evaluate/confusionmatrix.md

@@ -12,39 +12,38 @@ Shows proportions between the predicted and actual class.
 - Selected Data: data subset selected from confusion matrix
 - Data: data with the additional information on whether a data instance was selected
 
-The [Confusion Matrix](https://en.wikipedia.org/wiki/Confusion_matrix) gives the number/proportion of instances between the predicted and actual class. The selection of the elements in the matrix feeds the corresponding instances into the output signal. This way, one can observe which specific instances were misclassified and how.
+The [Confusion Matrix](https://en.wikipedia.org/wiki/Confusion_matrix) gives the number/proportion of instances between the predicted and actual class. Each row corresponds to a correct class, while columns represent the predicted classes. The rightmost column gives the number of instances from each class and the bottom row gives the number of instances classified into each class.
 
-The widget usually gets the evaluation results from [Test & Score](../evaluate/testandscore.md); an example of the schema is shown below.
+The selection of the elements in the matrix feeds the corresponding instances into the output signal. This way, one can observe which specific instances were misclassified and how.
+
+The widget usually gets the evaluation results from [Test and Score](../evaluate/testandscore.md); an example of the schema is shown below.
 
 ![](images/ConfusionMatrix-stamped.png)
 
-1. When evaluation results contain data on multiple learning algorithms, we have to choose one in the *Learners* box.
-   The snapshot shows the confusion matrix for [Tree](../model/tree.md) and [Naive Bayesian](../model/naivebayes.md) models trained and tested on the *iris* data. The right-hand side of the widget contains the matrix for the naive Bayesian model (since this model is selected on the left). Each row corresponds to a correct class, while columns represent the predicted classes. For instance, four instances of *Iris-versicolor* were misclassified as *Iris-virginica*. The rightmost column gives the number of instances from each class (there are 50 irises of each of the three classes) and the bottom row gives the number of instances classified into each class (e.g., 48 instances were classified into virginica).
-2. In *Show*, we select what data we would like to see in the matrix.
+1. *Learners*: Choose a learning algorithm to display.
+2. *Output*: define what is sent to the output, namely predicted classes (*Predictions*) or their probabilities (*Probabilities*).
+3. The widget outputs every change if *Send Automatically* is ticked. If not, the user will need to click *Send Selected* to commit the changes.
+4. *Show*: select what data to see in the matrix.
    - **Number of instances** shows correctly and incorrectly classified instances numerically.
-   - **Proportions of predicted** shows how many instances classified as, say, *Iris-versicolor* are in which true class; in the table we can read the 0% of them are actually setosae, 88.5% of those classified as versicolor are versicolors, and 7.7% are virginicae.
-   - **Proportions of actual** shows the opposite relation: of all true versicolors, 92% were classified as versicolors and 8% as virginicae.
-   ![](images/ConfusionMatrix-propTrue.png)
-3. In *Select*, you can choose the desired output.
-   - **Correct** sends all correctly classified instances to the output by selecting the diagonal of the matrix.
-   - **Misclassified** selects the misclassified instances.
-   - **None** annuls the selection.
-   As mentioned before, one can also select individual cells of the table to select specific kinds of misclassified instances (e.g. the versicolors classified as virginicae).
-4. When sending selected instances, the widget can add new attributes, such as predicted classes or their probabilities, if the corresponding options *Predictions* and/or *Probabilities* are checked.
-5. The widget outputs every change if *Send Automatically* is ticked. If not, the user will need to click *Send Selected* to commit the changes.
-6. Produce a report.
+   - **Proportions of predicted** shows how many instances classified as, say, *Iris-versicolor* are in which true class; in the table we can read the 0% of them are actually setosae, 95.9% of those classified as versicolor are versicolors, and 4.1% are virginicae.
+   ![](images/ConfusionMatrix-propPred.png)
+   - **Proportions of actual** shows the opposite relation: of all true versicolors, 94% were classified as versicolors and 6% as virginicae.
+   ![](images/ConfusionMatrix-propAct.png)
+   - **Sum of probabilities** sums the probabilities for each class. For example, there are 50 iris-setosas. This option sums probabilities for all *actual* versicolors to belong to iris-setosa (1.4), to iris-versicolor (41.4), and to iris-virginica (7.2).
+   ![](images/ConfusionMatrix-sumProb.png)
+5. Select the desired output:
+   - **Select Correct** sends all correctly classified instances to the output by selecting the diagonal of the matrix.
+   - **Select Misclassified** selects the misclassified instances.
+   - **Clear Selection** annuls the selection.
+   As mentioned before, one can also select individual cells of the table to select specific kinds of misclassified instances.
 
 Example
 -------
 
-The following workflow demonstrates what this widget can be used for.
-
-![](images/ConfusionMatrix-Schema.png)
-
-[Test & Score](../evaluate/testandscore.md) gets the data from [File](../data/file.md) and two learning algorithms from [Naive Bayes](../model/naivebayes.md) and [Tree](../model/tree.md). It performs cross-validation or some other train-and-test procedures to get class predictions by both algorithms for all (or some) data instances. The test results are fed into the **Confusion Matrix**, where we can observe how many instances were misclassified and in which way.
+The following workflow demonstrates what this widget can be used for. We are using the *iris* dataset for demonstration.
 
-In the output, we used [Data Table](../data/datatable.md) to show the instances we selected in the confusion matrix. If we, for instance, click *Misclassified*, the table will contain all instances which were misclassified by the selected method.
+![](images/ConfusionMatrix-Example.png)
 
-The [Scatter Plot](../visualize/scatterplot.md) gets two sets of data. From the [File](../data/file.md) widget it gets the complete data, while the confusion matrix sends only the selected data, misclassifications for instance. The scatter plot will show all the data, with bold symbols representing the selected data.
+[Test and Score](../evaluate/testandscore.md) gets the data from [File](../data/file.md) and a learning algorithm, [Naive Bayes](../model/naivebayes.md). It performs cross-validation or some other train-and-test procedures to get class predictions for all (or some) data instances. The test results are fed into the **Confusion Matrix**, where we can observe how many instances were misclassified and in which way.
 
-![](images/ConfusionMatrix-Example.png)
+In the output, we used [Scatter Plot](../visualize/scatterplot.md), which gets two sets of data, the whole dataset from the File widget, and a subset from Confusion Matrix. Now, we wish to show where the misclassifications lie in the Scatter Plot. We click *Misclassified*, which selects all the off-diagonal (misclassified) instances from the confusion matrix. The scatter plot will show all the data in a graph, with bold symbols representing the selected (misclassified) data points.

source/widgets/evaluate/parameterfitter.md

@@ -8,11 +8,22 @@ Find the best hyper-parameters for a model.
 - Data: input data
 - Learner: learning algorithm
 
-Parameter fitter shows performance of a learning algorithms with different settings of a hyper-parameter. The widget is currently limited to a single integer parameter. Not all learning algorithms support hyper-parameter tuning.
+Parameter fitter shows performance of a learning algorithms with different settings of a hyper-parameter. The widget is currently limited to a single integer parameter. Not all learning algorithms support hyper-parameter tuning (currently, only [Random Forest](../model/randomforest.md) and [PLS](../model/pls.md)). The widget shows a plot of the model's performance at different values of the parameter. The graph shows AUC for classification problems and R2 for regression.
 
-![](images/ParameterFitter.png)
+![](images/ParameterFitter-stamped.png)
 
 1. Choose the parameter to fit.
-2. Define the lower and the upper limit; step size is determined automatically.
-3. Alternatively, specifies the values for the parameter. The widget also accepts `...`, e.g. `1, 2, 3, ..., 10` or `40, 60, ..., 100`. When the parameter has a minimal value (e.g. the number of components cannot be negative), one can also omit the lower bound, e.g. `..., 80, 100`; and if the parameter has a maximal value, one can omit the upper bound, e.g. `2, 4, 6, ...,`.
-4. A plot showing the performance at different values of the parameter. The graph shows AUC for classification problems and R2 for regression.
+   *Range*: Define the lower and the upper limit; step size is determined automatically.
+   *Manual*: Alternatively, specify the values for the parameter. The widget also accepts `...`, e.g. `1, 2, 3, ..., 10` or `40, 60, ..., 100`. When the parameter has a minimal value (e.g. the number of components cannot be negative), one can also omit the lower bound, e.g. `..., 80, 100`; and if the parameter has a maximal value, one can omit the upper bound, e.g. `2, 4, 6, ...,`.
+2. If *Apply Automatically* is ticked, changes are communicated automatically. Alternatively, click *Apply*.
+
+Example
+-------
+
+Here is a simple example on how to fit parameters using the **Parameter Fitter** widget. We are using the *heart-disease* data for this example and loading it with the [File](../data/file.md) widget. We pass the data to Parameter Fitter. The widget also needs a learner to fit, the [Random Forest](../model/randomforest.md) in this case.
+
+Parameter Fitter enables observing performance for a varying number of trees. We set the range from 1 to 10, namely we will observe performance for every number of trees up to 10.
+
+We see there's a slight peak in AUC value for cross-validation at 3 trees, while 8 trees seem to be optimal overall. (Note that this is just a toy example!)
+
+![](images/ParameterFitter-Example.png)

source/widgets/evaluate/predictions.md

@@ -10,24 +10,32 @@ Shows models' predictions on the data.
 
 **Outputs**
 
+- Selected Predictions: a subset of predictions
 - Predictions: data with added predictions
 - Evaluation Results: results of testing classification algorithms
 
 The widget receives a dataset and one or more predictors (predictive models, not learning algorithms - see the example below). It outputs the data and the predictions.
 
 ![](images/Predictions-stamped.png)
 
-1. Information on the input, namely the number of instances to predict, the number of predictors and the task (classification or regression). If you have sorted the data table by attribute and you wish to see the original view, press *Restore Original Order*.
-2. You can select the options for classification. If *Predicted class* is ticked, the view provides information on predicted class. If *Predicted probabilities for* is ticked, the view provides information on probabilities predicted by the classifier(s). You can also select the predicted class displayed in the view. The option *Draw distribution bars* provides a visualization of probabilities.
-3. By ticking the *Show full dataset*, you can view the entire data table (otherwise only class variable will be shown).
-4. Select the desired output.
-5. Predictions.
+1. *Show probabilities for* enables showing different kinds of prediction probabilities, namely for classes in the data, for classes known to the model, or classes in data and model.
+   If *Show classification errors* is ticked, an additional error column is shown, displaying 1 - probability assigned to the correct class.
+   If you have sorted the data table by attribute and you wish to see the original view, press *Restore Original Order*.
+2. *Show performance scores* shows scores for model comparison. For classification, one can set the *Target class* to compare scores on different classes. *(Average over classes)* averages individual class scores.
 
-The widget show the probabilities and final decisions of [predictive models](https://en.wikipedia.org/wiki/Predictive_modelling). The output of the widget is another dataset, where predictions are appended as new meta attributes. You can select which features you wish to output (original data, predictions, probabilities). The result can be observed in a [Data Table](../data/datatable.md). If the predicted data includes true class values, the result of prediction can also be observed in a [Confusion Matrix](../evaluate/confusionmatrix.md).
+The widget show the probabilities and final decisions of [predictive models](https://en.wikipedia.org/wiki/Predictive_modelling). There are three widget outputs, namely another dataset, where predictions are appended as new meta attributes, a selected subset of enhanced dataset, or evaluation results (predicted data icnluding a true class values) to be used with [Confusion Matrix](../evaluate/confusionmatrix.md), [ROC Analysis](../evaluate/rocanalysis.md), [Performance Curve](../evaluate/performancecurve.md) or [Calibration Plot](../evaluate/calibrationplot.md).  Predictions can be observed in a [Data Table](../data/datatable.md) or analyzed downstream.
+
+![](images/Predictions-Regression.png)
+
+For regression, the widget shows different types of regression errors, namely *Difference* (to true value), *Absolute difference*, *Relative difference* (in percentage), and *Absolute relative*. Note that "positive" differences (overestimations) are shown in red, while "negative" differences (underestimations) are shown in blue.
+
+On the left side, a pink line indicates the forecast value, while the dot indicates a true value. This, along with the error column, allows for a quick visual estimation of predictive errors.
 
 Examples
 --------
 
+#### Predict on new data
+
 In the first example, we will use *Attrition - Train* data from the [Datasets](../data/datasets.md) widget. This is a data on attrition of employees. In other words, we wish to know whether a certain employee will resign from the job or not. We will construct a predictive model with the [Tree](../model/tree.md) widget and observe probabilities in **Predictions**.
 
 For predictions we need both the training data, which we have loaded in the first **Datasets** widget and the data to predict, which we will load in another [Datasets](../data/datasets.md) widget. We will use *Attrition - Predict* data this time. Connect the second data set to **Predictions**. Now we can see predictions for the three data instances from the second data set.
@@ -36,16 +44,18 @@ The [Tree](../model/tree.md) model predicts none of the employees will leave the
 
 ![](images/Predictions-Example1.png)
 
-In the second example, we will see how to properly use [Preprocess](../data/preprocess.md) with **Predictions** or [Test & Score](../evaluate/testandscore.md).
+#### How to use preprocessing
+
+In the second example, we will see how to properly use [Preprocess](../data/preprocess.md) with **Predictions** or [Test &andScore](../evaluate/testandscore.md).
 
 This time we are using the *heart disease.tab* data from the [File](../data/file.md) widget. You can access the data through the dropdown menu. This is a dataset with 303 patients that came to the doctor suffering from a chest pain. After the tests were done, some patients were found to have diameter narrowing and others did not (this is our class variable).
 
-The heart disease data have some missing values and we wish to account for that. First, we will split the data set into train and test data with [Data Sampler](../data/datasampler.md).
+The heart disease data have some missing values and we wish to account for that. First, we will split the data set into train and test data with [Data Sampler](../data/datasampler.md), say 70% for training and 30% for testing.
 
-Then we will send the *Data Sample* into [Preprocess](../data/preprocess.md). We will use *Impute Missing Values*, but you can try any combination of preprocessors on your data. We will send preprocessed data to [Logistic Regression](../model/logisticregression.md) and the constructed model to **Predictions**.
+Then we will send the *Data Sample* into [Preprocess](../data/preprocess.md). We will use *Impute Missing Values*, but you can try any combination of preprocessors on your data. We will send *Preprocessed Data* to [Logistic Regression](../model/logisticregression.md) and the constructed model to **Predictions**.
 
 Finally, **Predictions** also needs the data to predict on. We will use the output of [Data Sampler](../data/datasampler.md) for prediction, but this time not the *Data Sample*, but the *Remaining Data*, this is the data that wasn't used for training the model.
 
-Notice how we send the remaining data directly to **Predictions** without applying any preprocessing. This is because Orange handles preprocessing on new data internally to prevent any errors in the model construction. The exact same preprocessor that was used on the training data will be used for predictions. The same process applies to [Test & Score](../evaluate/testandscore.md).
+Notice how we send the remaining data directly to **Predictions** without applying any preprocessing. This is because Orange handles preprocessing on new data internally to prevent any errors in the model construction. The exact same preprocessor that was used on the training data will be used for predictions. The same process applies to [Test and Score](../evaluate/testandscore.md).
 
 ![](images/Predictions-Example2.png)