Improve assessment of missing classes

[kadyb]

If the classifier is underfitted (too generalized) then we can't calculate the performance metrics because of the missing class in the prediction. We should improve this aspect in the future as follows: If class can't be predicted, assign it value of -1 and calculate the metrics. Then create a separate attribute with information how many classes are missing. Another problem is the differences in the occurrence of classes in the training and test sets.

Here is the current method:

num_class = length(unique(train$class))
pred_class = predict(model, test[, -1])
if (length(levels(pred_class)) != num_class) {
  accuracy = NA
} else {
  accuracy = accuracy_vec(test$class, pred_class)
}