Add CVRMSE (Coefficient of variation of the RMSE)#2209
Add CVRMSE (Coefficient of variation of the RMSE)#2209lopezco wants to merge 4 commits intobiolab:masterfrom lopezco:patch-3
Conversation
|
Jose LOPEZ seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
This reverts commit 5a3fe0a.
Codecov Report
@@ Coverage Diff @@
## master #2209 +/- ##
==========================================
+ Coverage 71.12% 71.13% +<.01%
==========================================
Files 318 318
Lines 54668 54671 +3
==========================================
+ Hits 38885 38888 +3
Misses 15783 15783Continue to review full report at Codecov.
|
janezd
left a comment
janezd
left a comment
There was a problem hiding this comment.
There are zillion of performance measures and I'd be hesitant to burden Orange, and widgets in particular, with all of them. This one looks sensible at first - it essentially gives the percentage of error. On the second look, it's maybe not: if a classifier misses my age by 10 %, I'd recalculate this to actual years to get a grasp of the error size. Wouldn't the user always know the range and be interested in the absolute error?
Another score that we could add is the sum of squares of [(y - y_predicted) / y] over all y. And we could probably come up with many more...
@BlazZupan, @astaric, @ajdapretnar: your opinions? Do we reorganize the widget to eventually support dozens of scores or do we stay with the essentials?
| Coefficient of variation of the RMSE | ||
| """ | ||
| def compute_score(self, results): | ||
| return RMSE(results)/np.nanmean(results.actual) * 100 |
There was a problem hiding this comment.
What if np.nanmean(results.actual) is zero (or very small) or negative?
|
|
||
| class CVRMSE(Score): | ||
| """ | ||
| Coefficient of variation of the RMSE |
There was a problem hiding this comment.
Wouldn't it be better to call this normalized RMSE? I know it is ambiguous, but CVRMSE doesn't tell me anything.
There was a problem hiding this comment.
I like the idea of adding more (not dozens maybe) scores. We had some requests like sensitivity and selectivity from scientists who are interested in using the infrared addon. Also, it would be nice, if the scores are displayed in an easily copy-pasteable table...
|
Perhaps it's a stupid idea, but could we create an add-on, that is essentially Orange on steroids? Which would enhance Orange with all sorts of measures and features, but we would keep the core distro light? |
|
Is it possible to add another input to test and score (like python function) to define custom metrics? This way users could define the metrics that they want to evaluate. |
|
+1 for additional input that accepts additional scorers. We already have this in Rank and Preprocess widgets, I cannot think of a reason not to add it to Test & Score. @janezd? |
|
I'm not too happy with the "powerpack" idea. It's great on the surface, but in the future it can lead to a "kitchen sink add on", and to a confusion about what is in the core and what is available only in the add-on. Not to mention that a user who would need (or like, but not necessarily need) a single extra feature would have to install the entire kitchen sink and pollute Orange with dozens of in-the-rare-case-that-anybody-would-be-interested-in-this features. I so don't like where this could be going; I'd like to have a more thorough discussion before doing this. As for the new signal, I'm skeptical of small, single-use signals for specific widgets. I believe signals should be general, and we shouldn't have too many different types. Specific signals don't make widgets reusable. It's like lego bricks that fit only on a single car model. Lego shouldn't have gone there. Considering the above two options, I'm more inclined towards the third one, which I disliked the first (but still the least): redesign the Test & Score widget to be able to accommodate a larger number of scorers without looking like a kitchen sink. |
|
I completely see the "kitchen sink" point and agree with not adding dozens of features. I like the third option of @janezd : maybe add scoring features and make a little icon on the widget itself that would allow the user to select which ones they want to display - then it is up to the individuals how much visual pollution they can bear with? |
|
But the idea of having an extra input with a custom function can be done by connecting a OWPythoScript with a metric function on one of its outputs... For instance out_object. |
|
Even if we use PythonScript for this, Test&Score would still need to accept an additional input. In this way, we can also add additional inputs of many different types to many widgets and let the users use PythonScript to provide the instances of this types. I don't like this path. But I have put redesigning the Test & Score on my agenda for this Friday. I'll try to find a non user-intimidating, but clean way to include dozens of scores into the widget. |
|
We can now decide about this. CVRMSE can be added since #2257 ensures that the user doesn't see too many columns. On the other hand, it can be a part of add-on (which?!) since #2269 will pull all existing scorers into the Test & Score widget. In either case, this PR is incompatible with #2269: changing owtestlearners.py is no longer necessary (and will cause a conflict). Besides, it has to define the |
|
I moved this code (in a slightly adapted form) to #2269. Thanks! |
7 participants
Description of changes
Scoring metric for regression
Includes