Comparison of features

[Stormur]

When using the comparison function between two CoNLL-U files, I gather that the feature precision puts together both the FEAT and the MISC field. I think that this is not so desirable, though... they are distinct and the importance of the respective information is different. I mean that I care much more for the precision of morphological features than for whatever else is added as miscellanea.

Could the two precision be split?

[jheinecke]

Hi, looks like a bug. However I cannot reproduce it. I have two files with a sentence and the only difference is a different MISC column, but the feature precision is 100% as expected. Or did I not understand you question correctly?

[Stormur]

You are right, but after some further examination I might have found out why I was confused: the feature accuracy is computed according over the nodes with some wrong feature, not on the total number of features as I was expecting!

I recognise the solution is not easy when the number of features varies between gold and examined. But I fear it does not give the best picture when, for example, I have 4 nodes with a wrong feature out of 7, and so an accuracy of ca 57%, but I actually have just a handful of errors over tens of features.

[jheinecke]

That's a good point. I took the metric from the CoNLL 2017 and 2018 challenges: number of correct features (of the test sentence) / number of all features (of gold sentence). I wonder what metric could be more useful in your case. Counting only a specified list of features, or only those feature present in both files (say only Number and Person on verb forms, and ignoring Mood if absent in one of the files)?

[Stormur]

This problem looks quite difficult (and actually matter for some publication if there is not any...).

For the moment being I could envision to make this clear: it is a per-node binary accuracy, counting a node as false if there is just one off feature.

For the other, maybe a cluster similarity should be used. But maybe it cannot be pondered so close to Christmas :-)

As further material, maybe it is much easier to also implement UAS as an evaluation measure, beside LAS?

[jheinecke]

UAS should be doable quickly. I'll see what could be done for features (and I think adding the same thing for MISC)

[jheinecke]

Christmas present for all UD annotators! I added UAS, Deplabels and MISC evaluations.
I leave the Feature (and ultimately MISC) evaluation issue for after Christmas