The problem about metric BS-R and BS-C

[zhouchang123]

Hi,from the paper,the detailed implementations of the metrics BS-R and BS-C have been presented, but the detailed implementation of /AutoSchemaKG/EvaluateKGC/SchemaQuality/eval.py seems to differ from part C.2. The BS-R metric corresponds to the method get_bert_global_recall, but BS-C does not provide a corresponding calculation method, and later only a simple F1 calculation was done. I implemented a version of get_bert_coverage myself and conducted experiments on the test set you provided a few days ago, and found that the results were not ideal. For the BS-C of just the event_type, it was only 12% instead of the 99% reported in the paper.

[hhy-huang]

Hi, thank you for your message.

We provided both BS-C and BS-R scripts in our code:

get_bert_score is for BS-R: In this function, we calculate the averaged bertscore-based recall between a series of sets pairs.(golden sets and predicted sets)

get_bert_global_recall is for BS-C: In this function, we merge all golden types in to a set and also merge all predicted types into a set. Then we calculate the averaged bertscore-based recall between this two sets.

Sorry for the misleading due to the function name.

[zhouchang123]

So the accurate description of the two methods should be that get_bert_score corresponds to sample-level matching, while get_bert_global_recall corresponds to set-level matching.
It seems that both scripts use the Hungarian algorithm for matching.

[hhy-huang]

Yes, and we have also elaborated that in our paper. And both of them need to employ the Hungarian algorithm for matching among different candidate pools (sample-level and entire testing set level).