Precision Recall for each row

[sachinruk]

Hi, just wanted to ask which metric I should as I think "macro" and "micro" version of precision/ recall doesn't fall under this definition.

So what I want happening is for a given row suppose that the true labels are ["a", "b", "c"] and my algorithm proposes ["a", "e"]. So what I'm expecting is for precision to say 50% (as out of the proposed labels, only half is correct, and recall to be 33%. The behaviour I would like to see is that this precision/ recall averaged across all rows to give final metric.

Do I need to write a custom metric for this, or is there a name for what I'm looking for? This discussion on macro vs micro is talking about precision/ recall for each label, which is not what I want I want.

This is the behaviour I'm expecting:

import torch


class Precision:
    def __init__(self, threshold: float):
        self.threshold = threshold
        self.sum = torch.FloatTensor([0.0])
        self.count = 0

    def calculate_step(self, y: torch.LongTensor, prob: torch.FloatTensor) -> torch.FloatTensor:
        y_pred = prob > self.threshold
        true_positives = y * y_pred
        true_positives_sum = true_positives.sum(dim=-1)
        predicted_pos = y_pred.sum(dim=-1)
        precision = true_positives_sum / predicted_pos

        self.sum += precision * len(precision)
        self.count += len(precision)

        return precision.mean()

    def calculate_epoch(self) -> torch.FloatTensor:
        return self.sum / self.count

    def reset(self):
        self.sum = torch.FloatTensor([0.0])
        self.count = 0


class Recall(Precision):
    def __init__(self, threshold: float):
        super().__init__(threshold)

    def calculate_step(self, y: torch.LongTensor, prob: torch.FloatTensor) -> torch.FloatTensor:
        y_pred = prob > self.threshold
        true_positives = y * y_pred
        true_positives_sum = true_positives.sum(dim=-1)
        actual_pos = y.sum(dim=-1)
        recall = true_positives_sum / actual_pos

        self.sum += recall * len(recall)
        self.count += len(recall)

        return recall.mean()