Do compute_metrics() compute results over the whole dataset (all batches) or a single batch?

[bryanbocao]

Dear mmlab community,

Hope you are doing well!

I am trying to define a custom metric score but it operates in the dataset-wise instead of batch-wise fashion - it requires to process all the results from all samples (all batches) in a dataset and cannot be done in just a single or several batches.

I checked the compute_metric() in the tutorial https://mmengine.readthedocs.io/en/latest/tutorials/evaluation.html:

 def compute_metrics(self, results: List):
        """Compute the metrics from processed results.

        Args:
            results (dict): The processed results of each batch.

        Returns:
            Dict: The computed metrics. The keys are the names of the metrics,
            and the values are corresponding results.
        """

        # aggregate the classification prediction results and category labels for all samples
        preds = np.concatenate([res['pred'] for res in results])
        gts = np.concatenate([res['gt'] for res in results])

        # calculate the classification accuracy
        acc = (preds == gts).sum() / preds.size

        # return evaluation metric results
        return {'accuracy': acc}

The description of Args says

Args:
            results (dict): The processed results of each batch.

My confusion is whether the each batch refers to (1) one batch or (2) all batches in the whole dataset. The actual implementation later seems to suggest the latter (2) all batches in the whole dataset:

        # aggregate the classification prediction results and category labels for all samples
        preds = np.concatenate([res['pred'] for res in results])
        gts = np.concatenate([res['gt'] for res in results])

        # calculate the classification accuracy
        acc = (preds == gts).sum() / preds.size

        # return evaluation metric results
        return {'accuracy': acc}

It would be great to provide further clarification. Thanks!