Early Stopping depending on I-AUROC and other metrics

[waschsalz]

Hello.

For more efficient training of various models, I wanted to implement a custom EarlyStopping callback so that the training stops once two threholds (I-AUROC and I-F1Score) are surpassed during validation. Therefore I used:

from lightning.pytorch.callbacks import Callback

class MultiMetricEarlyStopping(Callback):
    def __init__(self, auroc_target=0.4, f1_target=0.4,
                 auroc_key="image_AUROC", f1_key="image_F1Score"):
        super().__init__()
        self.auroc_target = float(auroc_target)
        self.f1_target = float(f1_target)
        self.auroc_key = auroc_key
        self.f1_key = f1_key

    def state_dict(self):
        return {}

    def load_state_dict(self, state_dict):
        pass

    def on_validation_end(self, trainer, pl_module):
        metrics = trainer.callback_metrics
        auroc = metrics.get(self.auroc_key)
        f1    = metrics.get(self.f1_key)

        print("\nValidation metrics:") # assure the metrics are calculated
        for k, v in trainer.callback_metrics.items():
            try:
                print(f"{k}: {float(v):.4f}")
            except (TypeError, ValueError):
                print(f"{k}: {v}")

        if auroc is not None:
            auroc = float(auroc)
        if f1 is not None:
            f1 = float(f1)

        if auroc is not None and f1 is not None:
            if auroc >= self.auroc_target and f1 >= self.f1_target:
                if hasattr(trainer, "request_stop"):
                    trainer.request_stop()
                else:
                    trainer.should_stop = True
                

However I realized that during validation the metrics are not calculated (or logged I guess). Therefore this Callback cannot work.

How do I define what to log to use this type of callback? Or am I on the wrong path entirely?

[waschsalz]

I want to add that I found out that I could just use the Evaluator for this matter. It works for AUROC but it raises an error for the F1Score metric:

from anomalib.metrics import Evaluator, F1Score, AUROC
from CustomCallbacksSADT import MultiMetricEarlyStopping

f1_score = F1Score(fields=["pred_label","gt_label"])
auroc = AUROC(fields=["pred_score","gt_label"])
evaluator = Evaluator(val_metrics=[auroc,f1_score])

engine = Engine(max_steps=15000,accelerator="cuda",callbacks=[MultiMetricEarlyStopping(auroc_target=0.8,f1_target=0.8)],num_sanity_val_steps=0)

This leads to the aforementioned error:

ValueError: Cannot update metric of type <class 'anomalib.metrics.f1_score.F1Score'>. Passed dataclass instance does not have a value for field with name pred_label.

I dont really understand where this problem with pred_label is coming from.

[abc-125]

Hello, I believe you need to use pred_score instead of pred_label:

f1_score = F1Score(fields=["pred_score","gt_label"])

[waschsalz]

Thank you for your response @abc-125.

I tried to run it with your proposed change to the f1_score, but it raises a RuntimeError.
RuntimeError: Predictions and targets are expected to have the same shape, but got torch.Size([8, 1]) and torch.Size([8]).

Both lines for AUROC and the F1 Score are identical to the ones shown in the anomalib docs.