❓ How can I test the model and obtain its evaluation metrics after each epoch

[1813574898]

Your Question

When I was training the model, I found that testing was only conducted after all epochs had ended. I need to obtain the evaluation metrics of the model on the test set after each epoch, but I couldn't find the relevant configuration parameters in the Engine.

Forum Check

• I have searched the discussions forum for an answer to my question.

[github-actions[bot]]

Thanks for submitting this issue! It has been added to our triage queue. A maintainer will review it shortly.

[alfieroddan]

Okay this is a little complicated because there are some models who do not train over multiple validation epochs.

For example Pathcore only does one training epoch.

But for those that do, say Fastflow we can set up an Evaluator and log the results with a logger like AnomalibTensorBoardLogger:

from anomalib.data import MVTecAD
from anomalib.engine import Engine
from anomalib.models import Fastflow
from anomalib.metrics import Evaluator, AUROC
from anomalib.loggers import AnomalibTensorBoardLogger

auroc = AUROC(fields=["pred_score", "gt_label"])

evaluator = Evaluator(val_metrics=[auroc])

model = Fastflow(evaluator=evaluator)
datamodule = MVTecAD()
engine = Engine(
    logger=AnomalibTensorBoardLogger(
        save_dir="logs",
        name="fastflow",
    ),
)

engine.fit(
    model=model,
    datamodule=datamodule,
)

Then you can visualise them in your Tensorboard:

Obviously may want more than just AUROC. I suggest looking at the evaluator docs to see more on what metrics to log:

https://anomalib.readthedocs.io/en/latest/markdown/guides/how_to/evaluation/evaluator.html