Different inference output of trainer.predict() and model(image)

[tominator95]

I trained a Padim model for anomaly detection and want to use a custom script for inference.
I evaluated two options for inference:

• 1st: using the trainer.predict() funtionality of PyTorch Lightning like in the docs
• 2nd: using model(image)

I thought that both options must return the same output but I get different ones.

• 1st: returns a dictionary containing the key/value pairs for image, image_path, anomaly_maps, pred_scores, pred_labels and pred_masks.
• 2nd: only provides the anomaly maps.

The anomaly maps for the same image with both options are identical. Why is the second option not showing the same key/value pairs? Can someone explain this behaviour to me? Am I doing something wrong?

# loading Padim model
model = PadimLightning.load_from_checkpoint("model.ckpt")
model.eval()

# 1st option: use trainer.predict()
trainer = Trainer()
dataset = InferenceDataset(
        "rgb_17.png", image_size=tuple([300,300])
    )
dataloader = DataLoader(dataset)
output = trainer.predict(model=model, dataloaders=dataloader)
# output contains the following information : image, image_path, anomaly_maps, pred_scores, pred_labels, pred_masks

# 2nd Option: use model(input)
image = Image.open("rgb_17.png")
image = image.convert('RGB')
image_array = np.asarray(image)
transforms = A.Compose(
    [
        A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
        ToTensorV2(),
    ]
)
image_trans = transforms(image=image_array)
output2 = model(image_trans['image'].unsqueeze(0))
# output2 just contains the anomaly map

[ashwinvaidya17]

@tominator95 The lightning model sits on top of the torch model. The output of the torch model is the anomaly map. In the base anomaly lightning model (https://github.com/openvinotoolkit/anomalib/blob/d80d37db79bee58ff69fe856fac8579f7c6340ac/anomalib/models/components/base/anomaly_module.py#L82) we add the masks, labels, and scores etc to the outputs of the torch model. This is because when we export the torch model for deployment (onnx, openvino, etc), the anomaly map generation can be frozen with the model graph. The other computations can always be done on top of the these outputs.

Our tools/inference/torch_inference.py script (https://github.com/openvinotoolkit/anomalib/blob/d80d37db79bee58ff69fe856fac8579f7c6340ac/tools/inference/torch_inference.py#L73) might be more handy if you want to use the torch model. This script creates an inferencer (https://github.com/openvinotoolkit/anomalib/blob/d80d37db79bee58ff69fe856fac8579f7c6340ac/anomalib/deploy/inferencers/torch_inferencer.py#L121) which gets the anomaly map from the torch model and adds the other fields to the output dictionary when you call predict on it (https://github.com/openvinotoolkit/anomalib/blob/d80d37db79bee58ff69fe856fac8579f7c6340ac/anomalib/deploy/inferencers/base_inferencer.py#L84).

Let me know if this clears things up.

[tominator95]

@ashwinvaidya17, thank you very much for your reply. That helped a lot! 🥇 Its now clearer to me but I have some further questions now. Maybe you can help me:

1. How do you calculate the pred_scores? Is it just the mean value of the pixels which are above the threshold (anomolous)?
2. The pred_label just returns true/false. That means anomaly or not right? And a true is returned if any pixels (also just one) in the anomaly map are greater or equal the threshold?
3. I trained the model with the standard config.yaml of Padim where the image/pixel threshold is set to 3 and the adaptive threshold is true. If I use model.image_threshold.value I get 11.3447. How is this adaptive threshold calculated?

[ashwinvaidya17]

1. The classification models such as Ganomaly and DFKDE return pred_scores as outputs. The other models return anomaly maps. The pred_scores for these models are the max of the returned heat map. https://github.com/openvinotoolkit/anomalib/blob/35df574dc7a6d3a6c5b54fd5e73c9572ce80ed69/anomalib/models/components/base/anomaly_module.py#L157
2. Yes, https://github.com/openvinotoolkit/anomalib/blob/35df574dc7a6d3a6c5b54fd5e73c9572ce80ed69/anomalib/models/components/base/anomaly_module.py#L82
3. When you use adaptive threshold (set to True), the values that you define in the configuration are ovewritten by the thresholds computed by finding the threshold that achieves the highest F1 score https://github.com/openvinotoolkit/anomalib/blob/35df574dc7a6d3a6c5b54fd5e73c9572ce80ed69/anomalib/utils/metrics/adaptive_threshold.py#L36-L44

[tominator95]

Thank you very much!

[Kyle-fang]

Hello! I used onnxruntime to reason about onnx model and got the output as an anomaly map of [1, 1, 256, 256], but I want to know if the current image is an anomaly or a person normal sample what should I do? Is there code to accomplish this?

[blaz-r]

Answered in #1031

[udasinnayan]

[blaz-r]

Hi. Which model are you using? I am not familiar with tensorrt, but maybe we can figure something out if we know which model you use.