weird results of inferences

[FedericoDeBona]

Hi, I'm experiencing some strange results in my inferences and would like to understand if I'm doing something wrong or if I'm missing something.
I'm using anomalib 1.2.0dev and I've two different setup using the metal_nut dataset from MVTec using PADiM on both:

• in the first part I'm using everything as is
• in the second part I'm using the metal_nut dataset but keeping only 1 bad image, so the folder structure will be

| metal_nut   
   | ground_truth
      | bent
          000_mask.png
   | test
      | bent
          000.png
      | good
          000.png
          ...
          024.png
   | train
      | good
          000.png
          ...
          219.png

First part results

Every inference has an high value, even for the good, and this let to a green anomaly map.

Second part results

Training the model with only 1 bad image gives this results:

Example of good

Example of bad

this time the results make sense, but the difference between the pred_score and the anomaly_map.max() is even greater

Questions

Why there is so much difference between this two experiments?
Also, the difference between the pred_score and the anomaly_map.max(), in the case if the first part, lead to this problem where the image is good, but the segmentation result will have something, is this intended? In the second part, 0.528 means that the model isn't sure, but the anomaly map max is at 1.0.

Code used

TRAINING

# ==============================
CATEGORY = "metal_nut" # [metal_nut, metal_nut_2]
# ==============================

ROOT = "/rd/"

datamodule = MVTec(category=CATEGORY, seed=42)

model = Padim()
engine = Engine()

# Train the model
engine.fit(datamodule=datamodule, model=model)

engine.export(
    model=model,
    export_type=ExportType.TORCH)

inferencer = TorchInferencer(
    path=f"{ROOT}/results/{MODEL}/MVTec/{CATEGORY}/latest/weights/torch/model.pt",
    device="auto")

INFERENCE

# ======================
DEFECT = "bent"
ONLY_STATS = False
# ======================

visualizer = ImageVisualizer(mode=VisualizationMode.FULL, task=TaskType.SEGMENTATION)

for defect in glob(f"{ROOT}/datasets/MVTec/{CATEGORY}/test/{DEFECT}"):
    if ONLY_STATS == False:
        print("================================================")
        print("DEFECT: ", os.path.basename(defect))
        print("================================================")
    max_for_defect = 0
    min_for_defect = 1
    mean = 0
    for img_path in glob(defect+"/*"):
        predictions = inferencer.predict(image=img_path)
        score = predictions.pred_score
        anomaly_map_max = predictions.anomaly_map.max()
        if score < min_for_defect:
            min_for_defect = score
        if score > max_for_defect:
            max_for_defect = score
        mean = mean + score
        if ONLY_STATS == False:
            print(f"{os.path.basename(img_path)} | Score: {str(score)} | Max Anomaly Map: {str(anomaly_map_max)}")
            display(Image.fromarray(cv2.resize(visualizer.visualize_image(predictions), (3000,750))))
            print("------------------------------------------------")

    print("================================================")
    print("STATS FOR: ", os.path.basename(defect))
    print(f"min: {min_for_defect} | max: {max_for_defect} | mean: {mean/len(glob(defect+'/*'))}")
    print("================================================")

[blaz-r]

Hi, Padim only uses good images for training, so model should in both cases predict the same value. However, images in test set are used for normalization, so that's why you see different results when you have more images with defects.

The normalization uses threshold values that provide the best F1 score. This can lead to some results that are not easily explainable, as would be in this case I assume. Threshold that over the entire test set gives the optimal F1 could mean that some cases (like this 018.png) get some false positives, but overall the segmentation and labels should be better.

[Sbaffinator98]

Hello, i have another question relating this issue.
If there is a known issue on pred_score being different that anomaly_map.max(), so is possible to have pred_score = 0.45 and label = good and still have mask segmentation.
Would it be more accurate to always take pred_score from anomaly_map.max() during the normalization min and max findings?
This would change the desired behavior of the model architecture right? If done after training would have the best F1 score and would prevent false positive.
Am i missing something?

[blaz-r]

This is not an issue, it's how the thresholding and normalization is handled. Image level (anomaly score) and pixel-level (anomaly maps) predictions undergo a separate normalization and thresholding. I am not sure how this would behave if they min and max were combined (max should actually be same already, but not the min). Most works do this separately, but if you want to you can set manual thresholds.