The adaptive threshold is not in [0, 1]

[Cassie07]

Describe the bug
Hello! I made a test using PatchCore on the original MVTec dataset without any modification of config or dataset. I print the return of AdaptiveThreshold() because I am interested in viewing the value of the adaptive threshold. From my understanding, this threshold should not be larger than 1. However, the output I got is larger than 1. I am wondering whether I did not find a proper way to view the threshold or if I misunderstood anything?

Thanks for the help!

To Reproduce

1. I am using the config.yaml under ./model/patchcore folder
2. I add a print(self.value) before the return in adaptive_threshold.py (./anomalib/anomalib/utils/metrics/adaptive_threshold.py)

[samet-akcay]

Hi @Cassie07, It is the anomaly score that the algorithm calculates. Patchcore doesn't generate scores that fall between [0, 1]. The score may be fairly high depending on the application. Higher scores in this case would suggest a possible anomaly. The best metric, such as F1, is produced by the threshold score found by AdaptiveThreshold.

Let me know if this is any clearer?

[Cassie07]

@samet-akcay Thanks for the response. Maybe my description is not clear. Actually, I did not focus on the scores generated by Patchcore. My issue is that I want to know the exact threshold score (optimal F1) which AdaptiveThreshold generates. I suppose this threshold should be in [0, 1]. However, I found it is larger than 1.

I print the output of adaptive_threshold and I supposed this value is the threshold. Please feel free to let me know if I did not find the proper function for generating the threshold.

Thank you so much!

[fujikosu]

Hi I was also trying to understand how adaptive threshold is determined and when it's actually applied so let me share my understanding here. During training, patchcore model outputs unnormalized values. At the end of validation epoch, threshold is computed and determined to maximize F1 score over validation set like below with _compute_adaptive_threshold.

def validation_epoch_end(self, outputs):
        """Compute threshold and performance metrics.

        Args:
          outputs: Batch of outputs from the validation step
        """
        if self.adaptive_threshold:
            self._compute_adaptive_threshold(outputs)

As threshold is determined with un-normalized raw model output score, the value is not limited to [0, 1].

That threshold is used during inference. predict function runs pre_process, model's forward and post_process like below.

        processed_image = self.pre_process(image_arr)
        predictions = self.forward(processed_image)
        anomaly_map, pred_scores = self.post_process(predictions, meta_data=meta_data)

Adaptive threshold determined during training is used as part of post_process like below.
post_process in TorchInferencer in torch.py

        if "min" in meta_data and "max" in meta_data:
            anomaly_maps = normalize_min_max(
                anomaly_maps, meta_data["pixel_threshold"], meta_data["min"], meta_data["max"]
            )
            pred_scores = normalize_min_max(
                pred_scores, meta_data["image_threshold"], meta_data["min"], meta_data["max"]
            )

That's why you see normalized anomaly score as output of inference.py

[cyj95]

which python file is the code below from?
after Adaptive threshold is determined, meta_data["pixel_threshold"] and meta_data["image_threshold"] are also normalized? are they normalized to 0.5?

Adaptive threshold determined during training is used as part of post_process like below.
post_process in TorchInferencer in torch.py

    if "min" in meta_data and "max" in meta_data:
        anomaly_maps = normalize_min_max(
            anomaly_maps, meta_data["pixel_threshold"], meta_data["min"], meta_data["max"]
        )
        pred_scores = normalize_min_max(
            pred_scores, meta_data["image_threshold"], meta_data["min"], meta_data["max"]
        )

[samet-akcay]

normalize_min_max function is defined here:
https://github.com/openvinotoolkit/anomalib/blob/development/anomalib/post_processing/normalization/min_max.py#L31

Basically, based on the threshold, min_val and max_val, the scores are scaled to [0, 1] range and the threshold value is centered around 0.5. Therefore, anything larger than 0.5 becomes anomaly.

[innat]

@samet-akcay

Basically, based on the threshold, min_val and max_val, the scores are scaled to [0, 1] range and the threshold value is centered around 0.5. Therefore, anything larger than 0.5 becomes anomaly.

• So, at the end, technically, the threshold value becomes 0.5, right? And this is for adaptive and non-adaptive cases also?
• Is it documented how to pick threshold values for non-adaptive cases? Some analysis for best practices? Sone analytical tools / functions from anomalib?
• Adaptive threshold (image-leve, pixel-level) are not in [0, 1] - is it for all models?
• In doc, is the behaviour of adaptive threshold documented anywhere?

[samet-akcay]

@djdameln, can you provide your insight here please?

[djdameln]

So, at the end, technically, the threshold value becomes 0.5, right? And this is for adaptive and non-adaptive cases also?

Yes, this is the case for both adaptive and non-adaptive case. This is done to obtain a standardized score between the different models that is more human-intepretable. If you're interested in the raw not-normalized model scores, normalization can be disabled by setting the model.normalization_method parameter to none.

Adaptive threshold (image-leve, pixel-level) are not in [0, 1] - is it for all models?

Yes, the threshold values are expressed in the original anomaly score domain. These values are needed by the normalizer to center the anomaly scores around the value of 0.5 in the [0, 1] range.

Is it documented how to pick threshold values for non-adaptive cases? Some analysis for best practices? Sone analytical tools / functions from anomalib?
Adaptive threshold (image-leve, pixel-level) are not in [0, 1] - is it for all models?

It is hard to suggest appropriate values for the manual threshold, because the range of anomaly scores differs widely between the different algo implementations. In general, a good practice would be to train the model without normalization and then inspect the values of the predicted scores for both normal and anomalous images. This should help you get an idea of what could be a good threshold value.

In doc, is the behaviour of adaptive threshold documented anywhere?

Unfortunately, our documentation is lacking a bit on this topic. To get a better idea of how the adaptive threshold works, I would suggest to have a look at the code of the AdaptiveThreshold class, and the base anomaly module which triggers the adaptive threshold computation at the end of the validation epoch.

[samet-akcay]

Thanks for the clarification @fujikosu, this is more or less how the thresholding works!