Filter out the major and minor defects from prediction mask

[udasinnayan]

I am interested to know the way of filtering major and minor defects from test images. Few versions back of anomaly module there was a function in base.py script named def _superimpose_segmentation_mask(self, meta_data: dict, anomaly_map: np.ndarray, image: np.ndarray): for superimpose segmentation mask on top of images, and the location of the script was deploy-> inferencers-> base.py , the script has been renamed to base_inferencers.py in current version. For normalized output prediction, it was easier back then to filter out prediction mask based on compute_mask() anomaly map, by default it's 0.5 and by changing this value in previous module was easier to filter out the defects from prediction mask, what to do if I would like to do call the same function of current module, let's say I am training and testing a custom dataset based on patchcore model concept or which function should I call to filter out the of prediction mask score and show only the results after filtering.

Is it possible to get a features for filtering out the major and minor defects from prediction mask in the config file?

Any feedback would be appreciated, thanks in advance.

[djdameln]

Hi, thanks for your question. I agree that it is currently not straightforward to change the sensitivity of your models. One way to achieve this would be to train your models with a manual threshold, of which you could vary the value to change the sensitivity of your model. Currently, this is a bit cumbersome as the manual threshold can only be set during training (though a PR has just been submitted which would allow you to set a different threshold value during inference). To change the threshold value, just set dataset.metrics.threshold.method to manual and specify the manual threshold values that you want to use.

The tricky part will be to determine what would be a good threshold value. Since the threshold value is applied before normalizing the predicted anomaly scores, the range of possible values is unbounded and may vary widely between different models. A good approach could be to first train the model with adaptive thresholding enabled, and then inspect the computed adaptive threshold value to get an idea. You could then decrease or increase that value by trial and error to make the model more or less sensitive.

Hope that helps. If anything is still unclear, please do not hesitate to reach out!

[djdameln]

The computed threshold values are stored in the state dict of the saved model which is saved to the results directory. Within the state dict, the values can be found under the image_threshold.value and pixel_threshold.value keys. When exporting (ONNX/OpenVINO) is enabled during training, the threshold values are also written to the meta_data.json file which is written to the onnx or openvino folder in the results directory.

The file system location of the results directory depends on the project/path setting in your config, but by default it's under ./results in your working directory.

[nixczhou]

Is training with manual threshold will affect the model performance like Patchcore?

[djdameln]

Quantitatively, a manual threshold will always lead to a lower or equal F1 score when compared to the adaptive threshold. This is because the adaptive threshold is computed by maximizing the F1 score over the validation set. Hence, the adaptive threshold results in the highest possible F1 score that can be achieved with the model's raw (anomaly score) predictions. The AUROC score is a threshold independent metric, so it will not be affected by the value of the manual (or adaptive) threshold.

Qualitatively, you may observe different effects of changing the threshold depending on the dataset. For example, some datasets with both small and large anomalies may benefit from a lower threshold because it helps in detecting the small anomalies. These could otherwise be missed because they contribute little to the total positive count in the F1 score computation due to their small size.

[chaiiii12345]

hey so stupid question, but what is the difference between the image_threshold and pixel_threshold?

[djdameln]

Most models in Anomalib generate both pixel-level anomaly scores (a pixel map with an anomaly score for every pixel in the image, used to obtain the segmentation predictions) and image-level anomaly scores (a single score for the entire image, used to classify the image as normal or anomalous). By default, the image-level anomaly score is obtained by taking the maximum value observed in the pixel-level anomaly scores for a given image, but some models (e.g. PatchCore) implement a custom image score computation.

Both types of predictions are evaluated with their own metrics (e.g. image_F1Score and image_AUROC for image-level predictions, vs pixel_F1Score and pixel_AUROC for pixel-level predictions). The optimal value of the threshold may differ between the pixel-level and image-level domain, so we compute the thresholds separately for both type of predictions. Both image_threshold and pixel_threshold are computed by maximizing the F1 score over the validation set. The only difference is that pixel_threshold uses the pixel-level anomaly scores and the ground-truth pixel masks, while image_threshold uses the image-level anomaly scores and the image labels.