How to train a model with GrayScale images?

[bluenight12]

    def load_model(self):
        self.model = Patchcore(
            backbone="wide_resnet50_2",
            pre_trained=True,
            coreset_sampling_ratio=0.1,
            layers=["layer2", "layer3"],
        )

i want to train with grayscale image this code. There's no problem with training. But if i input grayscale image, occurs error.

Failed to set tensor. Check 'is_dynamic || port.get_shape() == tensor->get_shape()' failed at src/inference/src/dev/isync_infer_request.cpp:286:
The input tensor size is not equal to the model input type: got [100,60] expecting [1,3,100,60].

this is my error code

i want to change model input size(channel). how can i do?

[alexriedel1]

Please show the full training code so that we can replicate the bug

[bluenight12]

class TrainMode():
    def __init__(self):
        self.imgsz = (100, 60)

    def load_datamodule(self):
        # Create the datamodule
        self.datamodule = Folder(
            name='example',
            root="datasets/0829",
            normal_dir="good",
            normal_test_dir="test_good",
            #abnormal_dir = 'bad',
            test_split_mode=TestSplitMode.NONE,
            val_split_mode=ValSplitMode.FROM_TRAIN,
            image_size= self.imgsz,
            task="classification",
            eval_batch_size = 1,
        )
    
    def load_model(self):
        self.model = Patchcore(
            backbone="wide_resnet50_2",
            pre_trained=True,
            coreset_sampling_ratio=0.1,
            layers=["layer2", "layer3"],
        )

    
    def load_engine(self):
        self.engine = Engine(
            normalization=NormalizationMethod.MIN_MAX,
            task=TaskType.CLASSIFICATION,    
            accelerator="cpu",
            default_root_dir="./results/logs",    
            deterministic=True,
        )
    
    def export_model(self, path):
        self.engine.export(model=self.model, 
            input_size=self.imgsz,
            export_type=ExportType.ONNX, 
            #compression_type=CompressionType.INT8,
            export_root=os.path.join(os.getcwd(), path),
            datamodule=self.datamodule, 
            ckpt_path='./results/logs/Patchcore/example/latest/weights/lightning/model.ckpt',
        )
    
    def run(self):
        self.load_datamodule()
        self.load_model()
        self.load_engine()
        self.engine.train(model=self.model, datamodule=self.datamodule)
        self.export_model(args.path)

if __name__ == "__main__":
    detector = TrainMode()
    detector.run()

i use this code convert color dataset to grayscale

for root, dirs, files in os.walk(directory):
        for file in files:
            if file.endswith('.png'):
                file_path = os.path.join(root, file)
                img = cv2.imread(file_path)
                gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
                new_dir = root + '_gray'
                os.makedirs(new_dir, exist_ok=True)
                new_file_path = os.path.join(new_dir, file)
                cv2.imwrite(new_file_path, gray_img)
                print(f"Converted {file_path} to grayscale and saved as {new_file_path}")

Train runs normally, but when the input channel changes back to 3 and the Train data is re-entered for inference, the error mentioned above occurs.

and, this is my inference code

self.onx = OpenVINOInferencer(os.path.join(path, 'model.onnx'), task=TaskType.CLASSIFICATION)
img = cv2.imread('img.png', cv2.IMREAD_GRAYSCALE)
pred_score = self.onx.predict(img, self.metadata).pred_score

[alexriedel1]

You are mixing the anomalib image reading based on Pillow with cv2 grayscale conversion and with cv2 grayscale loading. This results in more problems than you are currently aware of. The way to go would be to use an image transform on your dataset for grayscaling your images.

Here you can read how to read image transforms in anomalib https://anomalib.readthedocs.io/en/v1.1.1/markdown/guides/how_to/data/transforms.html

What you will need is this transform: https://pytorch.org/vision/main/generated/torchvision.transforms.v2.Grayscale.html

[bluenight12]

As far as I know, when doing onnxinference, you use numpy, not tensor. You can see this in the following code.

 # Convert file path or string to image if necessary
        if isinstance(image, str | Path):
            image = Image.open(image)

        # Convert PIL image to numpy array
        if isinstance(image, Image.Image):
            image = np.array(image, dtype=np.float32)
        if not isinstance(image, np.ndarray):
            msg = f"Input image must be a numpy array or a path to an image. Got {type(image)}"
            raise TypeError(msg)

this is openvino_inferencer.py

and,

this is my onnx structure. I want to change the channel among these 4(1, 3, 100, 60)

I would like to know how to change the input channel when learning in anomalib.

[samet-akcay]

You could use GrayScale transform from torchvision as part of your transforms. The advantage of this is that Anomalib will use this transform for your inference model, so you don't need to manually apply this transform before forward-passing in your inference.

For more details, you could refer to the custom transforms documentation here.
https://anomalib.readthedocs.io/en/v1.1.1/markdown/guides/how_to/data/transforms.html

[vmiller987]

Is this the appropriate way to maintain images in grayscale?

I have grayscale datasets, and the read_image converts to RGB. If I overwrite this, then the model seems (or somewhere else gets grumpy) about not being in RGB format.

My main concern is the memory footprint print(image.nbytes).

• Grayscale:
  • 1048576
• RGB:
  • 3145728

Being able to stay in Grayscale might make computations faster for this model, but potentially allow me to use more images in the dataset while training these models without dealing with OOM issues.

[alexriedel1]

Is this the appropriate way to maintain images in grayscale?

I have grayscale datasets, and the read_image converts to RGB. If I overwrite this, then the model seems (or somewhere else gets grumpy) about not being in RGB format.

My main concern is the memory footprint print(image.nbytes).

• Grayscale:

  • 1048576

• RGB:

  • 3145728

Being able to stay in Grayscale might make computations faster for this model, but potentially allow me to use more images in the dataset while training these models without dealing with OOM issues.

The memory consumption doesn't come from the very first cnn layer but (depending on your model) from the number of features that are stored in your model. Unless your OOM issues come from too large batch sizes (which you could simply reduce), using grayscale won't have much effect on the model size

[alexriedel1]

If you really want to try using grayscale images you could try what fast.ai is doing for pre-trained CNNs, they just sum up the weights of the three input conv layer channels:
https://github.com/fastai/fastai/blob/87827a10c946ea024fb09ecaf00073395e4d950d/fastai/vision/learner.py#L33

(i'm sure this will drastically lower the models accuracy)

[vmiller987]

The memory consumption doesn't come from the very first cnn layer but (depending on your model) from the number of features that are stored in your model. Unless your OOM issues come from too large batch sizes (which you could simply reduce), using grayscale won't have much effect on the model size

Thanks for the explanation! I'll take your advice and not worry about it. I am still learning and I appreciate all the help you've offered.

Luckily my anomaly datasets generally have small image sizes. I have some 3D images for segmentation that are 15-25Gb in size.

I've been wanting to try out some PatchCore but that would OOM on like 80 images compared to PaDiM which lets me easily do ~1500 images. I just need to find the time to see how I could potentially resolve it and feed PatchCore more images.

[bluenight12]

The memory consumption doesn't come from the very first cnn layer but (depending on your model) from the number of features that are stored in your model. Unless your OOM issues come from too large batch sizes (which you could simply reduce), using grayscale won't have much effect on the model size

Thanks for the explanation! I'll take your advice and not worry about it. I am still learning and I appreciate all the help you've offered.

Luckily my anomaly datasets generally have small image sizes. I have some 3D images for segmentation that are 15-25Gb in size.

I've been wanting to try out some PatchCore but that would OOM on like 80 images compared to PaDiM which lets me easily do ~1500 images. I just need to find the time to see how I could potentially resolve it and feed PatchCore more images.

"I think I used torchvision’s grayscale in the method I tried. I also changed the normalization to [0.5, 0.5], and I believe I had set it to grayscale. It's been a while since I tried using grayscale, so I’m not entirely sure, but I do recall applying grayscale in PatchCore.