Questions on the selection of Tiling Data for small defect targets in large-scale images

[1094558673]

I was surprised to see that there is a "Tiling" data processing method in the project, which is more friendly to small targets, but I have a few questions:

1. If my data image resolution is 8000*2000, and not each of them happens to be a full number (the resolution may be other numbers, just the image scale is approximate), then my tile_size if consistent with the stride, the whole image does not seem to be evenly divided. And I'm not sure how much random_tile_count I should use; Also, when will "remove_border_count" and "use_random_tiling" be enabled?

2. I chose the following default tiling parameter, and the training effect on 8000*2000 data is not better than that without using it; Or what is the best way to handle the data for such large aspect ratios? I would appreciate it if you could reply.

tiling:
apply: true
tile_size: 256
stride: 256
remove_border_count: 0
use_random_tiling: False
random_tile_count: 16

[blaz-r]

Hello.

In your case, you don't really need random_tile_count. You can just set apply to True, and use tile_size and stride, others can stay as they are. You also don't have to worry if your images aren't exactly divisible, as tiler takes care of that, but preferably, you'd want to be as close as possible. If stride < tile_size it will still work but you'll get overlapping tiles.

If you want the tiler to work, you'll also need to add this to callbacks:

tiling = TilerConfigurationCallback(
    enable=config.dataset.tiling.apply,
    tile_size=config.dataset.tiling.tile_size,
    stride=config.dataset.tiling.stride,
)
callbacks.append(tiling)

If you don't, then tiling won't even happen, that might explain why you don't see the difference.
I also think that tiler is a good thing for cases like this, but it doesn't necessarily improve the results.

Hope this helped 😄

[1094558673]

@blaz-r Thank you very much for your answer! I now understand the problem. However, I would like to ask which file you would like to add the callbacks to. Because I am not a computer major, some details may not be clear. Thank you very much!

[blaz-r]

Inside get_callbacks function:

https://github.com/openvinotoolkit/anomalib/blob/02fd346a20a64085911722cf6bb10ad1c55549ca/src/anomalib/utils/callbacks/__init__.py#L47

add

tiling = TilerConfigurationCallback(
    enable=config.dataset.tiling.apply,
    tile_size=config.dataset.tiling.tile_size,
    stride=config.dataset.tiling.stride,
)
callbacks.append(tiling)

[1094558673]

Inside get_callbacks function:

https://github.com/openvinotoolkit/anomalib/blob/02fd346a20a64085911722cf6bb10ad1c55549ca/src/anomalib/utils/callbacks/__init__.py#L47

add

tiling = TilerConfigurationCallback(
    enable=config.dataset.tiling.apply,
    tile_size=config.dataset.tiling.tile_size,
    stride=config.dataset.tiling.stride,
)
callbacks.append(tiling)

Ok, I'll try it, thank you very much!!

[1094558673]

@blaz-r I'm sorry to bother you again, but if I enable tiling operation, the image_size in config.yaml is still set to 256, or it needs to be adjusted to the original image size, I have some doubts about this.

[blaz-r]

Image size stays the same, because the image loaded is still in its original size, same with the final anomaly map. The models that support tiling will then use tiling configuration to split images from original size into the defined tiles.

[1094558673]

@blaz-r Thank you for your timely reply. I have some questions after reading the code again: For example, I found that there is a call of "self.tiler.tile" in torch_model.py of the patchcore model, but it is not found in the efficient_ad file, or it is enough to add the code you mentioned above in the callback. (Because I find def tile is more important in class Tiler)
Also, I wonder if this tilling operation is done in train or if validation and lightning_inference.

[blaz-r]

Not all models support tiling due to various reasons. Efficient AD doesn't.
I'm not sure about the lightning inference and tiling, but I think it should work as tiler is added in the callback.

[willyfh]

Image size stays the same, because the image loaded is still in its original size, same with the final anomaly map. The models that support tiling will then use tiling configuration to split images from original size into the defined tiles.

@blaz-r I have question about this, when the tiling is applied, did you mean the image size stay the same when it is loaded (via dataloader) regardless the setting of image_size in the dataset?

[blaz-r]

No, sorry, what I've written is a bit confusing.
For tiling, the image_size config is the original resolution we want to process, (but we still can change its value to whatever we want) we just set the tile_size and stride to specify tiling config. Tiler then takes care to prepare the input into a smaller shape inside each model (that supports tiling).

For example, let's say we have an image with resolution of 1024x1024. We keep the image_size as is, that is image_size=1024. Then set the tiler to let's say tile_size=512 and stride=512. Dataloader will in this case load 1024x1024 images, and tiler will handle the tiling into 512x512. So models will internally have a size of 512, but the resulting anomaly map will still be 1024x1024.

Hope this clears things up, if not, let me know and I'll gladly explain better, it is afterall quite a specific thing in Anomalib.

[Testoxe]

I am having the same situation where I am supposed to be working on high dimension images with small defects, but this needs a huge amount of memory.
From my understanding, each tile has it's own model. So reducing the tile size should reduce the memory needed by the model.
But I can't manage to reduce the memory usage even when using small tiles tile_size=128 and stride=128.

Any insights ?

[blaz-r]

You will additionally need to lower the batch size. We are actively working on a design that will perform the tiling and then train a separate model, to solve this issue. It will be available soon.

[Testoxe]

Looking forward to it !
I'll be trying with a lower batch size meanwhile.
Thank you.

[frankkubler]

Hi there,
I'm trying to undestand how to handle the tilling method, but i'm a little bit disappointed, on how to use it espececially with PatchCore with python API (I don't use CLI).
Is there any documentation or explaination about it ? I already searched in anomalib repo and didn't found any clear explanation.
Many thanks for the reply.

[blaz-r]

Tiling is only supported by some models. For PatchCore it should be, but I'm not sure how to set it up within the v1 as I don't see TilerConfigurationCallback used anywhere.
I think that with API, you'll need to setup the Tiler class manually:

from anomalib.models import Patchcore
from anomalib.data.utils.tiler import Tiler

model = Patchcore()
model.model.tiler = Tiler(tile_size=256, stride=128)
...
...train code 
...
model.eval()
x = torch.rand(2, 3, 256, 512)
model(x)```

[blaz-r]

The docs on this will be added with PR #1960, the proper procedure to enable tiling either with CLI or API is denoted there.

[samet-akcay]

@blaz-r, when you have time can you help here?

[samet-akcay]

@kubikub, as @blaz-r mentioned above, a more comprehensive tiling mechanism will be enabled with this PR #1226. If you are interested, please keep an eye on it.

There is also a paper that we recently published explaining how it works
https://arxiv.org/pdf/2403.04932v1.pdf