Manually adding additional backbones?

[PabloFerrerGo]

Is there a way to add new backbones as for example an EfficientNet?

As far as i am aware there are only four backbones supported: cait_m48_448, deit_base_distilled_patch16_384, resnet18, wide_resnet50_2

[samet-akcay]

@PabloFerrerGo, it is possible. We use timm feature extractor to extract features. This means that you could use any backbone that timm supports as long as you know the backbone name and layers.

In addition to timm, we are also adding torch fx feature extractor in PR #675. You could get the list of supported feature extractors by

>>> model = torchvision.models.resnet18()
>>> train_nodes, eval_nodes = get_graph_node_names(model)

Once you know the name of the layers, you could extract the features by

>>> import torch
>>> from anomalib.models.components.feature_extractors import get_torchfx_feature_extractor
>>> from torchvision.models.efficientnet import EfficientNet_B5_Weights
>>> feature_extractor = get_torchfx_feature_extractor(
    backbone="efficientnet_b5", return_nodes=["features.6.8"], weights=EfficientNet_B5_Weights.DEFAULT
)
>>> input = torch.rand((32, 3, 256, 256))
>>> features = feature_extractor(input)
>>> [layer for layer in features.keys()]

Note that this has not been merged yet. We plan on merging it early next week

[Tekno-H]

Hello @PabloFerrerGo ,
Where you able to adjust the code to add additional backbones ?
If so, could you provide any leads?

@samet-akcay Can you point out where I should start editing the code in order to add a backbone?
What files I should be most interested in?

[samet-akcay]

@Tekno-H it depends on whether you want to modify it from the configuration file or from the code.

Feature extractor in the cflow model is defined here:
https://github.com/openvinotoolkit/anomalib/blob/6b799ce65c128dd7a1a867a7b87e9290378e7f55/anomalib/models/cflow/torch_model.py#L41

And FeatureExtractor object creates the backbone here using timm:
https://github.com/openvinotoolkit/anomalib/blob/6b799ce65c128dd7a1a867a7b87e9290378e7f55/anomalib/models/components/feature_extractors/feature_extractor.py#L48-L54

where backbone is the model string used by the timm library, and layers are the graph nodes, from which the features are extracted.

From the Code

>>> input = torch.rand((4, 3, 256, 256))

# MobileNetv2 feature extractor
>>> model = FeatureExtractor(backbone="mobilenetv2_050", layers=["blocks"], pre_trained=False)
>>> output = model(input)
>>> output["blocks"].shape
torch.Size([4, 8, 128, 128])

# FBNetv3 that uses MobileNetv3
>>> model = FeatureExtractor(backbone="fbnetv3_b", layers=["blocks"], pre_trained=False)
>>> output = model(input)
>>> output["blocks"].shape
torch.Size([4, 16, 128, 128])

From Config File

# MobileNetv2
model:
  name: cflow
  backbone: mobilenetv2_050
  pre_trained: true
  layers:
    - blocks
  decoder: freia-cflow
  ...

# FBNetv3 that uses MobileNetv3
model:
  name: cflow
  backbone: fbnetv3_b
  pre_trained: true
  layers:
    - blocks
  decoder: freia-cflow
  ...

You need to have a look at the models supported by timm and graph nodes for the layers.

Hope this answer your question.

[ashwinvaidya17]

In addition to what Samet mentioned, now that the TorchFX PR is merged, you can use the Python API as follows:

With torchvision models

>>> from torchvision.models.efficientnet import EfficientNet_B5_Weights
>>> from anomalib.models.components.feature_extractors import TorchFXFeatureExtractor
>>> feature_extractor = TorchFXFeatureExtractor(
        backbone="efficientnet_b5",
        return_nodes=["features.6.8"],
        weights=EfficientNet_B5_Weights.DEFAULT
    )
>>> input = torch.rand((32, 3, 256, 256))
>>> features = feature_extractor(input)
>>> [layer for layer in features.keys()]
    ["features.6.8"]
>>> [feature.shape for feature in features.values()]
    [torch.Size([32, 304, 8, 8])]

With custom models

>>> feature_extractor = TorchFXFeatureExtractor(
        "path.to.CustomModel", ["linear_relu_stack.3"], weights="path/to/weights.pth"
    )
>>> input = torch.randn(1, 1, 28, 28)
>>> features = feature_extractor(input)
>>> [layer for layer in features.keys()]
    ["linear_relu_stack.3"]

Here, you can also directly pass the model class rather than the path to it. Say you have DummyModel class already defined. Then you can do something like this.

>>> model = TorchFXFeatureExtractor(
    backbone=DummyModel,
    weights="dummy_model.pt",
    return_nodes=["conv3"],
   )
>>> features = model(test_input)
>>> features["conv3"].shape
torch.Size((32, 1, 244, 244))

[hungpthanh]

@samet-akcay @ashwinvaidya17 Hi, thank you for your valuable comments!
Currently, I want to change the backbone in padim from resnet18 to mobilenetv2_050, I have read your comments and tried but I face some trouble and need your help!

I have made several efforts as below:
First, I tried to find models supported by timm.

>>> import timm
>>> timm.list_models('*mobilenet*')
['mobilenetv2_035', 'mobilenetv2_050', 'mobilenetv2_075', 'mobilenetv2_100', 'mobilenetv2_110d', 'mobilenetv2_120d', 'mobilenetv2_140', 'mobilenetv3_large_075', 'mobilenetv3_large_100', 'mobilenetv3_large_100_miil', 'mobilenetv3_large_100_miil_in21k', 'mobilenetv3_rw', 'mobilenetv3_small_050', 'mobilenetv3_small_075', 'mobilenetv3_small_100', 'tf_mobilenetv3_large_075', 'tf_mobilenetv3_large_100', 'tf_mobilenetv3_large_minimal_100', 'tf_mobilenetv3_small_075', 'tf_mobilenetv3_small_100', 'tf_mobilenetv3_small_minimal_100']

So, I choose to change resnet18 to mobilenetv2_050.

To find the layers of mobilenetv2_050, I follow the example below:

>>> model = torchvision.models.resnet18()
>>> train_nodes, eval_nodes = get_graph_node_names(model)

However, from torchvision.models, I cannot import mobilenetv2_050. I only can import mobilenet_v2().
Therefore, I tried to find the layers of mobilenet_v2() as:

>>>m = mobilenet_v2()
>>>train_nodes, eval_nodes = get_graph_node_names(mobilenet_v2())

And, I got these layers for train_nodes:
['x', 'features.0', 'features.1.conv.0', 'features.1.conv.1', 'features.1.conv.2', 'features.2.conv.0', 'features.2.conv.1', 'features.2.conv.2', 'features.2.conv.3', 'features.3.conv.0', 'features.3.conv.1', 'features.3.conv.2', 'features.3.conv.3', 'features.3.add', 'features.4.conv.0', 'features.4.conv.1', 'features.4.conv.2', 'features.4.conv.3', 'features.5.conv.0', 'features.5.conv.1', 'features.5.conv.2', 'features.5.conv.3', 'features.5.add', 'features.6.conv.0', 'features.6.conv.1', 'features.6.conv.2', 'features.6.conv.3', 'features.6.add', 'features.7.conv.0', 'features.7.conv.1', 'features.7.conv.2', 'features.7.conv.3', 'features.8.conv.0', 'features.8.conv.1', 'features.8.conv.2', 'features.8.conv.3', 'features.8.add', 'features.9.conv.0', 'features.9.conv.1', 'features.9.conv.2', 'features.9.conv.3', 'features.9.add', 'features.10.conv.0', 'features.10.conv.1', 'features.10.conv.2', 'features.10.conv.3', 'features.10.add', 'features.11.conv.0', 'features.11.conv.1', 'features.11.conv.2', 'features.11.conv.3', 'features.12.conv.0', 'features.12.conv.1', 'features.12.conv.2', 'features.12.conv.3', 'features.12.add', 'features.13.conv.0', 'features.13.conv.1', 'features.13.conv.2', 'features.13.conv.3', 'features.13.add', 'features.14.conv.0', 'features.14.conv.1', 'features.14.conv.2', 'features.14.conv.3', 'features.15.conv.0', 'features.15.conv.1', 'features.15.conv.2', 'features.15.conv.3', 'features.15.add', 'features.16.conv.0', 'features.16.conv.1', 'features.16.conv.2', 'features.16.conv.3', 'features.16.add', 'features.17.conv.0', 'features.17.conv.1', 'features.17.conv.2', 'features.17.conv.3', 'features.18', 'adaptive_avg_pool2d', 'flatten', 'classifier.0', 'classifier.1']

Then, I changed my config from padim config as below:

model:
  name: padim
  backbone: mobilenetv2_050
  pre_trained: true
  layers:
    - features
  normalization_method: min_max # options: [none, min_max, cdf]

However, I got the error:

Traceback (most recent call last):
  File "tools/train.py", line 79, in <module>
    train(args)
  File "tools/train.py", line 58, in train
    model = get_model(config)
  File "/home/hungpt/workspace/anomalib/src/anomalib/models/__init__.py", line 132, in get_model
    model = getattr(module, f"{convert_snake_to_pascal_case(config.model.name)}Lightning")(config)
  File "/home/hungpt/workspace/anomalib/src/anomalib/models/padim/lightning_model.py", line 123, in __init__
    super().__init__(
  File "/home/hungpt/workspace/anomalib/src/anomalib/models/padim/lightning_model.py", line 49, in __init__
    self.model: PadimModel = PadimModel(
  File "/home/hungpt/workspace/anomalib/src/anomalib/models/padim/torch_model.py", line 75, in __init__
    self.n_features_original, self.n_patches = _deduce_dims(self.feature_extractor, input_size, self.layers)
  File "/home/hungpt/workspace/anomalib/src/anomalib/models/padim/torch_model.py", line 40, in _deduce_dims
    first_layer_resolution = dimensions_mapping[layers[0]]["resolution"]
  File "/home/hungpt/miniconda3/envs/python3.8/lib/python3.8/site-packages/omegaconf/listconfig.py", line 218, in __getitem__
    self._format_and_raise(key=index, value=None, cause=e)
  File "/home/hungpt/miniconda3/envs/python3.8/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/home/hungpt/miniconda3/envs/python3.8/lib/python3.8/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/home/hungpt/miniconda3/envs/python3.8/lib/python3.8/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/home/hungpt/miniconda3/envs/python3.8/lib/python3.8/site-packages/omegaconf/listconfig.py", line 215, in __getitem__
    key=index, value=self.__dict__["_content"][index]
omegaconf.errors.ConfigIndexError: list index out of range
    full_key: model.layers[0]
    object_type=list

Could you give me suggestions to overcome the error?
Thank you very much!