How to load annotations in test_pipeline or inference process?

[mai-jingming]

Hello, I am implementing an idea on noisy annotations correction. The idea comes from a kind of Teacher-Student Learning, where the process looks like:

Noisy annotations + data -> Teacher model(inference) + correction algorithm -> less noisy supervision -> Student model(training).

Now, I want to use the first part of this process for annotations correction, which is like:

Noisy annotations + data -> Model(inference) + correction algorithm -> less noisy annotations -> override the old annotations.

Since this process only involves inference, it should be implemented using test_pipeline. So, is it possible to load annotations in test_pipeline? If yes, then how to do that?

[zachcoleman]

You can add LoadAnnotations to a test_pipeline, but you'll need to prevent Collect from discarding this information and manually control what's passed into the model's inference call.

This is the code snippet I use to inference manually for this kind of use case where data = dataset[0] for like a CocoDataset:

other_keys = []
for k in data:
    if k not in ["img", "img_metas"]:
        other_keys.append(k)
gt_data = {k: data[k] for k in other_keys}
data = {k: data[k] for k in ["img", "img_metas"]}

# fix inputs
data["img_metas"] = data["img_metas"].data
data["img"] = data["img"].data
if next(model.parameters()).is_cuda:
    device = next(model.parameters()).device
    data = scatter(data, [device])[0]

with torch.inference_mode():
    res = model.simple_test(data["img"].unsqueeze(0), [data["img_metas"]])

This kind of code works with pipelines like this:

train_pipeline = [
    dict(type="LoadImageFromFile"),
    dict(type="LoadAnnotations", with_mask=True),
    dict(type="DefaultFormatBundle"),
    dict(type="Collect", keys=["img", "gt_bboxes", "gt_labels", "gt_masks"]),
]

You may need to add other meta data depending on the model like pad_shape to img_metas.

[mai-jingming]

Thanks for your advice. Although it doesn't work directly in my case, it is still very close to the right solution and helps me a lot.