How to interpret occlusion annotator outputs?

[jneeven]

The occlusion annotator outputs a 1D numpy array with dtype [("instanceId", "<u4"), ("semanticId", "<u4"), ("occlusionRatio", "<f4")].

This seems sensible enough, except that in my case I don't have semanticIds (I am assigning semantic labels through replicator, because I found that otherwise the semantic segmentation annotator does not obtain any semantic info), so they are always zero. I therefore want to use the instanceIds, which appear to be unique for every entry but seem to be entirely unrelated to any instanceId I can obtain through code.

For example, if I also enable the instance segmentation annotator, instance_seg_data["info"]["idToSemantics"] lookls like this:

{'0': {'class': 'BACKGROUND'}, '1': {'class': 'UNLABELLED'}, '52': {'class': 'door_00027', 'door': 'door'}, '21': {'class': 'floor_00001', 'floor': 'floor'}, '20': {'class': 'floor_00001', 'floor': 'floor'}, '26': {'class': 'door_00001', 'door': 'door'}}

And the occlusion data, filtered for entries for which the occlusionRatio is not -1 (which I think means that the instance is not visible at all?), looks like this:

[(177, 0, 0.    ) (184, 0, 0.    ) (186, 0, 0.    ) (165, 0, 0.    )
 (253, 0, 0.    ) (260, 0, 0.    ) (261, 0, 0.7883) (194, 0, 0.    )
 (196, 0, 0.    ) (197, 0, 0.    )]

There is neither any correspondence between these two different concepts of "instanceId`, nor is the number of outputs the same. I suspect that the occlusion annotator outputs occlusion even for things that do not have any semantic labels, which is fine. But then my question is: how on earth do I interpret the occlusion instanceIds so I can trace them back to my actual instances of interest?

Update:
Even if I replace rep.modify.semantics with add_update_semantics(prim, semantic_label=label), all semanticIds in the occlusion data are still zero. If neither of these two approaches has any effect on the occlusion data, then what is the correct way to get the semantic labels to be non-zero?

[production23]

Hi there, saw your Isaac Sim post about occlusion outputs not lining up with your objects/labels.
At EvoLearns we have a solution for you that resolves those outputs to scene entities and produces per-class visibility analytics and curation suggestions. If you’re open to a private walkthrough (NDA is fine), I can show a minimal example on synthetic data and you can decide if it’s useful for your setup. Best, Aram @ EvoLearns