weird time consuming of FCOS head when changing the backbone

[goldentimecoolk]

Hi, I want to test the FPS of FCOS and my detector SSN with different backbones. But I found the results below is weird.

(time consuming of backbone+neck+head)
Resnext101-64x4d
SSN 28+55+60
FCOS 35+2+120

R50
SSN 13+16+60
FCOS 15+2.5+48

My question is that why the time of SSN neck (FPN-like structure) and FCOS head would change a lot with different backbones?

By the way, the neck and head structure of both detectors should not be changed under two backbones and I confirm this point by two ways. The first is to merely change backbone dict in the same config file (to prevent other settings changed). And the other one is to display model structure by the provided script get_flops.py.

Thank you.