Low AUROC

[spacecraft1013]

The description describes a mean AUROC of 0.847, however when I run the model I get a mean AUROC of 0.489, here's the output from running the model.py script:

ambouk3@u109023:/data/ambouk3/CheXNet$ python3 model.py
=> no checkpoint found
model.py:77: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
  input_var = torch.autograd.Variable(inp.view(-1, c, h, w).cuda(), volatile=True)
The average AUROC is 0.489
The AUROC of Atelectasis is 0.5147218977086381
The AUROC of Cardiomegaly is 0.4816967965324666
The AUROC of Effusion is 0.5525329228378363
The AUROC of Infiltration is 0.4785231304193095
The AUROC of Mass is 0.5108256156533197
The AUROC of Nodule is 0.4956833510675879
The AUROC of Pneumonia is 0.4754862089500211
The AUROC of Pneumothorax is 0.5329778292671846
The AUROC of Consolidation is 0.49934268439528523
The AUROC of Edema is 0.429807208063108
The AUROC of Emphysema is 0.3639704709500116
The AUROC of Fibrosis is 0.5687886732196019
The AUROC of Pleural_Thickening is 0.443773228540649
The AUROC of Hernia is 0.5042597897539615

I am using the latest version of PyTorch and running it on a Quadro RTX 6000 with a batch size of 8, any ideas as to why this is happening?

[alaadewer]

i have the same low AUROC, do you have any idea how to raise it?

[bzhangj13zzz]

Yeah I got exactly the same problem, I wonder if the provided pth file is the trained weights or not

[Aliktk]

@bzhangj13zzz did you improved the AUROC?

[bzhangj13zzz]

@bzhangj13zzz did you improved the AUROC?

Hey sorry for the late reply, I did manage to get a more reasonable score, turned out there was some issue with weight loading, anyways I recommend checking out this repo https://github.com/mlmed/torchxrayvision It has a more easy-to-use API.

[alaadewer]

@bzhangj13zzz did you improved the AUROC?

Hey sorry for the late reply, I did manage to get a more reasonable score, turned out there was some issue with weight loading, anyways I recommend checking out this repo https://github.com/mlmed/torchxrayvision It has a more easy-to-use API.

Please, I want to know how to run this repo using resnet50?

[han-yeol]

In model.py, please rewrite code

if os.path.isfile(CKPT_PATH):
    print("=> loading checkpoint")
    modelCheckpoint = torch.load(CKPT_PATH)['state_dict']
    for k in list(modelCheckpoint.keys()):
        index = k.rindex('.')
        if (k[index - 1] == '1' or k[index - 1] == '2'):
            modelCheckpoint[k[:index - 2] + k[index - 1:]] = modelCheckpoint[k]
            del modelCheckpoint[k]
    model.load_state_dict(modelCheckpoint)
    print("=> loaded checkpoint")
else:
    print("=> no checkpoint found")

[stallbr]

In model.py, please rewrite code

if os.path.isfile(CKPT_PATH):
    print("=> loading checkpoint")
    modelCheckpoint = torch.load(CKPT_PATH)['state_dict']
    for k in list(modelCheckpoint.keys()):
        index = k.rindex('.')
        if (k[index - 1] == '1' or k[index - 1] == '2'):
            modelCheckpoint[k[:index - 2] + k[index - 1:]] = modelCheckpoint[k]
            del modelCheckpoint[k]
    model.load_state_dict(modelCheckpoint)
    print("=> loaded checkpoint")
else:
    print("=> no checkpoint found")

yeah... this fixes it. Thanks!