Gradient Problems

[Squeemos]

I'm trying to call:

gradient_saliency = saliency.GradientSaliency()
vanilla_mask_3d = gradient_saliency.GetMask(im, call_model_function, call_model_args)

but I'm stuck in a loop where I can't solve the problem.

If I setup my input like this:
im = img.unsqueeze(dim=0).to("cuda").requires_grad_(True)
Then I get
RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.

But if I change it to
im = img.unsqueeze(dim = 0).to(device)
I get
RuntimeError: One of the differentiated Tensors does not require grad

It makes sense that it requires the gradient given its task, but why does it try to call .numpy() without doing a .detach()? Maybe I'm setting up something else wrong, but it seems to be isolated to this specific section.