TPU training and WandB implementation causes "AssertionError: can only test a child process"

[adamDhalla]

Hi all,

I'm not raising this issue as a bug since it's already been raised here, and was seemingly solved.

I have pretty much the exact problem - I'm trying to implement weights and biases with a TPU and it keeps showing "AssertionError: can only test a child process". The person who raised the old error also replicated the issue in the boring model here.

The thing is he never explains how he solves the issue - he explains the issue itself, but doesn't say how he solves it. I'm wondering where I should be initializing the wandb object:

    config = {'hparam1' : number, 'hparam2' : number2 ..... }
    wandb_logger = WandbLogger(name='EXP_NAME',project='PROJECT_NAME',log_model=True,config=config)

Right now, I'm initializing both of those lines above the trainer. The trainer requires wandb_logger be passed through, so I can't do the wandb_logger = WandbLogger... line any higher up.

The whole problem seems to come about as a problem with distributed processing and WandB.

Thanks,
Adam

[morganmcg1]

@adamDhalla is this still an issue for you? I think wandb support for TPUs should be better in the latest version