ValueError('signal only works in main thread')

[maxwass]

Has anyone else run into this error:
ValueError('signal only works in main thread')

I'm running a hyper parameter sweep using Weights and Biases's framework.

Running on a GPU on Google Colab which causes all launched runs to fail. Running it locally (Mac OS) prompts 'signal only works in main thread' to be printed to stdout (which also happens on Colab) but it doesn't crash.

Any ideas? It seems people using Ray with PL have come across this. The hacky solution presented there (os.environ['SLURM_JOB_NAME'] = 'bash') doesn't work in my case (neither on Mac OS or Colab).

[borisdayma]

Hi @maxwass

Does this notebook work if you run it locally?

I have another example but it may be outdated with latest PL versions (haven't tested it recently).

[maxwass]

I removed PL and made a dummy hp sweep as such:

def train():
   with wandb.init() as run:
      config = wandb.config
      a, b, c = random numbers
      run.log( {"a": a, "b": b, "c": c} )

sweep_config = { "method": "random", "metric": {"name": "a", "goal": "minimize"}, "parameters": {some_parameters} }
sweep_id = wandb.sweep(sweep_config, project='my_project')
wandb.agent(sweep_id=sweep_id, project=project, function=train)

On my local machine: MacBook Air (13-inch, Early 2015), macOS Big Sur version 11.2.2, running python 3.7, wandb version 0.12.2. It works, but still prints signal only works in main thread on each run.

I also tried it with wandb version 0.12.1 with the same outcome. Will test this on another machine (linux, ubuntu machine), but I believe I will have the same problem.

The main differences I see between this and my desired hp search with my PL model/trainer are: whatever happens in the PL/W&B callback code (in my dummy I directly use the run object, and maybe some multi-threading/process stuff with many workers for the dataloaders. Although this is definitely not a comprehensive list.

Any ideas?

[maxwass]

Update: My code successfully runs on an aws server, and my local Mac printing 'signal only works in main thread', but does not fail. On Colab, it fails with Run snk3j2ue errored: ValueError('signal only works in main thread')

[vanpelt]

The only place we're using signals is when we're wrapping the terminal console to capture stdout and stderr. The team is working on some improvements to multiprocessing that should help with this, but in the meantime can you try setting the WANDB_CONSOLE environment variable to "off" in your environment and see if the error still occurs?

[garrett361]

@vanpelt having this same issue and tried setting WANDB_CONSOLE to off, but it doesn't fix it. Same error. Any other workarounds?

[garrett361]

Discovered that it was the upgrade to pl v1.5.0 released yesterday which causes the problem: started thread here. Reverting to v1.4.9 fixes the problem.

[maxwass]

Now running on Colab and no longer getting the printed 'signal only works in main thread' nor the ValueError('signal only works in main thread') : I had to port my sweep over to a yaml file and run it with the wandb login/sweep/agent calls from the CLI, not the python API.

[borisdayma]

Just letting you know that we are looking into the issue here.

[garrett361]

@borisdayma any update on this? Same ValueError issue started happening for me running sweeps in Colab today. Seemed to be working fine one second, then can't sweep at all now. Syncing a single run works fine, though.

[garrett361]

Discovered that it was the upgrade to pl v1.5.0 released yesterday which causes the problem: started thread here. Reverting to v1.4.9 fixes the problem.

[rkechols]

Same issue of ray tune + pl, and going back to pytorch-lightning==1.4.9 fixed it for me too 👍

[max0x7ba]

Just letting you know that we are looking into the issue here.

I encounter this error as well with ray tune 1.8.0 and pl.

[kaushikb11]

@max0x7ba @borisdayma The issue has been with #10610

[denisoliveirac]

A regression was introduced in version 1.5.5 and the problem returned.
This PR solved the problem and was released in version 1.5.3. It also works in version 1.5.4, but in version 1.5.5 the problem returned.

[garrett361]

I'm also getting a ValueError('signal only works in main thread') message once again, but the runs actually sync up with wandb as expected despite the message (setup described in #10336).

[garrett361]

Hopefully fixed in #11124