Colab TPU crashes when printing jax.numpy.array #12011
Unanswered
MatiasCovarrubias
asked this question in
Q&A
Replies: 1 comment
-
This is also reported here: googlecolab/colabtools#3009 It's unclear what is going on. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Since yersterday, I am getting the weirdest Colab TPU crashes in code that I use all the time. GPUs and CPUs work fine. This is a minimal implementation:
The log shows the following message:
external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
I have tried solutions specific to that error (like setting os.environ['CUDA_VISIBLE_DEVICES'] = "0") but it doesn't work. The TPU is also freezing at compilation time, but I wanted to start with this more reproducible error first. Something very weird is happening. Any help appreciated!
Beta Was this translation helpful? Give feedback.
All reactions