Training stuck at 0% while training with bfloat16 precision

[y199387]

Hi,
I am trying to run the PYTORCH LIGHTNING CIFAR10 ~94% BASELINE TUTORIAL using cpu. When I try to switch to bf16 precision through Trainer(precision='bf16'), the training % is stuck at 0% and never progresses.

• Machine: Ubuntu 20.04
• Pytorch Lightining Version: 1.6.4
• Pytorch Version: 1.11.0

[akihironitta]

@y199387 Would you mind sharing your environment detail here to reproduce the behaviour?

wget https://raw.githubusercontent.com/Lightning-AI/lightning/master/requirements/collect_env_details.py
python collect_env_details.py

[y199387]

Sorry for replying late.
Sure, here is my environment detail:

* CUDA:
        - GPU:               None
        - available:         False
        - version:           None
* Lightning:
        - lightning-bolts:   0.5.0
        - pytorch-lightning: 1.6.4
        - torch:             1.11.0+cpu
        - torchaudio:        0.11.0+cpu
        - torchmetrics:      0.9.3
        - torchvision:       0.12.0+cpu
* Packages:
        - absl-py:           1.2.0
        - aiohttp:           3.8.1
        - aiosignal:         1.2.0
        - async-timeout:     4.0.2
        - asynctest:         0.13.0
        - attrs:             22.1.0
        - cachetools:        5.2.0
        - certifi:           2022.6.15
        - charset-normalizer: 2.1.0
        - frozenlist:        1.3.1
        - fsspec:            2022.7.1
        - google-auth:       2.10.0
        - google-auth-oauthlib: 0.4.6
        - grpcio:            1.47.0
        - idna:              3.3
        - importlib-metadata: 4.12.0
        - lightning-bolts:   0.5.0
        - markdown:          3.4.1
        - markupsafe:        2.1.1
        - multidict:         6.0.2
        - numpy:             1.21.6
        - oauthlib:          3.2.0
        - packaging:         21.3
        - pillow:            9.2.0
        - pip:               22.1.2
        - protobuf:          3.19.4
        - pyasn1:            0.4.8
        - pyasn1-modules:    0.2.8
        - pydeprecate:       0.3.2
        - pyparsing:         3.0.9
        - pytorch-lightning: 1.6.4
        - pyyaml:            6.0
        - requests:          2.28.1
        - requests-oauthlib: 1.3.1
        - rsa:               4.9
        - setuptools:        58.0.4
        - six:               1.16.0
        - tensorboard:       2.10.0
        - tensorboard-data-server: 0.6.1
        - tensorboard-plugin-wit: 1.8.1
        - torch:             1.11.0+cpu
        - torchaudio:        0.11.0+cpu
        - torchmetrics:      0.9.3
        - torchvision:       0.12.0+cpu
        - tqdm:              4.64.0
        - typing-extensions: 4.3.0
        - urllib3:           1.26.11
        - werkzeug:          2.2.2
        - wheel:             0.37.1
        - yarl:              1.8.1
        - zipp:              3.8.1
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                -
        - processor:         x86_64
        - python:            3.7.10
        - version:           #1 SMP Sat May 14 06:58:30 EDT 2022

[akihironitta]

@y199387 Let me look into this. I'll get back to you when I find something!