Incompatibility with Spyder profiler

[OverLordGoldDragon]

Bug description

TL;DR, import lightning makes Spyder's profiler outputs blank; furthest I've traced the import chain is to

from lightning.fabric.loggers.tensorboard import TensorBoardLogger

in fabric/loggers/__init__.py. Popen with DEVNULL redirection could explain it; question is, is that something that happens via the above import? If not, what could be the problem?

Report generated w/ Gemini 2.5 below.

---

Debug summary

1. Initial Problem: Profiling a simple import lightning statement in Spyder on Windows resulted in a blank output, despite the profiler working for other imports (import torch) and scripts.
2. Initial Localization: We identified the trigger within the lightning codebase as the import of TensorBoardLogger in fabric/loggers/__init__.py. Commenting out from lightning.fabric.loggers.tensorboard import TensorBoardLogger reliably prevented the blank profiler output.
3. Potential Underlying Mechanism (External Factor): Separate testing (detailed in the provided MREs) revealed a similar blank profiler output issue in Spyder when profiling scripts that use subprocess.Popen to launch specific external commands (like conda or git) with stdout and stderr redirected to subprocess.DEVNULL on Windows. This issue occurs independently of the lightning library.
4. Hypothesized Connection: It's highly possible that the import and initialization path triggered by TensorBoardLogger (potentially involving tensorboard, tensorboardX, or related filesystem/environment checks) executes a subprocess.Popen call matching the pattern identified in the separate MREs (external command + stdout/stderr=subprocess.DEVNULL on Windows). This specific type of subprocess interaction, when run under Spyder's profiler, appears to prevent the profiler from successfully collecting or displaying statistics for the parent script, leading to the observed blank output. The issue isn't necessarily a bug within lightning's logic itself, but rather an interaction between a specific code pattern potentially used within its dependency chain (TensorBoardLogger) and the Spyder profiler's behavior on Windows regarding certain subprocess calls with redirected I/O.

Popen MREs

spyder_mre_conda_minimal.py

# spyder_mre_conda_minimal.py
import subprocess
import sys
import shutil
import time
from pathlib import Path

IS_WINDOWS = sys.platform == "win32"

# --- Minimal Configuration ---
# Try 'conda.bat' first, adjust if you primarily use mamba
CONDA_EXE_NAME = "conda.bat" if IS_WINDOWS else "conda"
OUTPUT_FILENAME = Path("./temp_environment_minimal.yaml")
# --- End Minimal Configuration ---

conda_exe_path = shutil.which(CONDA_EXE_NAME)

if conda_exe_path:
    command = [
        str(conda_exe_path), "env", "export", "--no-builds", "--file",
        str(OUTPUT_FILENAME.resolve())
    ]
    try:
        process = subprocess.Popen(
            command,
            stdout=subprocess.DEVNULL,
            stderr=subprocess.DEVNULL,
            close_fds=not IS_WINDOWS
        )
        # Minimal wait to allow potential profiler interaction
        time.sleep(2)
        # Optional: Terminate if still running, ignore errors
        try:
            process.terminate()
        except:
            pass
    except Exception:
        # Ignore exceptions for MRE, failure is implicit
        pass
    finally:
         # Optional cleanup, ignore errors
         try:
            OUTPUT_FILENAME.unlink(missing_ok=True)
         except:
            pass

spyder_mre_git_minimal.py

# spyder_mre_git_minimal.py
import subprocess
import sys
from pathlib import Path
import time

IS_WINDOWS = sys.platform == "win32"
HELPER_SCRIPT = Path(__file__).resolve().parent / "spyder_mre_git_helper_minimal.py"

if HELPER_SCRIPT.exists():
    command = [
        sys.executable,
        str(HELPER_SCRIPT)
    ]
    try:
        process = subprocess.Popen(
            command,
            stdout=subprocess.DEVNULL,
            stderr=subprocess.DEVNULL,
            close_fds=not IS_WINDOWS
        )
        # Minimal wait to allow potential profiler interaction
        time.sleep(2)
        # Optional: Terminate if still running, ignore errors
        try:
            process.terminate()
        except: 
            pass
    except Exception:
        # Ignore exceptions for MRE, failure is implicit
        pass

spyder_mre_git_helper_minimal.py

# spyder_mre_git_helper_minimal.py
import subprocess
import sys

try:
    subprocess.run(['git', 'status'], check=True, capture_output=True)
    sys.exit(0)
except:
    sys.exit(1)

What version are you seeing the problem on?

2.5.1

Environment

Current environment

• CUDA:
  - GPU:
  - NVIDIA GeForce RTX 4060 Laptop GPU
  - available: True
  - version: 12.4
• Lightning:
  - gpytorch: 1.14
  - lightning: 2.5.1
  - lightning-utilities: 0.14.3
  - pytorch-ignite: 0.5.1
  - pytorch-lightning: 2.5.1
  - torch: 2.5.1
  - torchmetrics: 1.7.1
• Packages:
  - absl-py: 2.2.0
  - aiohappyeyeballs: 2.6.1
  - aiohttp: 3.11.14
  - aiosignal: 1.3.2
  - alabaster: 1.0.0
  - alembic: 1.15.2
  - annotated-types: 0.7.0
  - anyio: 4.9.0
  - appdirs: 1.4.4
  - argon2-cffi: 23.1.0
  - argon2-cffi-bindings: 21.2.0
  - arrow: 1.3.0
  - astroid: 3.3.9
  - asttokens: 3.0.0
  - async-lru: 2.0.5
  - asyncssh: 2.20.0
  - atomicwrites: 1.4.1
  - attrs: 25.3.0
  - audioread: 3.0.1
  - autocommand: 2.2.2
  - autopep8: 2.0.4
  - babel: 2.17.0
  - backports.tarfile: 1.2.0
  - bcrypt: 4.3.0
  - beautifulsoup4: 4.13.3
  - binaryornot: 0.4.4
  - black: 25.1.0
  - bleach: 6.2.0
  - blinker: 1.9.0
  - brotli: 1.1.0
  - cached-property: 1.5.2
  - cachetools: 5.5.2
  - certifi: 2025.1.31
  - cffi: 1.17.1
  - chardet: 5.2.0
  - charset-normalizer: 3.4.1
  - click: 8.1.8
  - cloudpickle: 3.1.1
  - colorama: 0.4.6
  - comm: 0.2.2
  - contourpy: 1.3.0
  - cookiecutter: 2.6.0
  - crc32c: 2.7.1
  - cryptography: 44.0.2
  - cycler: 0.12.1
  - databricks-sdk: 0.49.0
  - datasets: 3.4.1
  - debugpy: 1.8.13
  - decorator: 5.2.1
  - defusedxml: 0.7.1
  - deprecated: 1.2.18
  - diff-match-patch: 20241021
  - dill: 0.3.8
  - dnspython: 2.7.0
  - docker: 7.1.0
  - docker-pycreds: 0.4.0
  - docstring-to-markdown: 0.15
  - docutils: 0.21.2
  - donfig: 0.8.1.post1
  - email-validator: 2.2.0
  - eval-type-backport: 0.2.2
  - exceptiongroup: 1.2.2
  - executing: 2.1.0
  - fastapi: 0.115.12
  - fastapi-cli: 0.0.7
  - fastjsonschema: 2.21.1
  - ffmpeg-python: 0.2.0
  - ffmpy: 0.3.0
  - filelock: 3.18.0
  - flake8: 7.1.2
  - flask: 3.1.0
  - fonttools: 4.53.1
  - fqdn: 1.5.1
  - frozenlist: 1.5.0
  - fsspec: 2024.12.0
  - future: 1.0.0
  - gitdb: 4.0.12
  - gitpython: 3.1.44
  - google-api-core: 2.24.2
  - google-auth: 2.38.0
  - google-cloud-core: 2.4.3
  - google-cloud-storage: 3.1.0
  - google-crc32c: 1.7.0
  - google-resumable-media: 2.7.2
  - googleapis-common-protos: 1.69.2
  - gpytorch: 1.14
  - graphene: 3.4.3
  - graphql-core: 3.2.6
  - graphql-relay: 3.2.0
  - greenlet: 3.1.1
  - grpcio: 1.71.0
  - gssapi: 1.9.0
  - h11: 0.14.0
  - h2: 4.2.0
  - hpack: 4.1.0
  - httpcore: 1.0.7
  - httptools: 0.6.4
  - httpx: 0.28.1
  - huggingface-hub: 0.29.3
  - hyperframe: 6.1.0
  - idna: 3.10
  - imagesize: 1.4.1
  - importlib-metadata: 8.6.1
  - importlib-resources: 6.5.2
  - inflect: 7.3.1
  - inflection: 0.5.1
  - intervaltree: 3.1.0
  - ipykernel: 6.29.5
  - ipython: 8.34.0
  - ipywidgets: 8.1.5
  - isoduration: 20.11.0
  - isort: 6.0.1
  - itsdangerous: 2.2.0
  - jams: 0.3.5a0
  - jaraco.classes: 3.4.0
  - jaraco.collections: 5.1.0
  - jaraco.context: 6.0.1
  - jaraco.functools: 4.1.0
  - jaraco.text: 3.12.1
  - jaxtyping: 0.2.36
  - jedi: 0.19.2
  - jellyfish: 1.1.3
  - jinja2: 3.1.6
  - joblib: 1.4.2
  - json5: 0.10.0
  - jsonpointer: 3.0.0
  - jsonschema: 4.23.0
  - jsonschema-specifications: 2024.10.1
  - jupyter: 1.1.1
  - jupyter-client: 8.6.3
  - jupyter-console: 6.6.3
  - jupyter-core: 5.7.2
  - jupyter-events: 0.12.0
  - jupyter-lsp: 2.2.5
  - jupyter-server: 2.15.0
  - jupyter-server-terminals: 0.5.3
  - jupyterlab: 4.3.6
  - jupyterlab-pygments: 0.3.0
  - jupyterlab-server: 2.27.3
  - jupyterlab-widgets: 3.0.13
  - keyring: 25.6.0
  - kiwisolver: 1.4.7
  - lazy-loader: 0.4
  - librosa: 0.11.0
  - lightning: 2.5.1
  - lightning-utilities: 0.14.3
  - linear-operator: 0.6
  - llvmlite: 0.43.0
  - mako: 1.3.9
  - markdown: 3.6
  - markdown-it-py: 3.0.0
  - markupsafe: 3.0.2
  - matplotlib: 3.9.1
  - matplotlib-inline: 0.1.7
  - mccabe: 0.7.0
  - mdurl: 0.1.2
  - mir-eval: 0.8.2
  - mistune: 3.1.3
  - mlflow: 2.21.2
  - mlflow-skinny: 2.21.2
  - more-itertools: 10.6.0
  - mouseinfo: 0.1.3
  - mpmath: 1.3.0
  - msgpack: 1.1.0
  - multidict: 6.2.0
  - multiprocess: 0.70.16
  - munkres: 1.1.4
  - musdb: 0.4.2
  - museval: 0.4.1
  - mypy-extensions: 1.0.0
  - nbclient: 0.10.2
  - nbconvert: 7.16.6
  - nbformat: 5.10.4
  - nest-asyncio: 1.6.0
  - networkx: 3.4.2
  - norbert: 0.2.1
  - notebook: 7.3.3
  - notebook-shim: 0.2.4
  - numba: 0.60.0
  - numcodecs: 0.15.1
  - numpy: 2.0.2
  - numpydoc: 1.8.0
  - nussl: 1.1.9
  - opentelemetry-api: 1.31.1
  - opentelemetry-sdk: 1.31.1
  - opentelemetry-semantic-conventions: 0.52b1
  - overrides: 7.7.0
  - packaging: 24.2
  - pandas: 2.2.3
  - pandocfilters: 1.5.0
  - paramiko: 3.5.1
  - parso: 0.8.4
  - pathspec: 0.12.1
  - pexpect: 4.9.0
  - pickleshare: 0.7.5
  - pillow: 10.4.0
  - pip: 25.0.1
  - pkgutil-resolve-name: 1.3.10
  - platformdirs: 4.3.7
  - pluggy: 1.5.0
  - ply: 3.11
  - pooch: 1.8.2
  - prometheus-client: 0.21.1
  - prometheus-flask-exporter: 0.23.2
  - prompt-toolkit: 3.0.50
  - propcache: 0.2.1
  - proto-plus: 1.26.1
  - protobuf: 5.29.3
  - psutil: 7.0.0
  - ptyprocess: 0.7.0
  - pure-eval: 0.2.3
  - pyaml: 25.1.0
  - pyarrow: 19.0.1
  - pyasn1: 0.6.1
  - pyasn1-modules: 0.4.1
  - pyautogui: 0.9.54
  - pycodestyle: 2.12.1
  - pyconify: 0.2.1
  - pycparser: 2.22
  - pydantic: 2.10.6
  - pydantic-core: 2.27.2
  - pydocstyle: 6.3.0
  - pyflakes: 3.2.0
  - pygetwindow: 0.0.9
  - pygithub: 2.6.1
  - pygments: 2.19.1
  - pyjwt: 2.10.1
  - pylint: 3.3.5
  - pylint-venv: 3.0.4
  - pyloudnorm: 0.1.1
  - pyls-spyder: 0.4.0
  - pymsgbox: 1.0.9
  - pynacl: 1.5.0
  - pyopenssl: 25.0.0
  - pyparsing: 3.1.4
  - pyperclip: 1.9.0
  - pyqt5: 5.15.9
  - pyqt5-sip: 12.12.2
  - pyqtwebengine: 5.15.4
  - pyrect: 0.2.0
  - pyscreeze: 1.0.1
  - pysocks: 1.7.1
  - python-dateutil: 2.9.0.post0
  - python-dotenv: 1.1.0
  - python-json-logger: 2.0.7
  - python-lsp-black: 2.0.0
  - python-lsp-jsonrpc: 1.1.2
  - python-lsp-server: 1.12.2
  - python-multipart: 0.0.20
  - python-slugify: 8.0.4
  - python-xlib: 0.33
  - pytoolconfig: 1.2.5
  - pytorch-ignite: 0.5.1
  - pytorch-lightning: 2.5.1
  - pytweening: 1.2.0
  - pytz: 2024.1
  - pyu2f: 0.1.5
  - pyuca: 1.2
  - pywin32: 307
  - pywin32-ctypes: 0.2.3
  - pywinpty: 2.0.15
  - pyyaml: 6.0.2
  - pyzmq: 26.3.0
  - qdarkstyle: 3.2.3
  - qstylizer: 0.2.4
  - qtawesome: 1.4.0
  - qtconsole: 5.6.1
  - qtpy: 2.4.3
  - querystring-parser: 1.2.4
  - referencing: 0.36.2
  - regex: 2024.11.6
  - requests: 2.32.3
  - rfc3339-validator: 0.1.4
  - rfc3986-validator: 0.1.1
  - rich: 13.9.4
  - rich-toolkit: 0.11.3
  - roman-numerals-py: 3.1.0
  - rope: 1.13.0
  - rpds-py: 0.23.1
  - rsa: 4.9
  - rtree: 1.4.0
  - rubicon-objc: 0.5.0
  - safetensors: 0.5.3
  - scikit-learn: 1.6.1
  - scipy: 1.14.1
  - send2trash: 1.8.3
  - sentry-sdk: 2.24.1
  - setproctitle: 1.3.5
  - setuptools: 75.8.2
  - shellingham: 1.5.4
  - simplejson: 3.20.1
  - sip: 6.7.12
  - six: 1.17.0
  - smmap: 5.0.2
  - snakeviz: 2.2.2
  - sniffio: 1.3.1
  - snowballstemmer: 2.2.0
  - sortedcontainers: 2.4.0
  - soundfile: 0.13.1
  - soupsieve: 2.5
  - sox: 1.5.0
  - soxr: 0.5.0.post1
  - sphinx: 8.2.3
  - sphinxcontrib-applehelp: 2.0.0
  - sphinxcontrib-devhelp: 2.0.0
  - sphinxcontrib-htmlhelp: 2.1.0
  - sphinxcontrib-jsmath: 1.0.1
  - sphinxcontrib-qthelp: 2.0.0
  - sphinxcontrib-serializinghtml: 1.1.10
  - spyder: 6.0.5
  - spyder-kernels: 3.0.3
  - sqlalchemy: 2.0.40
  - sqlparse: 0.5.3
  - stack-data: 0.6.3
  - standard-aifc: 3.13.0
  - standard-sunau: 3.13.0
  - starlette: 0.46.1
  - stempeg: 0.2.3
  - superqt: 0.7.3
  - sympy: 1.13.1
  - tabulate: 0.9.0
  - tensorboard: 2.19.0
  - tensorboard-data-server: 0.7.0
  - terminado: 0.18.1
  - termtables: 0.2.4
  - text-unidecode: 1.3
  - textdistance: 4.6.3
  - threadpoolctl: 3.6.0
  - three-merge: 0.1.1
  - tinycss2: 1.4.0
  - tokenizers: 0.21.1
  - toml: 0.10.2
  - tomli: 2.2.1
  - tomlkit: 0.13.2
  - torch: 2.5.1
  - torchmetrics: 1.7.1
  - tornado: 6.4.2
  - tqdm: 4.67.1
  - traitlets: 5.14.3
  - transformers: 4.50.0
  - typeguard: 2.13.3
  - typer: 0.15.2
  - typer-slim: 0.15.2
  - types-python-dateutil: 2.9.0.20241206
  - typing-extensions: 4.12.2
  - typing-utils: 0.1.0
  - tzdata: 2025.2
  - ujson: 5.10.0
  - uri-template: 1.3.0
  - urllib3: 2.3.0
  - uvicorn: 0.34.0
  - waitress: 3.0.2
  - wandb: 0.19.8
  - watchdog: 6.0.0
  - watchfiles: 0.24.0
  - wcwidth: 0.2.13
  - webcolors: 24.11.1
  - webencodings: 0.5.1
  - websocket-client: 1.8.0
  - websockets: 15.0.1
  - werkzeug: 3.1.3
  - whatthepatch: 1.0.7
  - wheel: 0.45.1
  - widgetsnbextension: 4.0.13
  - win-inet-pton: 1.1.0
  - wrapt: 1.17.2
  - xxhash: 3.5.0
  - yapf: 0.43.0
  - yarl: 1.18.3
  - zarr: 3.0.6
  - zipp: 3.21.0
  - zstandard: 0.23.0
• System:
  - OS: Windows
  - architecture:
  - 64bit
  - WindowsPE
  - processor: Intel64 Family 6 Model 191 Stepping 2, GenuineIntel
  - python: 3.12.9
  - release: 11
  - version: 10.0.26100

[SkafteNicki]

I am going to close this as the identified problem is outside the scope of the lightning codebase, it is between one of our dependencies and spyder. Feel free to ping me and reopen if the problem get solved in the linked spyder issue and the issue still persist on lightning side.