Skip to content

Latest commit

 

History

History
184 lines (128 loc) · 5.83 KB

20_debugging_airflow_components.rst

File metadata and controls

184 lines (128 loc) · 5.83 KB

Debugging Airflow Components with Breeze

Breeze provides debugging support for Airflow components using the --debug and --debugger flags in the breeze start-airflow command.

Starting Airflow with Debug Support

To start Airflow with debugging enabled, use the --debug flag to specify which components you want to debug:

# Debug the scheduler
breeze start-airflow --debug scheduler

# Debug multiple components
breeze start-airflow --debug scheduler --debug triggerer

# Debug all components
breeze start-airflow --debug scheduler --debug triggerer --debug api-server --debug dag-processor

# Debug with CeleryExecutor
breeze start-airflow -b postgres -P 17 --executor CeleryExecutor  --debug scheduler --debug dag-processor --debug api-server --debug triggerer --debug celery-worker

# Debug Webserver for Airflow 2.x
breeze start-airflow --debug webserver

Available Components for Debugging

  • scheduler - The Airflow scheduler that monitors Dags and triggers task instances
  • triggerer - The triggerer service that handles deferred tasks and triggers
  • api-server - The Airflow REST API server
  • dag-processor - The Dag processor service (when using standalone Dag processor)
  • edge-worker - The edge worker service (when using EdgeExecutor)
  • celery-worker - Celery worker processes (when using CeleryExecutor)

Debugger Options

Breeze supports two debugger options:

  • debugpy (default)
  • pydevd-pycharm
# Use debugpy (default)
breeze start-airflow --debug scheduler --debugger debugpy

# Use PyCharm debugger
breeze start-airflow --debug scheduler --debugger pydevd-pycharm

Using mprocs Instead of tmux

By default, breeze start-airflow uses tmux to manage multiple Airflow components. You can use mprocs as an alternative process manager with the --use-mprocs flag:

# Use mprocs instead of tmux
breeze start-airflow --use-mprocs

# Use mprocs with debugging
breeze start-airflow --use-mprocs --debug scheduler --debug triggerer

Benefits of mprocs:

  • Modern TUI with intuitive navigation
  • Better keyboard shortcuts and mouse support
  • Easier process management (start/stop/restart individual processes)
  • Cleaner visual layout with process status indicators
  • Cross-platform compatibility

Setting up VSCode for Remote Debugging

  1. Install Required Extensions

    Install the following VSCode extensions: * Python (ms-python.python) * Python Debugger (ms-python.debugpy)

  2. Create Launch Configuration

    Create or update your .vscode/launch.json file. The easiest way is to run the setup script:

    python setup_vscode.py

    This will create debug configurations for all Airflow components. Here's an example configuration for the scheduler:

    {
    "name": "Debug Airflow Scheduler",
    "type": "debugpy",
    "request": "attach",
    "justMyCode": false,
    "connect": {
        "host": "localhost",
        "port": 50231
    },
    "pathMappings": [
        {
            "localRoot": "${workspaceFolder}",
            "remoteRoot": "/opt/airflow"
        }
    ]
}

  3. Port Mapping

    Each component uses a different debug port. These ports are automatically assigned by Breeze when you start Airflow with debugging enabled:

    • Scheduler: 50231
    • Dag Processor: 50232
    • Triggerer: 50233
    • API Server: 50234
    • Celery Worker: 50235
    • Edge Worker: 50236
    • Web Server: 50237

    These ports are exposed from the Breeze container to your host machine, allowing your IDE to connect to the debugger running inside the container.

Debugging Workflow

  1. Start Airflow with Debug Support

    breeze start-airflow --debug scheduler --debugger debugpy

  2. Set Breakpoints

    In VSCode, set breakpoints in your Airflow code by clicking in the gutter next to line numbers.

  3. Attach Debugger

    • Open the Debug panel in VSCode (Ctrl+Shift+D / Cmd+Shift+D)
    • Select the appropriate debug configuration (e.g., "Debug Airflow Scheduler")
    • Click the green play button or press F5

  4. Trigger Debugging

    Perform an action that will trigger the code path with your breakpoint:

    • For scheduler: Trigger a Dag or wait for scheduled execution
    • For API server: Make an API call
    • For triggerer: Create a deferred task
    • For Dag processor: Parse a Dag file

  5. Debug Session

    Once the breakpoint is hit:

    • Inspect variables in the Variables panel
    • Use the Debug Console to evaluate expressions
    • Step through code using F10 (step over), F11 (step into), F12 (step out)
    • Continue execution with F5