Skip to content

Ensure logging is configured on remote task workers#21379

Open
devin-ai-integration[bot] wants to merge 1 commit intomainfrom
devin/OSS-6581-1774992663
Open

Ensure logging is configured on remote task workers#21379
devin-ai-integration[bot] wants to merge 1 commit intomainfrom
devin/OSS-6581-1774992663

Conversation

@devin-ai-integration
Copy link
Copy Markdown
Contributor

When tasks run on remote Dask or Ray workers via a serialized context, the worker process may never have called setup_logging(). This means the APILogHandler is not attached to Prefect loggers and task run logs are silently lost.

Closes #18082

Changes

New API in prefect.logging.configuration:

  • ensure_logging_setup() — idempotent function that calls setup_logging() only if PROCESS_LOGGING_CONFIG has not been populated. This keeps the internal state check encapsulated rather than leaking it to callers.
  • is_logging_configured() — simple query for whether logging has been set up in this process.

Task engine (SyncTaskRunEngine / AsyncTaskRunEngine):

  • In initialize_run(), after entering hydrated_context, call ensure_logging_setup() when self.context is not None (i.e., remote execution only — local task runs are unaffected).
  • In the finally block, flush APILogHandler (sync) / await APILogHandler.aflush() (async) to ensure queued logs are sent before the task result is returned to the orchestrator.

Important review notes

  • The flush approaches differ between sync and async: sync iterates handlers on the "prefect.task_runs" logger and flushes matching APILogHandler instances, while async calls APILogHandler.aflush() (a classmethod that calls APILogWorker.drain_all()). The async path flushes all enqueued logs globally, not just for the current task run — worth verifying this is acceptable.
  • is_logging_configured() is added and tested but not called by the implementation itself (ensure_logging_setup() checks PROCESS_LOGGING_CONFIG directly). It's exposed as public API for external consumers.
  • No integration-level test exercises the full path (task engine with serialized context → logging setup → flush). The unit tests cover the new configuration functions in isolation using the existing dictConfigMock fixture pattern.

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
  • If this pull request adds new functionality, it includes unit tests that cover the changes
  • If this pull request removes docs files, it includes redirect settings in mint.json.
  • If this pull request adds functions or classes, it includes helpful docstrings.

Link to Devin session: https://app.devin.ai/sessions/c7590302b0bc40df9fba83c32ba15f91
Requested by: @desertaxle

When tasks execute on remote Dask/Ray workers via a serialized context,
the worker process may never have called setup_logging(), so the
APILogHandler is missing from Prefect loggers and task logs are silently
lost.

Add ensure_logging_setup() API in logging/configuration.py that
encapsulates the PROCESS_LOGGING_CONFIG check and only calls
setup_logging() if needed. Call it from both SyncTaskRunEngine and
AsyncTaskRunEngine initialize_run() when a serialized context is
present. Flush APILogHandler in the finally block to ensure logs are
sent before the task result is returned.

Closes #18082

Co-authored-by: Alex Streed <desertaxle@users.noreply.github.com>
Co-Authored-By: alex.s <ajstreed1@gmail.com>
@devin-ai-integration
Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions github-actions bot added bug Something isn't working integrations Related to integrations with other services labels Mar 31, 2026
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Mar 31, 2026

Merging this PR will not alter performance

✅ 2 untouched benchmarks


Comparing devin/OSS-6581-1774992663 (f7bb258) with main (69775fb)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working integrations Related to integrations with other services

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No logs from DaskTaskRunner with cluster address set

1 participant