Skip to content

Fix q global concurrency#211

Merged
maxdml merged 25 commits intomainfrom
fix-q-global-concurrency
Feb 20, 2025
Merged

Fix q global concurrency#211
maxdml merged 25 commits intomainfrom
fix-q-global-concurrency

Conversation

@maxdml
Copy link
Copy Markdown
Contributor

@maxdml maxdml commented Feb 18, 2025

  • Correctly retrieve the total number of tasks executed across all workers to compute the local limit
  • When doing this query, do a FOR UPDATE NO WAIT to lock the rows -- and fail early when failing to do so.
  • Adjusted the test_worker_concurrency_with_n_dbos_instances test to enforce global concurrency.

Note we now ignore LockNotAvailable errors.

Nit: this PR also adds a nifty information: an application_name which can be used for debugging.
Screenshot 2025-02-13 at 15 46 57

def test_worker_concurrency_with_n_dbos_instances(dbos: DBOS) -> None:

# Ensure children processes do not share global variables (including DBOS instance) with the parent
multiprocessing.set_start_method("spawn")
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found out that GHA uses "fork", which ofc shares the parent's global variable with the child, including the DBOS global instance, which would result in a DBOS Initialized multiple times with conflicting configuration / fastapi information error in the children.

@maxdml maxdml marked this pull request as draft February 19, 2025 18:20
@maxdml maxdml marked this pull request as ready for review February 19, 2025 19:51
@maxdml maxdml merged commit 1416d4d into main Feb 20, 2025
5 checks passed
@maxdml maxdml deleted the fix-q-global-concurrency branch February 20, 2025 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants