feat(tasks): Implement OS-based task routing logic #4995
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Modify add_task to publish base_os_version attribute. - Modify task consumers to select subscription based on BASE_OS_VERSION env. - Add unit tests for new publishing and consuming logic.
| queue_name = POSTPROCESS_QUEUE | ||
| base_os_version = environment.get_value('BASE_OS_VERSION') | ||
| if base_os_version: | ||
| queue_name = f'{queue_name}-{base_os_version}' |
There was a problem hiding this comment.
I think it could be an static configuration.
There was a problem hiding this comment.
That's a good point. In this case, the dynamic approach is needed so the bot can be aware of its own OS environment. The BASE_OS_VERSION is set at the infrastructure level (in the Docker image), allowing each worker to correctly identify which filtered queue it should pull from.
| queue_name = PREPROCESS_QUEUE | ||
| base_os_version = environment.get_value('BASE_OS_VERSION') | ||
| if base_os_version: | ||
| queue_name = f'{queue_name}-{base_os_version}' |
|
|
||
| # Determine base_os_version. | ||
| base_os_version = job.base_os_version | ||
| if environment.get_value('PROJECT_NAME') == 'oss-fuzz': |
There was a problem hiding this comment.
I think we could move this oss-fuzz to a constant to avoid this magic value
There was a problem hiding this comment.
Good point. I've updated the code to use the existing job.is_external() method, which removes the magic value.
- Revert generic change in get_regular_task. - Modify default_queue_suffix to be OS-aware for Linux platforms. - Use job.is_external() instead of magic string for OSS-Fuzz check. - Update and fix tests to reflect all refactoring.
This will be moved to a separate PR to accelerate the image build.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change implements the core application logic to support routing ClusterFuzz tasks to different worker pools based on the target operating system.
This is achieved via two main changes:
Task Publishing (
tasks.add_task):The
add_taskfunction now determines the requiredbase_os_versionfor a task by checking theJobandOssFuzzProjectentities. This OS version is then added as abase_os_versionattribute to the message published on Pub/Sub, effectively "tagging" the task for consumption by the correct worker fleet.Task Consuming (
get_*_taskfunctions):The consumer functions (
get_preprocess_task,get_postprocess_task,get_utask_mains) have been updated to be OS-aware. They now read aBASE_OS_VERSIONenvironment variable (expected to be set in the Docker image). If this variable is present, they dynamically construct the name of the Pub/Sub subscription to pull from (e.g.,preprocessbecomespreprocess-ubuntu-24-04).Unit tests have been added to cover both the publishing and consuming logic.
Dependencies:
This change depends on corresponding infrastructure changes in the
clusterfuzz-configrepository, where the filtered Pub/Sub subscriptions (e.g.,preprocess-ubuntu-24-04) are created. The code will fail to connect if the subscriptions do not exist.Fixes: b/441792657