-
Notifications
You must be signed in to change notification settings - Fork 719
Sanitize labels in AWS batch #6211
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
edmundmiller
wants to merge
6
commits into
nextflow-io:master
Choose a base branch
from
edmundmiller:sanitize-labels
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
1be6ed8
fix(aws): Implement label sanitization for AWS Batch tags in AwsBatch…
edmundmiller 765580d
test(aws): Clean up failing label sanitization tests
edmundmiller 437ee33
fix(aws): Improve AWS Batch label sanitization with comprehensive log…
edmundmiller 9f5c2ea
test(aws): Add comprehensive null value handling tests for label sani…
edmundmiller ef84c99
feat(aws): Add aws.batch.sanitizeTags config option for label sanitiz…
edmundmiller b004267
refactor(aws): Replace custom sanitization config with strict mode va…
edmundmiller File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -632,7 +632,7 @@ class AwsBatchTaskHandler extends TaskHandler implements BatchHandler<String,Job | |
return result | ||
} | ||
|
||
@Memoized | ||
@Memoized | ||
LogConfiguration getLogConfiguration(String name, String region) { | ||
LogConfiguration.builder() | ||
.logDriver('awslogs') | ||
|
@@ -779,7 +779,8 @@ class AwsBatchTaskHandler extends TaskHandler implements BatchHandler<String,Job | |
builder.jobQueue(getJobQueue(task)) | ||
builder.jobDefinition(getJobDefinition(task)) | ||
if( labels ) { | ||
builder.tags(labels) | ||
final tags = validateAwsBatchLabels(labels) | ||
builder.tags(tags) | ||
builder.propagateTags(true) | ||
} | ||
// set the share identifier | ||
|
@@ -864,6 +865,98 @@ class AwsBatchTaskHandler extends TaskHandler implements BatchHandler<String,Job | |
return builder.build() | ||
} | ||
|
||
/** | ||
* Validate AWS Batch labels for compliance with AWS naming requirements. | ||
* This method validates resource labels against AWS Batch tag constraints and | ||
* handles violations based on the nextflow.enable.strict setting: | ||
* | ||
* - When strict mode is disabled (default): logs warnings for invalid tags but allows them through | ||
* - When strict mode is enabled: throws ProcessUnrecoverableException for invalid tags | ||
* | ||
* AWS Batch tag constraints validated: | ||
* - Keys and values cannot be null | ||
* - Maximum key length: 128 characters | ||
* - Maximum value length: 256 characters | ||
* - Allowed characters: letters, numbers, spaces, and: _ . : / = + - @ | ||
* | ||
* @param labels The original resource labels map | ||
* @return The labels map (unchanged in validation mode) | ||
* @throws ProcessUnrecoverableException when strict mode is enabled and labels are invalid | ||
*/ | ||
protected Map<String, String> validateAwsBatchLabels(Map<String, String> labels) { | ||
if (!labels) return labels | ||
|
||
final strictMode = executor.session.config.navigate('nextflow.enable.strict', false) | ||
final violations = [] | ||
final result = new HashMap<String, String>() | ||
|
||
for (Map.Entry<String, String> entry : labels.entrySet()) { | ||
final key = entry.getKey() | ||
final value = entry.getValue() | ||
|
||
// Check for null keys or values and filter them out (not validation violations) | ||
if (key == null) { | ||
log.warn "AWS Batch label dropped due to null key: key=null, value=${value}" | ||
continue | ||
} | ||
if (value == null) { | ||
log.warn "AWS Batch label dropped due to null value: key=${key}, value=null" | ||
continue | ||
Comment on lines
+898
to
+904
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. should not these be violations instead of warns? |
||
} | ||
|
||
final keyStr = key.toString() | ||
final valueStr = value.toString() | ||
|
||
// Validate key length | ||
if (keyStr.length() > 128) { | ||
violations << "Label key exceeds 128 characters: '${keyStr}' (${keyStr.length()} chars)" | ||
} | ||
|
||
// Validate value length | ||
if (valueStr.length() > 256) { | ||
violations << "Label value exceeds 256 characters: '${keyStr}' = '${valueStr}' (${valueStr.length()} chars)" | ||
} | ||
|
||
// Validate key characters | ||
if (!isValidAwsBatchTagString(keyStr)) { | ||
violations << "Label key contains invalid characters: '${keyStr}' - only letters, numbers, spaces, and _ . : / = + - @ are allowed" | ||
} | ||
|
||
// Validate value characters | ||
if (!isValidAwsBatchTagString(valueStr)) { | ||
violations << "Label value contains invalid characters: '${keyStr}' = '${valueStr}' - only letters, numbers, spaces, and _ . : / = + - @ are allowed" | ||
} | ||
|
||
// Add valid entries to result | ||
result[keyStr] = valueStr | ||
} | ||
|
||
// Handle violations based on strict mode (but only for constraint violations, not null filtering) | ||
if (violations) { | ||
final message = "AWS Batch tag validation failed:\n${violations.collect{ ' - ' + it }.join('\n')}" | ||
if (strictMode) { | ||
throw new ProcessUnrecoverableException(message) | ||
} else { | ||
log.warn "${message}\nTags will be used as-is but may cause AWS Batch submission failures" | ||
} | ||
} | ||
|
||
return result | ||
} | ||
|
||
/** | ||
* Check if a string contains only characters allowed in AWS Batch tags. | ||
* AWS Batch allows: letters, numbers, spaces, and: _ . : / = + - @ | ||
* | ||
* @param input The string to validate | ||
* @return true if the string contains only valid characters | ||
*/ | ||
protected boolean isValidAwsBatchTagString(String input, int maxLength = 128) { | ||
if (!input) return false | ||
if (input.length() > maxLength) return false | ||
return input ==~ /^[a-zA-Z0-9\s_.\:\/=+\-@]*$/ | ||
} | ||
|
||
/** | ||
* @return The list of environment variables to be defined in the Batch job execution context | ||
*/ | ||
|
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there's NF.isStrictMode for this
nextflow/modules/nextflow/src/main/groovy/nextflow/NF.groovy
Line 71 in d558ee3
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nextflow.enable.strict
is a feature flag, which means it's intended only for language features and is not appropriate for controlling runtime behavior. Also it will eventually be deprecated in favor of strict syntax + static type checkingI would create a new config option for strict runtime behaviors like this. Perhaps something similar to
nextflow.retryPolicy