Merge pull request #4734 from linuxfoundation/unicron-support-empty-patterns-in-skip-cla

Add support for patterns matching empty or missing properties: login, email, name

Author: lukaszgryglicki

BOT_ALLOWLIST.md

@@ -6,26 +6,27 @@ This can be done on the GitHub organization level by setting the `skip_cla` prop
 
 Replace `{stage}` with either `dev` or `prod`.
 
-This property is a Map attribute that contains mapping from repository pattern to bot GitHub login, email and name pattern.
+This property is a map attribute that contains mapping from repository pattern to bot GitHub login, email and name pattern.
 
 Example `login` is `lukaszgryglicki` (like any `login` that can be accessed via `https://github.com/login`).
 
 This is sometimes called `username` but we use `login` to avoid confusion with the `name` attribute.
 
 Example name is `"Lukasz Gryglicki"`.
 
-Email pattern and name pattern are optional and `*` is assumed for them if not specified.
+Email pattern and name pattern are optional and `""` (empty) is assumed for them if not specified.
 
 Each pattern is a string and can be one of three possible types (and are checked tin this order):
 - `"name"` - exact match for repository name, GitHub login, email address, GitHub name.
+- `""` - (empty string) pattern is special and it matches missing property, property with null value or property with empty string value.
 - `"re:regexp"` - regular expression match for repository name, GitHub login, name, or email address.
 - `"*"` - matches all.
 
 So the format is like `"repository_pattern": "login_pattern;email_pattern;name_pattern"`. `;` is used as a separator.
 
 You can also specify multiple patterns so different set is used for multiple users - in such case configuration must start with `[`, end with `]` and be `||` separated.
 
-For example: `"[copilot-swe-agent[bot];*;*||re:(?i)^l(ukasz)?gryglicki$;*;re:Gryglicki]"`.
+For example: `"[;*;copilot-swe-agent[bot];||re:(?i)^l(ukasz)?gryglicki$;*;re:Gryglicki]"`.
 
 Full format is like `"repository_pattern": "[login_pattern;email_pattern;name_pattern||..]"`.
 
@@ -48,7 +49,7 @@ Example:
     "skip_cla": {
       "M": {
         "*": {
-          "S": "*;re:^\\d+\\+Copilot@users\\.noreply\\.github\\.com$;*"
+          "S": ";re:^\\d+\\+Copilot@users\\.noreply\\.github\\.com$;copilot-swe-agent[bot]"
         },
         "re:(?i)^repo[0-9]+$": {
           "S": "re:vee?rendra;*;*"
@@ -65,19 +66,19 @@ Algorithm to match pattern is as follows:
 - First we check repository name for exact match. Repository name is without the organization name, so for `https://github.com/linuxfoundation/easycla` it is just `easycla`. If we find an entry in `skip_cla` for `easycla` that entry is used and we stop searching.
 - If no exact match is found, we check for regular expression match. Only keys starting with `re:` are considered. If we find a match, we use that entry and stop searching.
 - If no match is found, we check for `*` entry. If it exists, we use that entry and stop searching.
-- If no match is found, we don't skip CLA check.
+- If no match is found, we don't skip CLA check for any author.
 - Now when we have the entry, it is in the following format: `login_pattern;email_pattern;name_pattern` or `"[login_pattern;email_pattern;name_pattern||...]" (array)`.
-- We check GitHub login, email address and name against the patterns. Algorithm is the same - login, email and name patterns can be either direct match or `re:regexp` or `*`.
+- We check GitHub login, email address and name against the patterns. Algorithm is the same - login, email and name patterns can be either direct match ("" is a special case that also matches missing or null) or `re:regexp` or `*`.
 - If login, email and name match the patterns, we skip CLA check. If login, email or name is not set but the pattern is `*` it means hit.
-- So setting pattern to `login_pattern;*;*` or `login_pattern` (which is equivalent) means that we only check for login match and assume all emails and names are valid.
+- So setting pattern to `login_pattern;*;*` means that we only check for login match and assume all emails and names are valid.
 - Any actor that matches any of the entries in the array will be skipped (logical OR).
 - If we set `repo_pattern` to `*` it means that this configuration applies to all repositories in the organization.
 - If there are also specific repository patterns, they will be used instead of `*` (fallback for all).
 
 
 There is a script that allows you to update the `skip_cla` property in the DynamoDB table. It is located in `utils/skip_cla_entry.sh`. You can run it like this:
 - `` MODE=mode ./utils/skip_cla_entry.sh 'org-name' 'repo-pattern' 'login-pattern;email-pattern;name_pattern' ``.
-- `` MODE=add-key ./utils/skip_cla_entry.sh 'sun-test-org' '*' 'copilot-swe-agent[bot];*;*' ``.
+- `` MODE=add-key ./utils/skip_cla_entry.sh 'sun-test-org' '*' ';*;copilot-swe-agent[bot]' ``.
 - Complex example: `` MODE=add-key ./utils/skip_cla_entry.sh 'sun-test-org' 're:(?i)^repo[0-9]+$' '[re:(?i)^l(ukasz)?gryglicki$;re:(?i)^l(ukasz)?gryglicki@;*||copilot-swe-agent[bot]]' ``.
 
 `MODE` can be one of:
@@ -107,7 +108,7 @@ aws --profile "lfproduct-prod" --region "us-east-1" dynamodb update-item \
   --key '{"organization_name": {"S": "linuxfoundation"}}' \
   --update-expression "SET skip_cla.#repo = :val" \
   --expression-attribute-names '{"#repo": "re:^easycla"}' \
-  --expression-attribute-values '{":val": {"S": "some-github-login"}}'
+  --expression-attribute-values '{":val": {"S": "some-github-login;*;*"}}'
 ```
 
 To delete a key from an existing `skip_cla` entry:
@@ -143,14 +144,14 @@ To check for log entries related to skipping CLA check, you can use the followin
 
 To add first `skip_cla` value for an organization:
 ```
-aws --profile lfproduct-prod --region us-east-1 dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "open-telemetry"}}' --update-expression 'SET skip_cla = :val' --expression-attribute-values '{":val": {"M": {"otel-arrow":{"S":"*;re:^\\d+\\+Copilot@users\\.noreply\\.github\\.com$;*"}}}}'
-aws --profile lfproduct-prod --region us-east-1 dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "openfga"}}' --update-expression 'SET skip_cla = :val' --expression-attribute-values '{":val": {"M": {"vscode-ext":{"S":"*;re:^\\d+\\+Copilot@users\\.noreply\\.github\\.com$;*"}}}}'
+aws --profile lfproduct-prod --region us-east-1 dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "open-telemetry"}}' --update-expression 'SET skip_cla = :val' --expression-attribute-values '{":val": {"M": {"otel-arrow":{"S":";re:^\\d+\\+Copilot@users\\.noreply\\.github\\.com$;copilot-swe-agent[bot]"}}}}'
+aws --profile lfproduct-prod --region us-east-1 dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "openfga"}}' --update-expression 'SET skip_cla = :val' --expression-attribute-values '{":val": {"M": {"vscode-ext":{"S":";re:^\\d+\\+Copilot@users\\.noreply\\.github\\.com$;copilot-swe-agent[bot]"}}}}'
 ```
 
 To add additional repositories entries without overwriting the existing `skip_cla` value:
 ```
-aws --profile lfproduct-prod --region us-east-1 dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "open-telemetry"}}' --update-expression 'SET skip_cla.#repo = :val' --expression-attribute-names '{"#repo": "*"}' --expression-attribute-values '{":val": {"S": "*;re:^\\d+\\+Copilot@users\\.noreply\\.github\\.com$"}}'
-aws --profile lfproduct-prod --region us-east-1 dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "openfga"}}' --update-expression 'SET skip_cla.#repo = :val' --expression-attribute-names '{"#repo": "*"}' --expression-attribute-values '{":val": {"S": "*;re:^\\d+\\+Copilot@users\\.noreply\\.github\\.com$"}}'
+aws --profile lfproduct-prod --region us-east-1 dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "open-telemetry"}}' --update-expression 'SET skip_cla.#repo = :val' --expression-attribute-names '{"#repo": "*"}' --expression-attribute-values '{":val": {"S": ";re:^\\d+\\+Copilot@users\\.noreply\\.github\\.com$;copilot-swe-agent[bot]"}}'
+aws --profile lfproduct-prod --region us-east-1 dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "openfga"}}' --update-expression 'SET skip_cla.#repo = :val' --expression-attribute-names '{"#repo": "*"}' --expression-attribute-values '{":val": {"S": ";re:^\\d+\\+Copilot@users\\.noreply\\.github\\.com$;copilot-swe-agent[bot]"}}'
 ```
 
 To delete a specific repo entry from `skip_cla`:
@@ -175,6 +176,6 @@ aws --profile "lfproduct-prod" dynamodb scan --table-name "cla-prod-github-orgs"
 
 Typical adding a new entry for an organization:
 ```
-STAGE=prod MODE=add-key DEBUG=1 ./utils/skip_cla_entry.sh 'open-telemetry' 'opentelemetry-rust' '*;re:^\d+\+Copilot@users\.noreply\.github\.com$;copilot-swe-agent[bot]'
+STAGE=prod MODE=add-key DEBUG=1 ./utils/skip_cla_entry.sh 'open-telemetry' 'opentelemetry-rust' ';re:^\d+\+Copilot@users\.noreply\.github\.com$;copilot-swe-agent[bot]'
 ```
 

cla-backend-go/github/bots.go

@@ -16,6 +16,7 @@ import (
 
 // propertyMatches returns true if value matches the pattern.
 // - "*" matches anything
+// - "" matches empty value
 // - "re:..." matches regex (value must be non-empty)
 // - otherwise, exact match
 func propertyMatches(pattern, value string) bool {
@@ -27,6 +28,9 @@ func propertyMatches(pattern, value string) bool {
 	if pattern == "*" {
 		return true
 	}
+	if pattern == "" && value == "" {
+		return true
+	}
 	if value == "" {
 		return false
 	}
@@ -54,12 +58,12 @@ func stripOrg(repoFull string) string {
 
 // isActorSkipped returns true if the actor should be skipped according to ANY pattern in config.
 // Each config entry is "<login_pattern>;<email_pattern>;<name_pattern>"
-// Any missing pattern defaults to "*"
+// Any missing pattern defaults to "" which is special and matches missing property, null property value or empty string property value
 func isActorSkipped(actor *UserCommitSummary, config []string) bool {
 	for _, pattern := range config {
 		parts := strings.Split(pattern, ";")
 		for len(parts) < 3 {
-			parts = append(parts, "*")
+			parts = append(parts, "")
 		}
 		loginPattern, emailPattern, namePattern := parts[0], parts[1], parts[2]
 
@@ -143,9 +147,10 @@ func parseConfigPatterns(config string) []string {
 //   - repo-name is the exact repository name under given org (e.g., "my-repo" not "my-org/my-repo")
 //   - re:repo-regexp is a regex pattern to match repository names
 //   - * is a wildcard that applies to all repositories
-//   - <login_pattern> is a GitHub login pattern (exact match or regex prefixed by re: or match all '*')
-//   - <email_pattern> is a GitHub email pattern (exact match or regex prefixed by re: or match all '*') if not specified defaults to '*'
-//   - <name_pattern> is a GitHub name pattern (exact match or regex prefixed by re: or match all '*') if not specified defaults to '*'
+//   - <login_pattern> is a GitHub login pattern (exact match or regex prefixed by re: or match all '*') if not specified defaults to ""
+//   - <email_pattern> is a GitHub email pattern (exact match or regex prefixed by re: or match all '*') if not specified defaults to ""
+//   - <name_pattern> is a GitHub name pattern (exact match or regex prefixed by re: or match all '*') if not specified defaults to ""
+//     "" matches empty value, null value or missing property
 //     The login, email and name patterns are separated by a semicolon (;). Email and name parts are optional.
 //     There can be an array of patterns for a single repository, separated by ||. It must start with a '[' and end with a ']': "[...||...||...]"
 //     If the skip_cla is not set, it will skip the allowlisted bots check.

cla-backend-go/swagger/common/github-organization.yaml

@@ -77,24 +77,26 @@ properties:
       Map of repository name or pattern (e.g. 'repo1', '*', 're:pattern') to a string or array-string of pattern entries for skipping CLA checks for certain bots.
 
       Each value can be either:
-      - A string in the form '<login_pattern>;<email_pattern>;<name_pattern>' (email and name patterns are optional, default to '*').
+      - A string in the form '<login_pattern>;<email_pattern>;<name_pattern>' (email and name patterns are optional, default to '').
       - Or an OR-array in the form '[<entry1>||<entry2>||...]', where each entry uses the same pattern format above.
 
       Patterns can be:
       - An exact match (e.g. 'repo1', 'login', 'Name Surname', 'email@domain').
+      - A special case of exact match is '' pattern - it matches empty string, null property value or missing property.
       - A regular expression prefixed with 're:' (e.g. 're:(?i)^bot.*$').
       - A wildcard '*' to match all.
 
       Example formats:
-      - "copilot-swe-agent[bot];*;*"
+      - ";*;copilot-swe-agent[bot]"
       - "re:vee?rendra;*;*"
       - "[re:(?i)^l(ukasz)?gryglicki$;re:(?i)^l(ukasz)?gryglicki@;*||copilot-swe-agent[bot]]"
-      - "login;*"
+      - "login;*;*"
       - "login;[email protected];Real Name"
     example:
-      "*": "copilot-swe-agent[bot];*;*"
-      "repo1": "re:vee?rendra;*;*"
-      "re:(?i)^repo[0-9]+$": "[re:(?i)^l(ukasz)?gryglicki$;re:(?i)^l(ukasz)?gryglicki@;*||copilot-swe-agent[bot]]"
+      '*': 'some-bot-login;*;*'
+      'repo1': 're:vee?rendra;*;*'
+      're:(?i)^repo[0-9]+$': '[re:(?i)^l(ukasz)?gryglicki$;re:(?i)^l(ukasz)?gryglicki@;*||;re:^\d+\+Copilot@users\.noreply\.github\.com$;copilot-swe-agent[bot]]'
+
   repositories:
     type: object
     properties:

cla-backend/cla/models/github_models.py

@@ -932,12 +932,15 @@ def property_matches(self, pattern, value):
         """
         Returns True if value matches the pattern.
         - '*' matches anything
+        - '' matches None or empty string
         - 're:...' matches regex - value must be set
         - otherwise, exact match
         """
         try:
             if pattern == '*':
                 return True
+            if pattern == '' and (value is None or value == ''):
+                return True
             if value is None or value == '':
                 return False
             if pattern.startswith('re:'):
@@ -952,7 +955,7 @@ def is_actor_skipped(self, actor, config):
         """
         Returns True if the actor should be skipped (allowlisted) based on config pattern.
         config: '<login_pattern>;<email_pattern>;<name_pattern>'
-        If any pattern is missing, it defaults to '*'
+        If any pattern is missing, it defaults to '' which is special and matches None or empty string.
         It returns true if ANY config entry matches or false if there is no match in any config entry.
         """
         try:
@@ -965,7 +968,7 @@ def is_actor_skipped(self, actor, config):
             # Otherwise, treat as string pattern
             parts = config.split(';')
             while len(parts) < 3:
-                parts.append('*')
+                parts.append('')
             login_pattern, email_pattern, name_pattern = parts[:3]
             login = getattr(actor, "author_login", None)
             email = getattr(actor, "author_email", None)
@@ -1028,9 +1031,10 @@ def skip_allowlisted_bots(self, org_model, org_repo, actors_missing_cla) -> Tupl
         - repo-name is the exact repository name under given org (e.g., "my-repo" not "my-org/my-repo")
         - re:repo-regexp is a regex pattern to match repository names
         - * is a wildcard that applies to all repositories
-        - <login_pattern> is a GitHub login pattern (exact match or regex prefixed by re: or match all '*')
-        - <email_pattern> is a GitHub email pattern (exact match or regex prefixed by re: or match all '*') - defaults to '*' if not set
-        - <name_pattern> is a GitHub name pattern (exact match or regex prefixed by re: or match all '*') - defaults to '*' if not set
+        - <login_pattern> is a GitHub login pattern (exact match or regex prefixed by re: or match all '*') - defaults to '' if not set
+        - <email_pattern> is a GitHub email pattern (exact match or regex prefixed by re: or match all '*') - defaults to '' if not set
+        - <name_pattern> is a GitHub name pattern (exact match or regex prefixed by re: or match all '*') - defaults to '' if not set
+        :note: '' is a special pattern that matches None or empty string.
         :note: The login (sometimes called username it's the same), email and name patterns are separated by a semicolon (;).
         :note: There can be an array of patterns - it must start with [ and with ] and be || separated.
         :note: If the skip_cla is not set, it will skip the allowlisted bots check.

utils/skip_cla_entry.sh

@@ -11,6 +11,7 @@
 # ./utils/scan.sh github-orgs organization_name sun-test-org
 # STAGE=dev DTFROM='1 hour ago' DTTO='1 second ago' ./utils/search_aws_log_group.sh 'cla-backend-dev-githubactivity' 'skip_cla'
 # MODE=delete-key ./utils/skip_cla_entry.sh 'sun-test-org' 're:(?i)^repo[0-9]+$'
+# STAGE=dev MODE=add-key DEBUG=1 ./utils/skip_cla_entry.sh 'sun-test-org' 'repo1' 'thakurveerendras;;*'
 # STAGE=prod MODE=add-key DEBUG=1 ./utils/skip_cla_entry.sh 'open-telemetry' 'opentelemetry-rust' '*;re:^\d+\+Copilot@users\.noreply\.github\.com$;copilot-swe-agent[bot]'
 
 if [ -z "$MODE" ]