Merge pull request #4747 from linuxfoundation/unicron-make-co-autors-support-conditional

Make co-authors support configurable per repo, or repo pattern or org-wide on github-org level

Author: lukaszgryglicki

CO_AUTHORS.md

@@ -0,0 +1,162 @@
+## Co-Authors support
+
+You can allow co-authors support on a specific repository, set of repositories matching regular expression pattern, or for all repositories in a GitHub organization.
+
+This can be done on the GitHub organization level by setting the `enable_co_authors` property on `cla-{stage}-github-orgs` DynamoDB table.
+
+Replace `{stage}` with either `dev` or `prod`.
+
+This property is a map attribute that contains mapping from repository pattern to co-authors support enabled or disabled.
+
+So the format is like `"repository_pattern": true|false`.
+
+There can be multiple entries under one Github Organization DynamoDB entry.
+
+Example:
+```
+{
+(...)
+    "organization_name": {
+      "S": "linuxfoundation"
+    },
+    "enable_co_authors": {
+      "M": {
+        "*": {
+          "BOOL": true,
+        },
+        "re:(?i)^repo[0-9]+$": {
+          "BOOL": false,
+        }
+      }
+    },
+(...)
+}
+```
+
+This enables co-authors support on all repos under "linuxfoundation", except for those matching the regular expression `re:(?i)^repo[0-9]+$`.
+
+Algorithm to match pattern is as follows:
+- First we check repository name for exact match. Repository name is without the organization name, so for `https://github.com/linuxfoundation/easycla` it is just `easycla`. If we find an entry in `enable_co_authors` for `easycla` that entry is used and we stop searching.
+- If no exact match is found, we check for regular expression match. Only keys starting with `re:` are considered. If we find a match, we use that entry and stop searching.
+- If no match is found, we check for `*` entry. If it exists, we use that entry and stop searching.
+- If no match is found, we don't support co-authors for that repository. Default is no co-author support.
+
+
+There is a script that allows you to update the `enable_co_authors` property in the DynamoDB table. It is located in `utils/enable_co_authors_entry.sh`. You can run it like this:
+- `` MODE=mode ./utils/enable_co_authors_entry.sh 'org-name' 'repo-pattern' t ``.
+- `` MODE=add-key ./utils/enable_co_authors_entry.sh 'sun-test-org' '*' f ``.
+
+`MODE` can be one of:
+- `put-item`: Overwrites/adds the entire `enable_co_authors` property. Needs all 3 arguments org, repo, and pattern.
+- `add-key`: Adds or updates a key/value inside the `enable_co_authors` map (preserves other keys). Needs all 3 args.
+- `delete-key`: Removes a key from the `enable_co_authors` map. Needs 2 arguments: org and repo.
+- `delete-item`: Deletes the entire `enable_co_authors` from the item. Needs 1 argument: org.
+
+
+You can also use AWS CLI to update the `enable_co_authors` property. Here is an example command:
+
+To add a new `enable_co_authors` entry:
+
+```
+aws --profile "lfproduct-prod" --region "us-east-1" dynamodb update-item \
+  --table-name "cla-prod-github-orgs" \
+  --key '{"organization_name": {"S": "linuxfoundation"}}' \
+  --update-expression 'SET enable_co_authors = :val' \
+  --expression-attribute-values '{":val": {"M": {"re:^easycla":{"BOOL": true}}}}'
+```
+
+To add a new key to an existing `enable_co_authors` entry (or replace the existing key):
+
+```
+aws --profile "lfproduct-prod" --region "us-east-1" dynamodb update-item \
+  --table-name "cla-prod-github-orgs" \
+  --key '{"organization_name": {"S": "linuxfoundation"}}' \
+  --update-expression "SET enable_co_authors.#repo = :val" \
+  --expression-attribute-names '{"#repo": "re:^easycla"}' \
+  --expression-attribute-values '{":val": {"BOOL": false}}'
+```
+
+To delete a key from an existing `enable_co_authors` entry:
+
+```
+aws --profile "lfproduct-prod" --region "us-east-1" dynamodb update-item \
+  --table-name "cla-prod-github-orgs" \
+  --key '{"organization_name": {"S": "linuxfoundation"}}' \
+  --update-expression "REMOVE enable_co_authors.#repo" \
+  --expression-attribute-names '{"#repo": "re:^easycla"}'
+```
+
+To delete the entire `enable_co_authors` entry:
+
+```
+aws --profile "lfproduct-prod" --region "us-east-1" dynamodb update-item \
+  --table-name "cla-prod-github-orgs" \
+  --key '{"organization_name": {"S": "linuxfoundation"}}' \
+  --update-expression "REMOVE enable_co_authors"
+```
+
+To see given organization's entry: `./utils/scan.sh github-orgs organization_name sun-test-org`.
+
+Or using AWS CLI:
+
+```
+aws --profile "lfproduct-prod" dynamodb scan --table-name "cla-prod-github-orgs" --filter-expression "contains(organization_name,:v)" --expression-attribute-values "{\":v\":{\"S\":\"linuxfoundation\"}}" --max-items 100 | jq -r '.Items'
+```
+
+To check for log entries related to skipping CLA check, you can use the following command: `` STAGE=dev DTFROM='1 hour ago' DTTO='1 second ago' ./utils/search_aws_log_group.sh 'cla-backend-dev-githubactivity' 'enable_co_authors' ``.
+
+# Example setup on prod
+
+To add first `enable_co_authors` value for an organization:
+```
+aws --profile lfproduct-prod --region us-east-1 dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "open-telemetry"}}' --update-expression 'SET enable_co_authors = :val' --expression-attribute-values '{":val": {"M": {"otel-arrow":{"BOOL":true}}}}'
+aws --profile lfproduct-prod --region us-east-1 dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "openfga"}}' --update-expression 'SET enable_co_authors = :val' --expression-attribute-values '{":val": {"M": {"vscode-ext":{"BOOL":true}}}}'
+```
+
+To add additional repositories entries without overwriting the existing `enable_co_authors` value:
+```
+aws --profile lfproduct-prod --region us-east-1 dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "open-telemetry"}}' --update-expression 'SET enable_co_authors.#repo = :val' --expression-attribute-names '{"#repo": "*"}' --expression-attribute-values '{":val": {"BOOL": true}}'
+aws --profile lfproduct-prod --region us-east-1 dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "openfga"}}' --update-expression 'SET enable_co_authors.#repo = :val' --expression-attribute-names '{"#repo": "*"}' --expression-attribute-values '{":val": {"BOOL": true}}'
+```
+
+To delete a specific repo entry from `enable_co_authors`:
+```
+aws --profile "lfproduct-prod" --region "us-east-1" dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "open-telemetry"}}' --update-expression 'REMOVE enable_co_authors.#repo' --expression-attribute-names '{"#repo": "*"}'
+aws --profile "lfproduct-prod" --region "us-east-1" dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "openfga"}}' --update-expression 'REMOVE enable_co_authors.#repo' --expression-attribute-names '{"#repo": "*"}'
+```
+
+To delete the entire `enable_co_authors` attribute:
+```
+aws --profile "lfproduct-prod" --region "us-east-1" dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "open-telemetry"}}' --update-expression 'REMOVE enable_co_authors'
+aws --profile "lfproduct-prod" --region "us-east-1" dynamodb update-item --table-name "cla-prod-github-orgs" --key '{"organization_name": {"S": "openfga"}}' --update-expression 'REMOVE enable_co_authors'
+```
+
+To check values:
+```
+aws --profile "lfproduct-prod" dynamodb scan --table-name "cla-prod-github-orgs" --filter-expression "contains(organization_name,:v)" --expression-attribute-values "{\":v\":{\"S\":\"open-telemetry\"}}" --max-items 100 | jq -r '.Items'
+aws --profile "lfproduct-prod" dynamodb scan --table-name "cla-prod-github-orgs" --filter-expression "contains(organization_name,:v)" --expression-attribute-values "{\":v\":{\"S\":\"openfga\"}}" --max-items 100 | jq -r '.Items'
+aws --profile "lfproduct-prod" dynamodb scan --table-name "cla-prod-github-orgs" --filter-expression "contains(organization_name,:v)" --expression-attribute-values "{\":v\":{\"S\":\"open-telemetry\"}}" --max-items 100 | jq -r '.Items[0].enable_co_authors.M["otel-arrow"]["BOOL"]'
+aws --profile "lfproduct-prod" dynamodb scan --table-name "cla-prod-github-orgs" --filter-expression "contains(organization_name,:v)" --expression-attribute-values "{\":v\":{\"S\":\"openfga\"}}" --max-items 100 | jq -r '.Items[0].enable_co_authors.M["vscode-ext"]["BOOL"]'
+```
+
+Typical adding a new entry for an organization:
+```
+STAGE=prod MODE=add-key DEBUG=1 ./utils/enable_co_authors_entry.sh 'open-telemetry' 'opentelemetry-rust' t
+```
+
+
+# How co-authors are processed
+
+When a commit is made to a repository that has co-authors support enabled, the backend will check if the commit message contains `co-authored-by:` lines/commit trailers (case insensitive).
+
+If it does, the backend will process the co-authors as follows, assume trailer value is `name <email>` like `Lukasz Gryglicki <[email protected]>`:
+
+- First we check if email is in format `[email protected]`. If it is we use number part as GitHub user ID and fetch the user from GitHub API. If the user is found, we use that user as co-author.
+
+- Second we check if email is in format `[email protected]`. If it is we use username part as GitHub username/login and fetch the user from GitHub API. If the user is found, we use that user as co-author.
+
+- Thirs we lookup for email using GitHub API. If the user is found, we use that user as co-author.
+
+- Finally we use the name part for `name <email>` and lookup using GitHub API assuming that this name is GitHub username/login (this is the case for some bots). If the user is found, we use that user as co-author.
+
+We use internal caching while doing all those lookups with cache key `name` and `email` and TTL 24 hours. We even cache by `(name, email)` when nothing is found because this is the most time consuming option. It will have a chance to be found in the future (up to 24 hours from lookup).

README.md

@@ -65,6 +65,10 @@ The following diagram explains the EasyCLA architecture.
 
 See [BOT_ALLOWLIST.md](BOT_ALLOWLIST.md) for information on configuring bots that are exempt from CLA checks.
 
+## Co-authors support
+
+See [CO_AUTHORS.md](CO_AUTHORS.md) for information on configuring co-authors support.
+
 ## EasyCLA Release Process
 
 The following diagram illustrates the EasyCLA release process:

cla-backend-go/github/bots.go

@@ -127,6 +127,54 @@ func parseConfigPatterns(config string) []string {
 	return []string{config}
 }
 
+// IsCoAuthorsEnabledForRepo returns whether co-authors are enabled for this repo
+func IsCoAuthorsEnabledForRepo(enableCoAuthors map[string]bool, orgRepo string) bool {
+	repo := stripOrg(orgRepo)
+	f := logrus.Fields{
+		"functionName": "github.IsCoAuthorsEnabledForRepo",
+		"orgRepo":      orgRepo,
+		"repo":         repo,
+	}
+	if enableCoAuthors == nil {
+		log.WithFields(f).Debugf("enable_co_authors is not set on '%s', skipping co-authors", orgRepo)
+		return false
+	}
+
+	// 1. Exact match
+	if v, ok := enableCoAuthors[repo]; ok {
+		log.WithFields(f).Debugf("enable_co_authors found for repo %s: %t (exact hit)", orgRepo, v)
+		return v
+	}
+
+	// 2. Regex pattern
+	log.WithFields(f).Debugf("No enable_co_authors found for repo %s, checking regex patterns", orgRepo)
+	for k, v := range enableCoAuthors {
+		if !strings.HasPrefix(k, "re:") {
+			continue
+		}
+		pattern := k[3:]
+		re, err := regexp.Compile(pattern)
+		if err != nil {
+			log.WithFields(f).Debugf("Invalid regex in enable_co_authors: %s (%v) for repo: %s", k, err, orgRepo)
+			continue
+		}
+		if re.MatchString(repo) {
+			log.WithFields(f).Debugf("Found enable_co_authors for repo %s: %t via regex pattern: %s", orgRepo, v, pattern)
+			return v
+		}
+	}
+
+	// 3. Wildcard fallback
+	if v, ok := enableCoAuthors["*"]; ok {
+		log.WithFields(f).Debugf("No enable_co_authors found for repo %s, using wildcard: %t", orgRepo, v)
+		return v
+	}
+
+	// 4. No match
+	log.WithFields(f).Debugf("No enable_co_authors found for repo %s, skipping co-authors", orgRepo)
+	return false
+}
+
 // SkipAllowlistedBots- check if the actors are allowlisted based on the skip_cla configuration.
 // Returns two lists:
 // - actors still missing cla: actors who still need to sign the CLA after checking skip_cla

cla-backend-go/github/github_repository.go

@@ -225,7 +225,7 @@ func (u UserCommitSummary) getUserInfo(tagUser bool) string {
 		tagValue = "@"
 	}
 	if u.CommitAuthor != nil {
-		if *u.CommitAuthor.Login != "" {
+		if u.CommitAuthor.Login != nil && *u.CommitAuthor.Login != "" {
 			sb.WriteString(fmt.Sprintf("login: %s%s / ", tagValue, *u.CommitAuthor.Login))
 		}
 
@@ -501,10 +501,11 @@ func GetCoAuthorCommits(
 	return summary
 }
 
-func GetPullRequestCommitAuthors(ctx context.Context, installationID int64, pullRequestID int, owner, repo string) ([]*UserCommitSummary, *string, error) {
+func GetPullRequestCommitAuthors(ctx context.Context, installationID int64, pullRequestID int, owner, repo string, withCoAuthors bool) ([]*UserCommitSummary, *string, error) {
 	f := logrus.Fields{
 		"functionName":  "github.github_repository.GetPullRequestCommitAuthors",
 		"pullRequestID": pullRequestID,
+		"withCoAuthors": withCoAuthors,
 	}
 	var userCommitSummary []*UserCommitSummary
 
@@ -554,7 +555,9 @@ func GetPullRequestCommitAuthors(ctx context.Context, installationID int64, pull
 			Affiliated:   false,
 			Authorized:   false,
 		})
-		ExpandWithCoAuthors(ctx, client, commit, pullRequestID, installationID, &userCommitSummary)
+		if withCoAuthors {
+			ExpandWithCoAuthors(ctx, client, commit, pullRequestID, installationID, &userCommitSummary)
+		}
 	}
 
 	// get latest commit SHA

cla-backend-go/github_organizations/models.go

@@ -20,6 +20,7 @@ type GithubOrganization struct {
 	AutoEnabledClaGroupID      string            `json:"auto_enabled_cla_group_id,omitempty"`
 	Version                    string            `json:"version,omitempty"`
 	SkipCLA                    map[string]string `json:"skip_cla,omitempty"`
+	EnableCoAuthors            map[string]bool   `json:"enable_co_authors,omitempty"`
 }
 
 // ToModel converts to models.GithubOrganization
@@ -37,6 +38,7 @@ func ToModel(in *GithubOrganization) *models.GithubOrganization {
 		BranchProtectionEnabled:    in.BranchProtectionEnabled,
 		ProjectSFID:                in.ProjectSFID,
 		SkipCla:                    in.SkipCLA,
+		EnableCoAuthors:            in.EnableCoAuthors,
 	}
 }
 

cla-backend-go/signatures/service.go

@@ -1021,8 +1021,9 @@ func (s service) updateChangeRequest(ctx context.Context, ghOrg *models.GithubOr
 	gitHubRepoName := utils.StringValue(githubRepository.Name)
 
 	// Fetch committers
+	withCoAuthors := github.IsCoAuthorsEnabledForRepo(ghOrg.EnableCoAuthors, gitHubRepoName)
 	log.WithFields(f).Debugf("fetching commit authors for PR: %d using repository owner: %s, repo: %s", pullRequestID, gitHubOrgName, gitHubRepoName)
-	authors, latestSHA, authorsErr := github.GetPullRequestCommitAuthors(ctx, ghOrg.OrganizationInstallationID, int(pullRequestID), gitHubOrgName, gitHubRepoName)
+	authors, latestSHA, authorsErr := github.GetPullRequestCommitAuthors(ctx, ghOrg.OrganizationInstallationID, int(pullRequestID), gitHubOrgName, gitHubRepoName, withCoAuthors)
 	if authorsErr != nil {
 		log.WithFields(f).WithError(authorsErr).Warnf("unable to get commit authors for %s/%s for PR: %d", gitHubOrgName, gitHubRepoName, pullRequestID)
 		return authorsErr

cla-backend-go/swagger/common/github-organization.yaml

@@ -96,6 +96,20 @@ properties:
       '*': 'some-bot-login;*;*'
       'repo1': 're:vee?rendra;*;*'
       're:(?i)^repo[0-9]+$': '[re:(?i)^l(ukasz)?gryglicki$;re:(?i)^l(ukasz)?gryglicki@;*||;re:^\d+\+Copilot@users\.noreply\.github\.com$;copilot-swe-agent[bot]]'
+  enableCoAuthors:
+    type: object
+    additionalProperties:
+      type: boolean
+    description: |
+      Map of repository name or pattern (e.g. 'repo1', '*', 're:pattern') to a bool value (true/false) specifying co-authors support for given repo(s). 
+      Patterns can be:
+      - An exact match (e.g. 'repo1').
+      - A regular expression prefixed with 're:' (e.g. 're:(?i)^repo[0-9]+$').
+      - A wildcard '*' to match all.
+    example:
+      '*': true
+      'repo1': false
+      're:(?i)^repo[0-9]+$': true
 
   repositories:
     type: object

cla-backend-go/v2/sign/helpers.go

@@ -16,6 +16,8 @@ import (
 )
 
 // updateChangeRequest is a helper function that updates PR - typically after the docusign is completed
+//
+//nolint:gocyclo
 func (s service) updateChangeRequest(ctx context.Context, installationID, repositoryID, pullRequestID int64, projectID string) error {
 	f := logrus.Fields{
 		"functionName":  "v1.signatures.service.updateChangeRequest",
@@ -54,8 +56,12 @@ func (s service) updateChangeRequest(ctx context.Context, installationID, reposi
 	gitHubRepoName := utils.StringValue(githubRepository.Name)
 
 	// Fetch committers
+	withCoAuthors := false
+	if ghOrg != nil {
+		withCoAuthors = github.IsCoAuthorsEnabledForRepo(ghOrg.EnableCoAuthors, gitHubRepoName)
+	}
 	log.WithFields(f).Debugf("fetching commit authors for PR: %d using repository owner: %s, repo: %s", pullRequestID, gitHubOrgName, gitHubRepoName)
-	authors, latestSHA, authorsErr := github.GetPullRequestCommitAuthors(ctx, installationID, int(pullRequestID), gitHubOrgName, gitHubRepoName)
+	authors, latestSHA, authorsErr := github.GetPullRequestCommitAuthors(ctx, installationID, int(pullRequestID), gitHubOrgName, gitHubRepoName, withCoAuthors)
 	if authorsErr != nil {
 		log.WithFields(f).WithError(authorsErr).Warnf("unable to get commit authors for %s/%s for PR: %d", gitHubOrgName, gitHubRepoName, pullRequestID)
 		return authorsErr

cla-backend/cla/models/dynamo_models.py

@@ -3913,6 +3913,7 @@ class Meta:
     enabled = BooleanAttribute(null=True)
     note = UnicodeAttribute(null=True)
     skip_cla = MapAttribute(of=UnicodeAttribute, null=True)
+    enable_co_authors = MapAttribute(of=BooleanAttribute, null=True)
 
 
 class GitHubOrg(model_interfaces.GitHubOrg):  # pylint: disable=too-many-public-methods
@@ -3922,7 +3923,8 @@ class GitHubOrg(model_interfaces.GitHubOrg):  # pylint: disable=too-many-public-
 
     def __init__(
             self, organization_name=None, organization_installation_id=None, organization_sfid=None,
-            auto_enabled=False, branch_protection_enabled=False, note=None, enabled=True, skip_cla=None,
+            auto_enabled=False, branch_protection_enabled=False, note=None, enabled=True,
+            skip_cla=None, enable_co_authors=None,
     ):
         super(GitHubOrg).__init__()
         self.model = GitHubOrgModel()
@@ -3936,6 +3938,7 @@ def __init__(
         self.model.note = note
         self.model.enabled = enabled
         self.model.skip_cla = skip_cla
+        self.model.enable_co_authors = enable_co_authors
 
     def __str__(self):
         return (
@@ -3948,7 +3951,8 @@ def __str__(self):
             f'branch_protection_enabled: {self.model.branch_protection_enabled},'
             f'note: {self.model.note},'
             f'enabled: {self.model.enabled},'
-            f'skip_cla: {self.model.skip_cla}'
+            f'skip_cla: {self.model.skip_cla},'
+            f'enable_co_authors: {self.model.enable_co_authors}'
         )
 
     def to_dict(self):
@@ -3997,6 +4001,9 @@ def get_branch_protection_enabled(self):
     def get_skip_cla(self):
         return self.model.skip_cla
 
+    def get_enable_co_authors(self):
+        return self.model.enable_co_authors
+
     def get_note(self):
         """
         Getter for the note.
@@ -4037,6 +4044,9 @@ def set_branch_protection_enabled(self, branch_protection_enabled):
     def set_skip_cla(self, skip_cla):
         self.model.skip_cla = skip_cla
 
+    def set_enable_co_authors(self, enable_co_authors):
+        self.model.enable_co_authors = enable_co_authors
+
     def set_note(self, note):
         self.model.note = note