Skip to content

Commit 0bf71e4

Browse files
jdiegosierraclaude
andcommitted
feat: add 4 new contributor quality metrics
Add Merger Diversity, Repo History, Profile Completeness, and Suspicious Patterns metrics to provide more comprehensive contributor analysis. New metrics: - Merger Diversity: tracks unique maintainers who merged contributor's PRs - Repo History: evaluates track record in the specific repository - Profile Completeness: assesses GitHub profile richness (bio, followers) - Suspicious Patterns: detects spam-like activity (auto-fail on critical) Changes: - Updated GraphQL queries to fetch mergedBy and profile fields - Added new metric implementations following existing patterns - Integrated all metrics into scoring engine - Added 5 new action inputs for threshold configuration - Updated README with metric documentation and input reference - All 167 tests passing Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 65e1ced commit 0bf71e4

20 files changed

Lines changed: 2334 additions & 108 deletions

README.md

Lines changed: 108 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -102,16 +102,20 @@ metrics** so maintainers can make informed decisions. It's the difference betwee
102102

103103
### Metrics
104104

105-
| Metric | Description | Default Threshold |
106-
| -------------------- | ---------------------------------------------- | ----------------- |
107-
| PR Merge Rate | Percentage of PRs that get merged vs closed | >= 0% |
108-
| Account Age | Age of the GitHub account | >= 0 days |
109-
| Positive Reactions | Positive reactions received on comments/issues | >= 0 |
110-
| Negative Reactions | Negative reactions received (maximum allowed) | <= 0 |
111-
| Repo Quality | Contributions to repos with stars | >= 0 |
112-
| Activity Consistency | Regular activity over time | >= 0% |
113-
| Issue Engagement | Issues created that receive engagement | >= 0 |
114-
| Code Reviews | Code reviews given to others | >= 0 |
105+
| Metric | Description | Default Threshold |
106+
| -------------------- | ----------------------------------------------- | ----------------- |
107+
| PR Merge Rate | Percentage of PRs that get merged vs closed | >= 0% |
108+
| Account Age | Age of the GitHub account | >= 0 days |
109+
| Positive Reactions | Positive reactions received on comments/issues | >= 0 |
110+
| Negative Reactions | Negative reactions received (maximum allowed) | <= 0 |
111+
| Repo Quality | Contributions to repos with stars | >= 0 |
112+
| Activity Consistency | Regular activity over time | >= 0% |
113+
| Issue Engagement | Issues created that receive engagement | >= 0 |
114+
| Code Reviews | Code reviews given to others | >= 0 |
115+
| Merger Diversity | Unique maintainers who merged contributor's PRs | >= 0 |
116+
| Repo History | Track record in the specific repository | >= 0 |
117+
| Profile Completeness | GitHub profile richness (bio, followers, etc.) | >= 0 |
118+
| Suspicious Patterns | Detection of spam-like activity patterns | N/A (auto) |
115119

116120
## Metric Details
117121

@@ -244,6 +248,70 @@ Reviewers who help others improve their code are valuable contributors. Spam acc
244248
- Look for opportunities to help other contributors
245249
- Share your knowledge and expertise
246250

251+
### Merger Diversity
252+
253+
**What it measures:** The number of unique maintainers who have merged the contributor's pull requests.
254+
255+
**Why it matters:** Having PRs merged by different maintainers across various projects demonstrates trust from the
256+
community. Contributors who only self-merge their own PRs on their own repositories haven't demonstrated external trust.
257+
258+
**How it's calculated:** Count of unique GitHub users who merged the contributor's PRs. Self-merges on own repositories
259+
are tracked separately.
260+
261+
**How to improve:**
262+
263+
- Contribute to established projects where maintainers review and merge your work
264+
- Build relationships with project maintainers
265+
- Focus on quality contributions that get merged by others
266+
267+
### Repo History
268+
269+
**What it measures:** The contributor's track record in the specific repository receiving the PR.
270+
271+
**Why it matters:** A history of successful contributions to a repository indicates familiarity with its codebase,
272+
contribution guidelines, and maintainer expectations.
273+
274+
**How it's calculated:** Merge rate and PR count specifically for the target repository.
275+
276+
**How to improve:**
277+
278+
- Start with smaller contributions to build trust
279+
- Follow the project's contribution guidelines carefully
280+
- Respond to reviewer feedback promptly
281+
282+
### Profile Completeness
283+
284+
**What it measures:** How complete the contributor's GitHub profile is (bio, company, followers, public repos).
285+
286+
**Why it matters:** A complete profile indicates a legitimate, invested GitHub user. Spam accounts typically have
287+
minimal or no profile information.
288+
289+
**How it's calculated:** Score based on presence of bio (20 points), company (20 points), followers (up to 40 points),
290+
and public repositories (up to 20 points).
291+
292+
**How to improve:**
293+
294+
- Add a bio describing yourself and your interests
295+
- Fill in your company or affiliation
296+
- Engage with the community to gain followers
297+
- Create and maintain public repositories
298+
299+
### Suspicious Patterns
300+
301+
**What it measures:** Detection of activity patterns commonly associated with spam accounts.
302+
303+
**Why it matters:** Certain combinations of behaviors (new account + high PR volume + many repos) are strong indicators
304+
of automated spam.
305+
306+
**Patterns detected:**
307+
308+
- **SPAM_PATTERN:** New account (<30 days) with >25 PRs across >10 repositories
309+
- **HIGH_PR_RATE:** More than 2 PRs per day on average
310+
- **SELF_MERGE_ABUSE:** High rate of self-merges on low-quality repositories
311+
- **REPO_SPAM:** Contributions to many repos with very low star counts
312+
313+
**Note:** This metric cannot be configured with a threshold. Critical patterns cause automatic failure.
314+
247315
## Usage
248316

249317
> **Note**: All metric thresholds default to `0`, making the action permissive by default. Configure stricter thresholds
@@ -287,6 +355,13 @@ jobs:
287355
threshold-account-age: '30' # Require 30+ day old accounts
288356
threshold-positive-reactions: '1' # Require at least 1 positive reaction
289357
threshold-negative-reactions: '5' # Allow max 5 negative reactions
358+
threshold-merger-diversity: '2' # Require 2+ unique maintainers who merged PRs
359+
threshold-repo-history-merge-rate: '0.3' # Require 30% merge rate in this repo
360+
threshold-repo-history-min-prs: '1' # Require at least 1 previous PR in repo
361+
threshold-profile-completeness: '20' # Require profile completeness score >= 20
362+
363+
# Spam detection (enabled by default)
364+
enable-spam-detection: 'true'
290365

291366
# Metrics that must pass (comma-separated)
292367
required-metrics: 'prMergeRate,accountAge'
@@ -321,24 +396,29 @@ jobs:
321396
322397
## Inputs
323398
324-
| Input | Required | Default | Description |
325-
| ------------------------------ | -------- | ------------------------ | ---------------------------------- |
326-
| `github-token` | Yes | `${{ github.token }}` | GitHub token for API access |
327-
| `thresholds` | No | `{}` | JSON object with custom thresholds |
328-
| `threshold-pr-merge-rate` | No | `0` | Minimum PR merge rate (0-1) |
329-
| `threshold-account-age` | No | `0` | Minimum account age in days |
330-
| `threshold-positive-reactions` | No | `0` | Minimum positive reactions |
331-
| `threshold-negative-reactions` | No | `0` | Maximum negative reactions |
332-
| `required-metrics` | No | `prMergeRate,accountAge` | Metrics that must pass |
333-
| `minimum-stars` | No | `100` | Min stars for quality repos |
334-
| `analysis-window` | No | `12` | Months of history to analyze |
335-
| `trusted-users` | No | Common bots | Comma-separated whitelist |
336-
| `trusted-orgs` | No | - | Comma-separated org whitelist |
337-
| `on-fail` | No | `comment` | Action when check fails |
338-
| `label-name` | No | `needs-review` | Label to apply |
339-
| `dry-run` | No | `false` | Log only, no actions |
340-
| `new-account-action` | No | `neutral` | Handling for new accounts |
341-
| `new-account-threshold-days` | No | `30` | Days to consider "new" |
399+
| Input | Required | Default | Description |
400+
| ----------------------------------- | -------- | ------------------------ | ------------------------------------- |
401+
| `github-token` | Yes | `${{ github.token }}` | GitHub token for API access |
402+
| `thresholds` | No | `{}` | JSON object with custom thresholds |
403+
| `threshold-pr-merge-rate` | No | `0` | Minimum PR merge rate (0-1) |
404+
| `threshold-account-age` | No | `0` | Minimum account age in days |
405+
| `threshold-positive-reactions` | No | `0` | Minimum positive reactions |
406+
| `threshold-negative-reactions` | No | `0` | Maximum negative reactions |
407+
| `threshold-merger-diversity` | No | `0` | Minimum unique maintainers who merged |
408+
| `threshold-repo-history-merge-rate` | No | `0` | Minimum merge rate in this repo (0-1) |
409+
| `threshold-repo-history-min-prs` | No | `0` | Minimum previous PRs in this repo |
410+
| `threshold-profile-completeness` | No | `0` | Minimum profile completeness (0-100) |
411+
| `enable-spam-detection` | No | `true` | Enable suspicious pattern detection |
412+
| `required-metrics` | No | `prMergeRate,accountAge` | Metrics that must pass |
413+
| `minimum-stars` | No | `100` | Min stars for quality repos |
414+
| `analysis-window` | No | `12` | Months of history to analyze |
415+
| `trusted-users` | No | Common bots | Comma-separated whitelist |
416+
| `trusted-orgs` | No | - | Comma-separated org whitelist |
417+
| `on-fail` | No | `comment` | Action when check fails |
418+
| `label-name` | No | `needs-review` | Label to apply |
419+
| `dry-run` | No | `false` | Log only, no actions |
420+
| `new-account-action` | No | `neutral` | Handling for new accounts |
421+
| `new-account-threshold-days` | No | `30` | Days to consider "new" |
342422

343423
## Outputs
344424

__fixtures__/testData.ts

Lines changed: 110 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,18 @@
22
* Shared test data factories to reduce duplication across tests
33
*/
44

5-
import type { GraphQLContributorData } from '../src/types/github.js'
5+
import type { GraphQLContributorData, PRContext } from '../src/types/github.js'
66
import type { ContributorQualityConfig } from '../src/types/config.js'
7-
import type { MetricCheckResult, ReactionData, PRHistoryData, AccountData } from '../src/types/metrics.js'
7+
import type {
8+
MetricCheckResult,
9+
ReactionData,
10+
PRHistoryData,
11+
AccountData,
12+
MergerDiversityData,
13+
RepoHistoryData,
14+
ProfileData,
15+
SuspiciousPatternData
16+
} from '../src/types/metrics.js'
817
import { DEFAULT_CONFIG } from '../src/config/defaults.js'
918

1019
/**
@@ -15,6 +24,12 @@ export function createContributorData(
1524
overrides: {
1625
login?: string
1726
createdAt?: Date
27+
bio?: string | null
28+
company?: string | null
29+
location?: string | null
30+
websiteUrl?: string | null
31+
followersCount?: number
32+
publicReposCount?: number
1833
pullRequests?: Partial<GraphQLContributorData['user']['pullRequests']>
1934
contributionsCollection?: Partial<GraphQLContributorData['user']['contributionsCollection']>
2035
issueComments?: Partial<GraphQLContributorData['user']['issueComments']>
@@ -27,6 +42,12 @@ export function createContributorData(
2742
user: {
2843
login: overrides.login ?? 'test-user',
2944
createdAt: (overrides.createdAt ?? defaultCreatedAt).toISOString(),
45+
bio: overrides.bio ?? 'A test user bio',
46+
company: overrides.company ?? 'Test Company',
47+
location: overrides.location ?? 'Test City',
48+
websiteUrl: overrides.websiteUrl ?? 'https://example.com',
49+
followers: { totalCount: overrides.followersCount ?? 50 },
50+
repositories: { totalCount: overrides.publicReposCount ?? 10 },
3051
pullRequests: {
3152
totalCount: 0,
3253
nodes: [],
@@ -65,10 +86,12 @@ export function createPRNode(
6586
deletions?: number
6687
owner?: string
6788
repo?: string
89+
mergedBy?: string | null
6890
} = {}
6991
): GraphQLContributorData['user']['pullRequests']['nodes'][0] {
7092
const state = overrides.state ?? 'MERGED'
7193
const merged = overrides.merged ?? state === 'MERGED'
94+
const defaultMergedBy = merged ? 'maintainer' : null
7295

7396
return {
7497
state,
@@ -78,6 +101,14 @@ export function createPRNode(
78101
closedAt: state !== 'OPEN' ? new Date().toISOString() : null,
79102
additions: overrides.additions ?? 50,
80103
deletions: overrides.deletions ?? 20,
104+
mergedBy:
105+
overrides.mergedBy !== undefined
106+
? overrides.mergedBy
107+
? { login: overrides.mergedBy }
108+
: null
109+
: defaultMergedBy
110+
? { login: defaultMergedBy }
111+
: null,
81112
repository: {
82113
owner: { login: overrides.owner ?? 'org' },
83114
name: overrides.repo ?? 'repo',
@@ -260,3 +291,80 @@ export function createLimitedDataContributor(): GraphQLContributorData {
260291
}
261292
})
262293
}
294+
295+
/**
296+
* Create MergerDiversityData for testing
297+
*/
298+
export function createMergerDiversityData(overrides: Partial<MergerDiversityData> = {}): MergerDiversityData {
299+
return {
300+
totalMergedPRs: 10,
301+
uniqueMergers: 3,
302+
selfMergeCount: 2,
303+
othersMergeCount: 8,
304+
selfMergesOnOwnRepos: 1,
305+
selfMergesOnExternalRepos: 1,
306+
externalReposWithMergePrivilege: ['org/repo'],
307+
onlySelfMergesOnOwnRepos: false,
308+
selfMergeRate: 0.2,
309+
mergerLogins: ['maintainer1', 'maintainer2', 'test-user'],
310+
...overrides
311+
}
312+
}
313+
314+
/**
315+
* Create RepoHistoryData for testing
316+
*/
317+
export function createRepoHistoryData(overrides: Partial<RepoHistoryData> = {}): RepoHistoryData {
318+
return {
319+
repoName: 'org/repo',
320+
totalPRsInRepo: 5,
321+
mergedPRsInRepo: 4,
322+
closedWithoutMergeInRepo: 1,
323+
repoMergeRate: 0.8,
324+
isFirstTimeContributor: false,
325+
...overrides
326+
}
327+
}
328+
329+
/**
330+
* Create ProfileData for testing
331+
*/
332+
export function createProfileData(overrides: Partial<ProfileData> = {}): ProfileData {
333+
return {
334+
followersCount: 50,
335+
publicReposCount: 10,
336+
hasBio: true,
337+
hasCompany: true,
338+
hasLocation: true,
339+
hasWebsite: true,
340+
completenessScore: 100,
341+
...overrides
342+
}
343+
}
344+
345+
/**
346+
* Create SuspiciousPatternData for testing
347+
*/
348+
export function createSuspiciousPatternData(overrides: Partial<SuspiciousPatternData> = {}): SuspiciousPatternData {
349+
return {
350+
detectedPatterns: [],
351+
prRate: 0.1,
352+
uniqueRepoCount: 5,
353+
selfMergeRate: 0.2,
354+
accountAgeInDays: 400,
355+
...overrides
356+
}
357+
}
358+
359+
/**
360+
* Create a PRContext for testing
361+
*/
362+
export function createPRContext(overrides: Partial<PRContext> = {}): PRContext {
363+
return {
364+
owner: 'test-org',
365+
repo: 'test-repo',
366+
prNumber: 123,
367+
prAuthor: 'test-user',
368+
...overrides
369+
}
370+
}

0 commit comments

Comments
 (0)