Copilot (app/copilot-swe-agent) EFO Ontology PR Analysis

[dragon-ai-agent]

This is a report of hits/failures for agent use in EFO, written by an agent on behalf of @cmungall. Feel free to close if not useful. For broader context see ai4curation/aidocs#60

Overall Statistics

Status	Count	Percentage
Merged	20	39.2%
Closed (not merged)	28	54.9%
Open	3	5.9%
Total	51	100%

Success rate of closed PRs: 20/48 = 41.7%

First-Try Success Rate

From sample analysis of extracted data, all 20 merged copilot PRs are categorized as merged_with_mods.

First-try success rate: ~0% (all merged PRs required modifications)

Note: Full analysis limited by file size (16GB extracted data file).

Failed PRs Checklist

Duplicate PRs (Same Issue, Multiple Attempts)

Issue #2490 - CHEBI terms for 'response to' EFO terms (13 PRs)

• PR Add has_primary_input axioms to EFO 'response to' terms linking to CHEBI #2491 - Add has_primary_input axioms to EFO 'response to' terms linking to CHEBI
• PR Add CHEBI terms to 'response to ...' EFO terms using has_primary_input relationships #2493 - Add CHEBI terms to 'response to ...' EFO terms using has_primary_input relationships
• PR Add has_primary_input axioms to response to terms with exact CHEBI matches #2494 - Add has_primary_input axioms to response to terms with exact CHEBI matches
• PR Systematic evaluation and addition of has_primary_input axioms to response_to terms with exact CHEBI matches #2497 - Systematic evaluation and addition of has_primary_input axioms
• PR Add has_primary_input axioms to 31 'response to' EFO terms with matching CHEBI terms #2501 - Add has_primary_input axioms to 31 'response to' EFO terms
• PR Add has_primary_input axioms to 35 EFO 'response to' terms with CHEBI matches #2502 - Add has_primary_input axioms to 35 EFO 'response to' terms
• PR Add CHEBI has_primary_input axioms to 25 'response to' EFO terms #2505 - Add CHEBI has_primary_input axioms to 25 'response to' EFO terms
• PR Add CHEBI has_primary_input axioms to 'response to' EFO terms #2507 - Add CHEBI has_primary_input axioms to 'response to' EFO terms
• PR Add RO:0004009 (has_primary_input) relationships to 'response to' EFO terms with CHEBI mappings #2509 - Add RO:0004009 (has_primary_input) relationships to 'response to' EFO terms
• PR Add has_primary_input relationships to 44 'response to' EFO terms with CHEBI entities #2511 - Add has_primary_input relationships to 44 'response to' EFO terms
• PR Add RO:0004009 (has_primary_input) relationships to 16 'response to' EFO terms #2514 - Add RO:0004009 (has_primary_input) relationships to 16 'response to' EFO terms
• PR Add has_primary_input (RO:0004009) axioms to chemical response terms #2558 - Add has_primary_input (RO:0004009) axioms to chemical response terms
• PR Add RO:0004009 (has_primary_input) axioms linking 'response to' terms to CHEBI chemicals #2559 - Add RO:0004009 (has_primary_input) axioms linking 'response to' terms to CHEBI

Issue #2546 - Bronchiectasis endotype terms (4 PRs)

• PR [WIP] Add new endotypes of bronchiectasis terms #2547 - [WIP] Add new endotypes of bronchiectasis terms
• PR Add bronchiectasis endotype terms from PMID:30215383 #2548 - Add bronchiectasis endotype terms from PMID:30215383
• PR [WIP] Add new endotypes of bronchiectasis terms #2553 - [WIP] Add new endotypes of bronchiectasis terms
• PR Add bronchiectasis inflammatory endotype terms from PMID:30215383 #2554 - Add bronchiectasis inflammatory endotype terms from PMID:30215383

Issue #2562 - GWAS drug response terms (4 PRs)

• PR [WIP] Add missing drug response terms for GWAS catalog #2563 - [WIP] Add missing drug response terms for GWAS catalog
• PR Add 8 drug response terms for GWAS Catalog with ChEBI linkage #2564 - Add 8 drug response terms for GWAS Catalog with ChEBI linkage
• PR Add drug response terms for GWAS catalog mapped to ChEBI via has_primary_input #2565 - Add drug response terms for GWAS catalog mapped to ChEBI
• PR [WIP] Add missing drug response terms for GWAS catalog #2569 - [WIP] Add missing drug response terms for GWAS catalog

Issue #2445 - Slide-seq update (2 PRs)

• PR Add bead-based spatial transcriptomics term and update Slide-seq hierarchy #2450 - Add bead-based spatial transcriptomics term and update Slide-seq hierarchy
• PR [WIP] [Update] Request for update to Slide-seq #2452 - [WIP] [Update] Request for update to Slide-seq

Other Failed PRs

• PR Import OBA body fat mass and tissue mass terms, update obsolete term replacement #2454 - Import OBA body fat mass and tissue mass terms (closed without explanation)
• PR [WIP] Using OLS, which term in NCIT is 'COVID-19 Terminology'? #2495 - [WIP] Using OLS, which term in NCIT is 'COVID-19 Terminology'? (query not implementation)
• PR Add DLCO change measurement term (EFO_0920004) #2532 - Add DLCO change measurement term (closed - similar term exists, needs discussion)
• PR Add mean diffusivity term and reparent mean fractional anisotropy measurement #2539 - Add mean diffusivity term (closed without explanation)
• PR Import 11 OBA protein measurement terms #2588 - Import 11 OBA protein measurement terms (most recent, closed without explanation)

Failure Pattern Summary

#	Failure Mode	PRs Affected	Count
1	Duplicate PRs for same issue	2491-2514, 2547-2554, 2558-2569	23
2	WIP placeholder PRs never completed	2452, 2495, 2547, 2553, 2563, 2569	6
3	Closed without clear resolution	2450, 2454, 2539, 2588	4
4	Scope/approach issues needing discussion	2532	1

Note: Many PRs appear in multiple categories (e.g., WIP and duplicate).

Detailed Failure Analysis Files

See individual analysis files:

• failure-analysis-2490-series.md - CHEBI terms duplicate PRs
• failure-analysis-2546-series.md - Bronchiectasis duplicate PRs
• failure-analysis-2562-series.md - GWAS terms duplicate PRs
• failure-analysis-2450.md - Slide-seq hierarchy
• failure-analysis-2454.md - OBA imports
• failure-analysis-2532.md - DLCO term
• failure-analysis-2539.md - Mean diffusivity
• failure-analysis-2588.md - OBA protein terms

Key Learnings

1. CRITICAL: Never create multiple PRs for the same issue - Agent created 13 PRs for issue Add CHEBI terms to 'response to ...' EFO terms #2490 alone
2. Update existing branches instead of creating new PRs when asked to revise
3. Complete WIP PRs before creating new ones - many [WIP] placeholders were abandoned
4. Discuss approach first for complex or contentious changes
5. Check for existing similar terms before adding new ones

[dragon-ai-agent]

Training Plan for Copilot on EFO Ontology

Executive Summary

Analysis of 51 PRs by app/copilot-swe-agent to EBISPOT/efo:

Status	Count	Percentage
Merged	20	39.2%
Closed (not merged)	28	54.9%
Open	3	5.9%
Total	51	100%

Success rate of closed PRs: 20/48 = 41.7%

This is significantly lower than comparable ontology work (GO: 77.6%). The primary cause is excessive duplicate PR creation - single issues generated 4-13 PRs each.

Caveat on silent closures: Several EFO PRs were closed without reviewer feedback. Treat these as uncertain root-cause cases and request clarification before updating training rules based on them.

---

Complete Failure Inventory

All 28 failed PRs are accounted for:

PR	Primary Failure Mode	Section
2491-2514, 2558-2559	Duplicate PRs (Issue #2490)	§1
2547, 2548, 2553, 2554	Duplicate PRs (Issue #2546)	§1
2563, 2564, 2565, 2569	Duplicate PRs (Issue #2562)	§1
2450, 2452	Duplicate PRs (Issue #2445)	§1
2452, 2495, 2547, 2553, 2563, 2569	WIP placeholders abandoned	§2
2532	Similar term exists, needs discussion	§3
2454, 2539, 2588	Closed without explanation	§4

---

Critical Failure Modes and Training Instructions

1. NEVER Create Duplicate PRs for Same Issue (23 PRs - 82% of failures)

Affected Issues:

• Issue Add CHEBI terms to 'response to ...' EFO terms #2490: 13 PRs created
• Issue [NTR] Request for new endotypes of bronchiectasis terms #2546: 4 PRs created
• Issue Add missing drug response terms for GWAS catalog #2562: 4 PRs created
• Issue [Update] Request for update to Slide-seq #2445: 2 PRs created

The Problem:
Agent created multiple PRs for the same issue, each starting fresh instead of updating existing work. Issue #2490 alone generated 13 PRs with varying term counts (4 to 79 terms).

Training Instructions:

RULE: ONE PR per issue - ALWAYS

Before starting work on any issue:
1. SEARCH for existing PRs on this issue
   Command: gh pr list -R EBISPOT/efo --search "in:title [issue keywords]"

2. If a PR exists:
   - Checkout that branch
   - Push new commits to it
   - NEVER create a new PR

3. If asked to revise an existing PR:
   - Checkout the existing branch
   - Make changes and push
   - NEVER create a new PR

Example:
- Reviewer: "Add more terms to this PR"
- CORRECT: git checkout existing-branch && <make changes> && git push
- WRONG: Create new PR with additional terms

4. If you must abandon a PR:
   - Comment explaining why
   - Close it explicitly
   - Only then create new PR if truly necessary

---

2. Complete WIP PRs - Never Abandon (6 PRs)

Affected PRs: 2452, 2495, 2547, 2553, 2563, 2569

The Problem:
Agent created [WIP] placeholder PRs but then started fresh in new PRs instead of completing them.

Training Instructions:

RULE: Never abandon WIP PRs

1. If you create a WIP PR:
   - You MUST complete it in subsequent sessions
   - NEVER create a new PR for the same issue

2. If blocked on a WIP PR:
   - Document the blocker in the PR
   - Ask for help
   - Keep the PR open

3. If you find an existing WIP PR for your assigned issue:
   - Continue work on that PR
   - Update the description with progress
   - Remove [WIP] when ready for review

4. WIP PRs are commitments - treat them as such

---

3. Check for Similar Existing Terms Before Adding New Ones (1 PR)

Affected PRs: 2532

The Problem:
PR #2532 proposed "DLCO change measurement" but reviewer noted similar term already exists (EFO_0009369 "diffusing capacity of the lung for carbon monoxide"). Existing term had issues needing resolution first.

Training Instructions:

RULE: Search before adding new terms

Before proposing ANY new term:
1. SEARCH for similar labels:
   - grep -i "term_name" src/ontology/efo-edit.owl
   - Search OLS for the concept

2. SEARCH for synonyms:
   - grep -i "alternative_name" src/ontology/efo-edit.owl

3. If similar terms exist:
   - Comment on the issue asking if new term is needed
   - Propose modifications to existing term instead
   - Wait for curator confirmation before creating new term

4. Check for related term issues:
   - Are there open issues about similar terms?
   - Are similar terms flagged as problematic?

Example search:
$ grep -i "diffus.*capacity\|DLCO" src/ontology/efo-edit.owl

---

4. Request Feedback on Silent PR Closures (3 PRs)

Affected PRs: 2454, 2539, 2588

The Problem:
Some PRs were closed without any comments or reviews, leaving no clear learning opportunity.

Training Instructions:

RULE: Always understand why a PR was closed

If a PR is closed without feedback:
1. Comment asking for clarification:
   "This PR was closed without review feedback. Could you help me understand:
    - Was there an issue with the approach?
    - Was this resolved via another PR?
    - What should I do differently next time?"

2. Check if issue was resolved elsewhere:
   - Look for merged PRs on same issue
   - Check if issue is still open

3. Learn from the closure to avoid repeating the same approach

---

Workflow Best Practices for EFO

Before Starting a Task

1. Check for existing PRs on the issue
2. Search for similar terms if adding new concepts
3. Read the issue thoroughly including all comments
4. Check for related open issues that might conflict

During Implementation

1. One PR per issue - always
2. Consistent methodology - same approach each run
3. Track term IDs - avoid proposing IDs already in use
4. Document blockers - if tools unavailable, say so

When Receiving Reviewer Feedback

1. Update existing branch - never create new PR
2. Address all comments - don't ignore any
3. Push commits to same PR - keep history clean
4. Ask for clarification if feedback is unclear

If PR is Closed

1. Request explanation if none given
2. Learn from feedback for next time
3. Check issue status - was it resolved?

---

Quick Reference Card

Scenario	Required Actions
Assigned new issue	1. Check for existing PRs, 2. Search similar terms, 3. Create ONE PR
Reviewer asks for changes	Update existing branch, push. NEVER create new PR
Blocked by tools/firewall	Document in PR comments, ask for help
Adding new term	Search for similar terms first, discuss if found
PR closed silently	Ask for feedback to understand why
Found existing WIP PR	Continue that PR, don't create new one

---

Success Metrics

Current state (51 PRs):

• Merged: 20 (39.2%)
• Failed: 28 (54.9%)
• Open: 3 (5.9%)

Current closed-PR success rate: 41.7% (20/48)

Target: 80%+ success rate on closed PRs

Improvement breakdown:

Fix	PRs Saved	New Success Rate
Eliminate duplicate PR creation	23	86% (20/23)
+ Complete WIP PRs	2	88% (22/25)
+ Check for similar terms	1	88%

Eliminating duplicate PRs alone would raise success rate from 41.7% to ~86%.

---

Appendix: PR Coverage Matrix

Issue	Duplicate PRs	WIP Abandoned	Similar Term	Silent Close
#2490	13 PRs
#2546	4 PRs	2
#2562	4 PRs	2
#2445	2 PRs	1
DLCO			✓
OBA				2 PRs
Misc				1 PR

---

Key Insight

The single most impactful change is: ONE PR PER ISSUE

23 of 28 failures (82%) were duplicate PRs. If the agent had updated existing branches instead of creating new PRs, success rate would jump from 41.7% to approximately 86%.

[aleixpuigb]

@dragon-ai-agent in you analysis you state that the first-try success rate is 0%, but there are PRs that were successfully closed in the first try (see #2583, #2581). Why those PRs were not considered?
Also, you could have a look in the agents file, since EFO is using a multi-agentic system.

[cmungall]

Let me investigate. The trainer agent is running off of deterministicalky generated jsonl summaries I’ll check if these are correct; but may be after holidays

…

On Tue, Dec 23, 2025 at 3:11 AM Aleix Puig ***@***.***> wrote: *aleixpuigb* left a comment (EBISPOT/efo#2592) <#2592 (comment)> @dragon-ai-agent <https://github.com/dragon-ai-agent> in you analysis you state that the first-try success rate is 0%, but there are PRs that were successfully closed in the first try (see #2583 <#2583>, #2581 <#2581>). Why those PRs were not considered? Also, you could have a look in the agents file, since EFO is using a multi-agentic system. — Reply to this email directly, view it on GitHub <#2592 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMMOKMGJTPTEUIR4Q2BED4DEPNNAVCNFSM6AAAAACPZINGU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMOBWGIZTEMBSGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>