feat: add daily token optimization advisor workflows#1620
Conversation
Add two new agentic workflows that read the latest token usage reports and produce actionable optimization recommendations: - copilot-token-optimizer: Reads token-usage-report issues, analyzes the most expensive Copilot workflow, and creates an issue with specific recommendations (tool trimming, pre-steps, prompt optimization, toolset restriction, cache improvements) - claude-token-optimizer: Same for Claude workflows, with additional Anthropic-specific cache write analysis (write vs read cost amortization, cache write rate metrics) Both workflows: - Run daily with skip-if-match to prevent duplicates - Auto-close older optimization issues (close-older-issues: true) - Download run artifacts to analyze actual tool usage vs loaded tools - Produce implementation checklists with estimated savings Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
There was a problem hiding this comment.
Pull request overview
Adds two new daily “token optimization advisor” agentic workflows that consume the latest token usage report issues and create actionable optimization recommendation issues for the most expensive workflow.
Changes:
- Introduces
copilot-token-optimizerworkflow prompt and compiled lock workflow. - Introduces
claude-token-optimizerworkflow prompt and compiled lock workflow. - Configures daily scheduling,
skip-if-match, safe-output issue creation, and auto-closing prior optimization issues.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| .github/workflows/copilot-token-optimizer.md | New Copilot optimizer agent prompt: locate latest report, analyze worst workflow + artifacts, and open an optimization issue. |
| .github/workflows/copilot-token-optimizer.lock.yml | Compiled GitHub Actions workflow for the Copilot optimizer. |
| .github/workflows/claude-token-optimizer.md | New Claude optimizer agent prompt, including Anthropic-specific cache write/read analysis guidance. |
| .github/workflows/claude-token-optimizer.lock.yml | Compiled GitHub Actions workflow for the Claude optimizer. |
Comments suppressed due to low confidence (6)
.github/workflows/copilot-token-optimizer.md:69
<workflow-name>is ambiguous here: the usage report may list the workflow display name (e.g. "Smoke Copilot") rather than the workflow file stem (e.g.smoke-copilot). As written,cat .github/workflows/<workflow-name>.mdwill fail for display names with spaces/case. Consider adding an explicit step to resolve the workflow file by matching thename:field inside.github/workflows/*.lock.yml(or by using the run URL/ID from the report directly) before trying to read the source file.
Read the workflow's `.md` source file to understand its configuration:
```bash
cat ".github/workflows/<workflow-name>.md"
**.github/workflows/claude-token-optimizer.md:70**
* Same `<workflow-name>` ambiguity as the Copilot optimizer: if the report contains a workflow display name, `.github/workflows/<workflow-name>.md` will not exist. Add a deterministic mapping step (e.g., find the workflow file whose `name:` matches the report, or use the run ID/URL from the report) before attempting to `cat` the `.md`.
Read the workflow's .md source file to understand its configuration:
cat ".github/workflows/<workflow-name>.md"**.github/workflows/copilot-token-optimizer.md:101**
* `grep` will exit with status 1 when there are no matching tool-call lines, and `find ... -exec grep ... \;` will propagate that non-zero status. In an agentic workflow this often gets treated as a command failure even though “no matches” is a valid outcome. Add `|| true` (and optionally `grep -h`) so the analysis doesn’t fail when there are simply no tool-call markers in the log.
Check agent stdio log for tool calls
find "$TMPDIR" -name "agent-stdio.log" -exec grep "^●" {} ;
**.github/workflows/claude-token-optimizer.md:101**
* Same `grep` exit-code problem: `find ... -exec grep ... \;` will fail the command when there are no matches. Add `|| true` (and optionally `-h`) so the workflow doesn’t treat “no tool calls found” as an error.
Check agent stdio log for tool calls
find "$TMPDIR" -name "agent-stdio.log" -exec grep "^●" {} ;
**.github/workflows/copilot-token-optimizer.md:88**
* `gh run list --workflow "<workflow-name>.lock.yml"` assumes the workflow identifier matches a `.lock.yml` filename derived from the report. In practice, the report may contain a workflow display name (with spaces) or a different identifier, so this lookup can return an empty list and leave `RUN_ID` blank. Consider deriving the workflow identifier from the report’s run URL/ID (preferred), or explicitly resolving workflow name ↔ file mapping before calling `gh run list`, and handle the “no successful run found” case gracefully.
Find the latest successful run
RUN_ID=$(gh run list --repo "$GITHUB_REPOSITORY"
--workflow ".lock.yml"
--status success --limit 1
--json databaseId --jq '.[0].databaseId')
**.github/workflows/claude-token-optimizer.md:88**
* `gh run list --workflow "<workflow-name>.lock.yml"` can fail to find runs if `<workflow-name>` comes from the report’s display name or otherwise doesn’t match the actual workflow filename, resulting in an empty/invalid `RUN_ID`. Consider using the run URL/ID already present in the report (or adding an explicit workflow-name→file resolution step) and add handling for “no successful run found”.
# Find the latest successful run
RUN_ID=$(gh run list --repo "$GITHUB_REPOSITORY" \
--workflow "<workflow-name>.lock.yml" \
--status success --limit 1 \
--json databaseId --jq '.[0].databaseId')💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| on: | ||
| schedule: daily | ||
| workflow_dispatch: | ||
| skip-if-match: | ||
| query: 'is:issue is:open label:claude-token-optimization' | ||
| max: 1 |
There was a problem hiding this comment.
schedule: daily compiles to a scattered cron, and the generated schedule currently runs before the Claude token usage analyzer. In the repo, claude-token-usage-analyzer.lock.yml is scheduled at 48 19 * * *, but this optimizer is at 33 12 * * *, so it may frequently run without a fresh report (or create recommendations from stale data). Consider switching to an explicit cron schedule (or a workflow_run trigger on the analyzer) so the optimizer reliably runs after the analyzer, consistent with the PR description.
| --json number,title,body,createdAt,url | ||
| ``` | ||
|
|
||
| If no report exists, create a brief issue stating that no token usage report was found and that the `copilot-token-usage-analyzer` workflow should run first. Then stop. |
There was a problem hiding this comment.
The instruction to create an issue when no report exists will create an open issue with the copilot-token-optimization label (via safe-outputs). Because skip-if-match blocks when any open issue has that label, the workflow can end up permanently skipping on subsequent days until someone manually closes that “no report found” issue. Prefer emitting a noop (report-as-issue) or creating an unlabelled/non-blocking issue in this branch so the optimizer can run automatically once reports start appearing.
This issue also appears in the following locations of the same file:
- line 65
- line 83
- line 99
| If no report exists, create a brief issue stating that no token usage report was found and that the `copilot-token-usage-analyzer` workflow should run first. Then stop. | |
| If no report exists, do **not** create a normal issue. Instead, emit a noop/report-as-issue style result stating that no token usage report was found and that the `copilot-token-usage-analyzer` workflow should run first. Then stop. |
| --json number,title,body,createdAt,url | ||
| ``` | ||
|
|
||
| If no report exists, create a brief issue stating that no token usage report was found and that the `claude-token-usage-analyzer` workflow should run first. Then stop. |
There was a problem hiding this comment.
Same issue as the Copilot optimizer: creating an issue when no report exists will leave an open issue with claude-token-optimization, which then causes skip-if-match to skip all future runs until manual intervention. Prefer noop (report-as-issue) or otherwise avoid creating an open, label-matching issue in the “no report found” path.
This issue also appears in the following locations of the same file:
- line 66
- line 83
- line 99
| If no report exists, create a brief issue stating that no token usage report was found and that the `claude-token-usage-analyzer` workflow should run first. Then stop. | |
| If no report exists, do **not** create an issue. Instead, return a no-op result briefly stating that no token usage report was found and that the `claude-token-usage-analyzer` workflow should run first. Then stop. |
|
Smoke Test: PASS
|
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
- Replace schedule: daily with workflow_run trigger (runs after analyzer completes, not before) - Add branches: [main] restriction for security - Fix no-report path to avoid creating blocking labeled issues (prevents skip-if-match from permanently blocking future runs) - Add workflow name resolution from display names to file paths (handles mismatch between report display names and file stems) - Add || true to grep commands to prevent false failures on no matches - Use -h flag on grep for cleaner multi-file output Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Smoke Test Results — PASS ✅ GitHub MCP: #1613 perf: reduce smoke-copilot token usage with pre-steps and tool trimming / #1446 fix: add picomatch overrides for GHSA-c2c7-rcm5-vvqj
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS
|
🤖 Smoke Test Results — Copilot Engine
Overall: PARTIAL — MCP + Bash verified; pre-step outputs not injected into prompt. PR by @lpcox · Reviewer:
|
Smoke Test: GitHub Actions Services Connectivity ✅All checks passed.
|
Smoke Test Results
|
Summary
Adds two daily agentic workflows that read the latest token usage reports and produce concrete optimization recommendations for the most expensive workflow.
New Workflows
copilot-token-optimizer— Readstoken-usage-reportissues, analyzes the most expensive Copilot workflow, downloads run artifacts to compare loaded vs used tools, and creates an issue with:claude-token-optimizer— Same pattern for Claude workflows (claude-token-usage-report), with additional Anthropic-specific analysis:Design
copilot-token-optimization/claude-token-optimizationclose-older-issues: true— auto-closes previous optimization issuesskip-if-match— prevents duplicate open issuesRelationship to Analyzers
Ref: #1604