Skip to content

ci: bootstrap trusted AI review verifier - #174

Merged
safal207 merged 4 commits into
mainfrom
ci/bootstrap-trusted-ai-review-verifier
Jul 6, 2026
Merged

ci: bootstrap trusted AI review verifier#174
safal207 merged 4 commits into
mainfrom
ci/bootstrap-trusted-ai-review-verifier

Conversation

@safal207

@safal207 safal207 commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Summary

Bootstrap the AI review verifier onto the trusted default branch before product PR #173 is allowed to execute it.

This PR intentionally adds exactly one file:

  • scripts/verify-ai-review-contract.cjs

Trust boundary

PR #173 currently checks out and executes its verifier from the PR head. That is self-referential: a pull request can modify the code that decides whether the same pull request passed its mandatory review gate.

After this bootstrap PR is merged, #173 will be changed to check out the verifier from its immutable base SHA/default branch instead of the untrusted PR head.

Verifier contract

The bootstrapped verifier:

  • anchors freshness to the immutable workflow-run creation time;
  • requires a trusted request created after that anchor and naming the full exact head SHA;
  • ignores edited request timestamps;
  • accepts only active submitted CodeRabbit reviews bound to the exact commit, or a successful bot-authored CodeRabbit commit status created in the same head-update window;
  • rejects pending, dismissed, maintainer-authored, summary-comment, and older-head evidence;
  • keeps Codex supplemental unless it publishes a native submitted exact-head review;
  • fails closed after a bounded 15-minute polling window.

Scope boundary

No product, CSS, service-worker, visual, generated, or integrity-manifest files are changed.

Merge order

  1. Review and merge this bootstrap PR.
  2. Rebase/update PR fix: correct Roby's wordmark colors #173 on the new main.
  3. Make fix: correct Roby's wordmark colors #173 execute the verifier from its immutable base SHA.
  4. Rerun exact-head CI and independent review for fix: correct Roby's wordmark colors #173.
  5. Merge fix: correct Roby's wordmark colors #173 only after all gates are green.

@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

No new commits to review since the last review.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 25fe8d11-b0ce-4cf7-91bd-ac22b3075a70

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Добавлен скрипт проверки exact-head AI-review evidence для PR: он опрашивает GitHub, сопоставляет свежие запросы review с review/status доказательствами CodeRabbit и Codex, затем завершает workflow успехом или ошибкой.

Changes

Проверка контракта AI-review

Layer / File(s) Summary
Константы и вспомогательные утилиты
scripts/verify-ai-review-contract.cjs
Определены доверенные логины и ассоциации, параметры poll-цикла и функции для парсинга времени, фильтрации active submitted review, извлечения команд из body и распознавания permission/credential ошибок.
Основной цикл опроса и поиска доказательств
scripts/verify-ai-review-contract.cjs
Реализована verifyAiReviewContract: проверка pull_request.head.sha, вычисление currentHead и headUpdateAnchor, параллельный опрос comments/reviews/statuses, поиск exact-head evidence для CodeRabbit и supplemental evidence для Codex, обработка transient и permission ошибок.
Успех и повторные попытки
scripts/verify-ai-review-contract.cjs
При совпадении условий формируется core.summary, вызывается core.notice и выполняется ранний return; при отсутствии evidence пишется core.info и выполняется ожидание между попытками.
Итоговая ошибка
scripts/verify-ai-review-contract.cjs
После исчерпания попыток вызывается core.setFailed с требованием trusted exact-head @coderabbitai review запроса и native CodeRabbit evidence для того же head, с добавлением последней transient ошибки при наличии.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related issues

Possibly related PRs

  • safal207/robys-coffee-house-demo#144: Схожая логика проверки AI-review evidence и разбора команд @coderabbitai review / @codex review для текущего head SHA.

Suggested labels: ci, automation, review-contract

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning Есть Summary и план, но отсутствуют обязательные Evidence, AI review, Solo maintainer decision и Checklist из шаблона. Добавьте разделы Evidence, AI review, Solo maintainer decision и Checklist, заполнив требуемые комментарии и проверяемые доказательства.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed Заголовок точно отражает bootstrap trusted AI review verifier и основной смысл изменений.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/bootstrap-trusted-ai-review-verifier

Comment @coderabbitai help to get the list of available commands.

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

CI: bootstrap trusted AI review verifier script

✨ Enhancement ⚙️ Configuration changes 🕐 40+ Minutes

Grey Divider

AI Description

• Add a trusted, default-branch AI review verifier to enforce exact-head CodeRabbit evidence.
• Anchor freshness to workflow-run creation time to avoid PR self-verification.
• Fail closed after bounded polling when required evidence is missing or stale.
Diagram

graph TD
  A["Workflow run (created_at)"] --> B["Verifier script"] --> C(["GitHub REST API"]) --> D{"Contract satisfied?"}
  D --> E["Pass: emit summary/notice"]
  D --> F["Fail closed after 15m"]
  subgraph Legend
    direction LR
    _proc["Process/step"] ~~~ _api(["API call"]) ~~~ _dec{"Decision"}
  end
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Rely solely on a required CodeRabbit check via branch protection
  • ➕ Simpler enforcement surface (no custom polling/logic)
  • ➕ Native GitHub protections; less custom code to maintain
  • ➖ May not enforce exact-head binding or trusted-request freshness semantics
  • ➖ Less control over excluded evidence types (edited comments, summaries, maintainer-authored triggers)
2. Publish a dedicated GitHub Check Run from a trusted workflow
  • ➕ First-class, explicit pass/fail artifact tied to the workflow run
  • ➕ Can encode the same contract while avoiding commit-status ambiguity
  • ➖ More setup complexity (check-run creation, permissions, naming conventions)
  • ➖ Still requires custom implementation; likely broader rollout changes
3. Move contract verification into a reusable workflow/action pinned by SHA
  • ➕ Clear trust boundary via pinning/versioning
  • ➕ Easier reuse across repositories or multiple workflows
  • ➖ Additional packaging/version management overhead
  • ➖ May be premature if this contract is repo-specific and still evolving

Recommendation: The PR’s approach (default-branch script anchored to immutable workflow-run creation time, failing closed) is appropriate for bootstrapping a trust boundary quickly. Consider evolving toward a pinned reusable workflow/check-run once the contract stabilizes and needs broader reuse.

Files changed (1) +194 / -0

Other (1) +194 / -0
verify-ai-review-contract.cjsAdd default-branch AI review contract verifier +194/-0

Add default-branch AI review contract verifier

• Introduces a CI verifier that anchors freshness to workflow-run creation time and requires a trusted exact-head @coderabbitai request plus native CodeRabbit evidence (submitted review or success status). Implements bounded polling with detailed job summary output and fails closed when evidence is missing, stale, or not exact-head.

scripts/verify-ai-review-contract.cjs

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

🚦 Roby’s Lighthouse contract

Mobile

Metric Current Baseline Delta Status
Performance 0.99 1.00 -1.00%
LCP 1793 ms 1746 ms +2.70%
TBT 0 ms 5 ms -100.00%
CLS 0.00 0.00 -0.00%
FCP 1398 ms
Speed Index 1398 ms
Total JS 21 KB 21 KB +0.11%
Hero file 78 KB 78 KB -0.15%
Hero transfer 0 KB
Hero request 0 ms

Hard assertions: ✅ · Regression: ✅
· Public Lighthouse report

Desktop

Metric Current Baseline Delta Status
Performance 1.00 1.00 0.00%
LCP 432 ms 460 ms -5.98%
TBT 0 ms 0 ms 0.00%
CLS 0.00 0.00 0.00%
FCP 380 ms
Speed Index 479 ms
Total JS 21 KB 21 KB +0.11%
Hero file 78 KB 78 KB -0.15%
Hero transfer 0 KB
Hero request 0 ms

Hard assertions: ✅ · Regression: ✅
· Public Lighthouse report

D.1 runs in observability mode. The final gate is enabled only after the reviewed baseline commit.

safal207 commented Jul 6, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

Exact head: d8992bf

@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail
gh pr view 174 --repo safal207/robys-coffee-house-demo --json headRefOid,headRefName,files,commits -q '{headSha: .headRefOid, files: [.files[].path]}'

Length of output: 424


@safal207 Confirmed: PR #174 head SHA d8992bf0dfbff9abe5373e84b8e65db99b2423ff matches your stated exact head, and the diff is scoped to exactly one file (scripts/verify-ai-review-contract.cjs), consistent with the bootstrap-only objective. Triggering review now.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@qodo-code-review

qodo-code-review Bot commented Jul 6, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used
✅ Compliance rules (platform): 11 rules

Grey Divider


Action required

1. Permission errors retried blindly ✓ Resolved 🐞 Bug ☼ Reliability
Description
verifyAiReviewContract() treats all GitHub API failures as transient, so missing GITHUB_TOKEN
permissions for Actions/statuses will be retried for ~15 minutes and then fail with a generic
message. This makes a deterministic misconfiguration look flaky and delays actionable feedback.
Code

scripts/verify-ai-review-contract.cjs[R72-149]

+    try {
+      if (headUpdateAnchor <= 0) {
+        const run = await github.rest.actions.getWorkflowRun({
+          owner,
+          repo,
+          run_id: context.runId,
+        });
+        headUpdateAnchor = parseTime(run.data.created_at);
+        if (headUpdateAnchor <= 0) {
+          throw new Error("workflow run has no immutable creation timestamp");
+        }
+      }
+
+      const [comments, reviews, statuses] = await Promise.all([
+        github.paginate(github.rest.issues.listComments, {
+          owner,
+          repo,
+          issue_number: pr.number,
+          per_page: 100,
+        }),
+        github.paginate(github.rest.pulls.listReviews, {
+          owner,
+          repo,
+          pull_number: pr.number,
+          per_page: 100,
+        }),
+        github.paginate(github.rest.repos.listCommitStatusesForRef, {
+          owner,
+          repo,
+          ref: currentHead,
+          per_page: 100,
+        }),
+      ]);
+
+      const freshRequest = (item, command) =>
+        TRUSTED_ASSOCIATIONS.has(item.author_association) &&
+        createdTimeOf(item) >= headUpdateAnchor &&
+        commandLinesOf(item).includes(command) &&
+        containsExactHead(item, currentHead);
+
+      codeRabbitRequestAt = latestCreatedTime(
+        comments.filter((item) => freshRequest(item, "@coderabbitai review")),
+      );
+      const codexRequestAt = latestCreatedTime(
+        comments.filter((item) => freshRequest(item, "@codex review")),
+      );
+
+      codeRabbitReview = reviews.find(
+        (review) =>
+          CODERABBIT_LOGINS.has(review.user?.login) &&
+          review.commit_id?.toLowerCase() === currentHead &&
+          isSubmittedActiveReview(review) &&
+          codeRabbitRequestAt > 0 &&
+          submittedTimeOf(review) >= codeRabbitRequestAt,
+      );
+
+      codeRabbitStatus = statuses.find(
+        (status) =>
+          status.context === CODERABBIT_STATUS_CONTEXT &&
+          status.state === "success" &&
+          CODERABBIT_LOGINS.has(status.creator?.login) &&
+          codeRabbitRequestAt > 0 &&
+          createdTimeOf(status) >= headUpdateAnchor,
+      );
+
+      nativeCodexReview = reviews.find(
+        (review) =>
+          CODEX_LOGINS.has(review.user?.login) &&
+          review.commit_id?.toLowerCase() === currentHead &&
+          isSubmittedActiveReview(review) &&
+          codexRequestAt > 0 &&
+          submittedTimeOf(review) >= codexRequestAt,
+      );
+    } catch (error) {
+      core.warning(
+        `Transient GitHub API error (${attempt}/${POLL_ATTEMPTS}): ${error.message}`,
+      );
+    }
Relevance

⭐⭐⭐ High

Team previously hardened permission-check paths to fail closed vs flaky behavior (accepted in
maintainer attestation workflow).

PR-#123

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The verifier requires Actions and commit-status APIs but currently treats any resulting permission
failures as transient, while the repo’s existing workflow permissions omit the required scopes.

scripts/verify-ai-review-contract.cjs[72-83]
scripts/verify-ai-review-contract.cjs[85-104]
scripts/verify-ai-review-contract.cjs[145-149]
.github/workflows/ai-review-contract.yml[7-11]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The verifier calls APIs that require additional workflow token permissions (`actions: read` for `actions.getWorkflowRun`, and typically `statuses: read` for `repos.listCommitStatusesForRef`). Today, all API errors are treated as transient and retried for the full polling window, which masks permission misconfigurations and wastes CI time.

### Issue Context
The repository’s existing AI review contract workflow uses explicit `permissions:` and does not include `actions:` or `statuses:`. When this verifier is wired in, a 403 will be logged as transient and retried until timeout.

### Fix Focus Areas
- scripts/verify-ai-review-contract.cjs[72-149]
- .github/workflows/ai-review-contract.yml[7-11]

### Suggested fix
1. In the `catch`, detect non-retryable auth/permission failures and fail immediately with a targeted message.
  - Example: if `error.status === 403` or the message contains `Resource not accessible by integration`, call `core.setFailed("Missing workflow permissions: add permissions: actions: read, statuses: read")` and `return`.
  - Keep bounded retries for network errors, 5xx, and 429.
2. Document (or enforce via a guard) the required `permissions:` block for workflows that invoke this verifier:
  - `actions: read`
  - `statuses: read`
  - plus existing `issues: read`, `pull-requests: read`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Status not after request ✓ Resolved 🐞 Bug ≡ Correctness
Description
CodeRabbit commit-status evidence is accepted if it was created after the workflow-run anchor, but
it is not required to be created after the latest trusted @coderabbitai request. This breaks the
“request → evidence” freshness chain enforced for PR reviews and can admit non-causal status
evidence within the same head-update window.
Code

scripts/verify-ai-review-contract.cjs[R128-135]

+      codeRabbitStatus = statuses.find(
+        (status) =>
+          status.context === CODERABBIT_STATUS_CONTEXT &&
+          status.state === "success" &&
+          CODERABBIT_LOGINS.has(status.creator?.login) &&
+          codeRabbitRequestAt > 0 &&
+          createdTimeOf(status) >= headUpdateAnchor,
+      );
Relevance

⭐⭐⭐ High

Team enforced request→evidence timestamp ordering for AI evidence (e.g., reactions constrained after
request/head cutoff).

PR-#114

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The review evidence path is explicitly bound to codeRabbitRequestAt, while the status evidence
path is only bound to the anchor timestamp, creating an inconsistency in freshness semantics.

scripts/verify-ai-review-contract.cjs[119-126]
scripts/verify-ai-review-contract.cjs[128-135]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The verifier enforces `submittedTimeOf(review) >= codeRabbitRequestAt` for PR review evidence, but for commit-status evidence it only enforces `createdTimeOf(status) >= headUpdateAnchor`. That allows passing with a qualifying status created before the latest trusted request (while still after the anchor).

### Issue Context
The goal is to ensure the trusted request precedes the accepted evidence. The status path currently doesn’t enforce that ordering.

### Fix Focus Areas
- scripts/verify-ai-review-contract.cjs[119-135]

### Suggested fix
Add a parallel freshness check for status evidence:
- require `createdTimeOf(status) >= codeRabbitRequestAt` (and keep `>= headUpdateAnchor`).

Optionally, consider taking the *latest* matching status explicitly (sort or reduce) rather than relying on API ordering.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. SHA required in request ✓ Resolved 🐞 Bug ≡ Correctness
Description
freshRequest() only recognizes a trusted request if the comment body contains the full exact head
SHA in addition to the command line. Current repo docs/templates describe posting the canonical
command alone, so users following existing guidance will generate requests the verifier ignores
(false negatives).
Code

scripts/verify-ai-review-contract.cjs[R106-110]

+      const freshRequest = (item, command) =>
+        TRUSTED_ASSOCIATIONS.has(item.author_association) &&
+        createdTimeOf(item) >= headUpdateAnchor &&
+        commandLinesOf(item).includes(command) &&
+        containsExactHead(item, currentHead);
Relevance

⭐⭐⭐ High

Repo repeatedly tightened AI-review commands/evidence to require exact current-head SHAs and updated
docs/templates accordingly.

PR-#114
PR-#121

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The verifier explicitly requires the head SHA in the request comment, but the repo’s documented
process and existing workflow request detection do not include that requirement.

scripts/verify-ai-review-contract.cjs[106-110]
scripts/verify-ai-review-contract.cjs[42-46]
.github/pull_request_template.md[11-20]
docs/ai-review-cooperation-policy.md[81-90]
.github/workflows/ai-review-contract.yml[44-53]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The verifier’s `freshRequest()` predicate requires `containsExactHead(item, currentHead)`, meaning the request comment must include the full 40-char SHA somewhere in its body. The repository’s existing guidance instructs users to post the command (e.g., `@coderabbitai review`) without requiring an embedded SHA, so typical requests won’t be counted.

### Issue Context
- PR template and policy list canonical commands without a SHA requirement.
- Existing workflow gate logic treats a request as the command line alone.

### Fix Focus Areas
- scripts/verify-ai-review-contract.cjs[42-46]
- scripts/verify-ai-review-contract.cjs[106-117]
- .github/pull_request_template.md[11-20]
- docs/ai-review-cooperation-policy.md[81-90]

### Suggested fix (choose one)
A) If embedding SHA in the request is truly required: update the PR template + docs to explicitly specify the exact format the verifier expects (e.g., two lines: `@coderabbitai review` and the full SHA on its own line) so behavior and verifier match.

B) If the existing command-only format should remain valid: remove the `containsExactHead(...)` requirement from `freshRequest()` and instead keep the exact-head binding solely on the evidence side (review/status tied to `currentHead`).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

Comment thread scripts/verify-ai-review-contract.cjs
Comment thread scripts/verify-ai-review-contract.cjs Outdated
Comment thread scripts/verify-ai-review-contract.cjs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/verify-ai-review-contract.cjs`:
- Around line 85-104: The polling loop in verify-ai-review-contract.cjs is
re-fetching all comments, reviews, and statuses on every iteration, which
creates unnecessary API load. Update the Promise.all call to use incremental
fetching for github.rest.issues.listComments by passing a since value derived
from headUpdateAnchor, while keeping the existing created_at filtering in
freshRequest for correctness. Leave the reviews and commit statuses logic intact
unless there is an equivalent safe incremental filter available.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: d29a6904-84cd-4dde-b8b4-7405b70d1597

📥 Commits

Reviewing files that changed from the base of the PR and between 39d77bc and d8992bf.

📒 Files selected for processing (1)
  • scripts/verify-ai-review-contract.cjs
📜 Review details
⏰ Context from checks skipped due to timeout. (5)
  • GitHub Check: profiles / profile (mobile)
  • GitHub Check: profiles / profile (desktop)
  • GitHub Check: VISUAL-001 screenshot diff
  • GitHub Check: verify
  • GitHub Check: DAST-001 passive web scan
⚠️ CI failures not shown inline (2)

GitHub Actions: Bot review disposition contract / REVIEW-LEDGER exact-head findings: ci: bootstrap trusted AI review verifier

Conclusion: failure

View job details

##[group]Run actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b
 with:
   script: const owner = context.repo.owner;
const repo = context.repo.repo;
const payloadPr = context.payload.pull_request;
const issuePr = context.payload.issue?.pull_request;
const prNumber = payloadPr?.number ?? (issuePr ? context.payload.issue.number : null);
if (!prNumber) {
  core.notice('Event is not associated with a pull request.');
  return;
}
const pr = (await github.rest.pulls.get({ owner, repo, pull_number: prNumber })).data;
const head = pr.head.sha.toLowerCase();
const [reviewComments, reviews, issueComments] = await Promise.all([
  github.paginate(
    github.rest.pulls.listReviewComments,
    { owner, repo, pull_number: prNumber, per_page: 100 },
  ),
  github.paginate(
    github.rest.pulls.listReviews,
    { owner, repo, pull_number: prNumber, per_page: 100 },
  ),
  github.paginate(
    github.rest.issues.listComments,
    { owner, repo, issue_number: prNumber, per_page: 100 },
  ),
]);
const reviewHeads = new Map(
  reviews.map((review) => [review.id, review.commit_id?.toLowerCase()]),
);
const reviewBots = new Set([
  'chatgpt-codex-connector',
  'chatgpt-codex-connector[bot]',
  'coderabbitai',
  'coderabbitai[bot]',
  'github-advanced-security',
  'github-advanced-security[bot]',
]);
const trustedAssociations = new Set(['OWNER', 'MEMBER', 'COLLABORATOR']);
const exactHeadBody = (body) => new RegExp(`(^|[^0-9a-f])${head}([^0-9a-f]|$)`, 'i').test(body ?? '');
const timeOf = (item) => Math.max(
  0,
  ...[item.submitted_at, item.created_at, item.updated_at]
    .map((value) => Date.parse(value ?? 0))
    .filter(Number.isFinite),
);
const isDispositionReply = (reply) => {
  const body = (reply.body ?? '').replaceAll('`', '');
  const disposition = /^Disposition:\s*(accepted|rejected-with-evidence|superseded)\s*$/im.test(body);
  const exactHead = new RegExp(`^Head:\\s*${head}\\s*$`, 'im').test(body);
  const maintainer = trustedAssociations.has(reply.author_as...

GitHub Actions: Bot review disposition contract / 0_REVIEW-LEDGER exact-head findings.txt: ci: bootstrap trusted AI review verifier

Conclusion: failure

View job details

##[group]Run actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b
 with:
   script: const owner = context.repo.owner;
const repo = context.repo.repo;
const payloadPr = context.payload.pull_request;
const issuePr = context.payload.issue?.pull_request;
const prNumber = payloadPr?.number ?? (issuePr ? context.payload.issue.number : null);
if (!prNumber) {
  core.notice('Event is not associated with a pull request.');
  return;
}
const pr = (await github.rest.pulls.get({ owner, repo, pull_number: prNumber })).data;
const head = pr.head.sha.toLowerCase();
const [reviewComments, reviews, issueComments] = await Promise.all([
  github.paginate(
    github.rest.pulls.listReviewComments,
    { owner, repo, pull_number: prNumber, per_page: 100 },
  ),
  github.paginate(
    github.rest.pulls.listReviews,
    { owner, repo, pull_number: prNumber, per_page: 100 },
  ),
  github.paginate(
    github.rest.issues.listComments,
    { owner, repo, issue_number: prNumber, per_page: 100 },
  ),
]);
const reviewHeads = new Map(
  reviews.map((review) => [review.id, review.commit_id?.toLowerCase()]),
);
const reviewBots = new Set([
  'chatgpt-codex-connector',
  'chatgpt-codex-connector[bot]',
  'coderabbitai',
  'coderabbitai[bot]',
  'github-advanced-security',
  'github-advanced-security[bot]',
]);
const trustedAssociations = new Set(['OWNER', 'MEMBER', 'COLLABORATOR']);
const exactHeadBody = (body) => new RegExp(`(^|[^0-9a-f])${head}([^0-9a-f]|$)`, 'i').test(body ?? '');
const timeOf = (item) => Math.max(
  0,
  ...[item.submitted_at, item.created_at, item.updated_at]
    .map((value) => Date.parse(value ?? 0))
    .filter(Number.isFinite),
);
const isDispositionReply = (reply) => {
  const body = (reply.body ?? '').replaceAll('`', '');
  const disposition = /^Disposition:\s*(accepted|rejected-with-evidence|superseded)\s*$/im.test(body);
  const exactHead = new RegExp(`^Head:\\s*${head}\\s*$`, 'im').test(body);
  const maintainer = trustedAssociations.has(reply.author_as...
🧰 Additional context used
📓 Path-based instructions (1)
scripts/**

⚙️ CodeRabbit configuration file

scripts/**: Ищи недетерминированность, небезопасную работу с путями и shell quoting,
скрытое ослабление проверок, сетевые зависимости, утечки секретов и случаи,
когда проверка может ложно завершиться успешно.

Files:

  • scripts/verify-ai-review-contract.cjs
🔇 Additional comments (5)
scripts/verify-ai-review-contract.cjs (5)

3-11: Проверка casing review.state и commit status API подтверждена — работает как задумано.

Убедился через документацию GitHub, что review.state действительно возвращается в верхнем регистре ("PENDING", "DISMISSED"), поэтому сравнения в isSubmittedActiveReview корректны и не создают ложно-успешного прохождения pending/dismissed ревью. Также подтверждено, что CodeRabbit публикует именно классический commit status (не Checks API) через настройку reviews.commit_status, так что обращение к listCommitStatusesForRef валидно.

Also applies to: 26-32, 128-135


106-135: 🎯 Functional Correctness

Асимметрия timing-условий: review требует порядок после запроса, status — нет.

Для codeRabbitReview жёстко требуется submittedTimeOf(review) >= codeRabbitRequestAt (ревью после запроса). Для codeRabbitStatus условие слабее — createdTimeOf(status) >= headUpdateAnchor, без привязки к codeRabbitRequestAt. Это значит, что автоматический успешный status, созданный CodeRabbit сразу после пуша (до того как доверенный мейнтейнер вообще написал @coderabbitai review <sha>), всё равно засчитывается как доказательство, как только позже появится любой валидный trusted-запрос с этим SHA — даже если status логически не связан с этим запросом.

Судя по описанию PR ("commit status created in the same head-update window"), это может быть намеренным упрощением (status генерируется автоматически при пуше, а не по команде). Стоит подтвердить, что это осознанный компромисс, а не пропущенная проверка порядка.


145-149: 🩺 Stability & Availability | ⚡ Quick win

Финальный core.setFailed не содержит причину транзиентных ошибок.

В catch ошибки только логируются через core.warning и теряются к моменту итогового setFailed (строка 191-193). Если 15 минут подряд падает, например, getWorkflowRun из-за нехватки прав или pulls.listReviews из-за rate limit, итоговое сообщение об ошибке будет говорить про "trusted exact-head review", хотя реальная причина — API-ошибка. Это усложняет диагностику сбоев обязательного review-гейта.

As per path instructions for scripts/**: "Ищи ... случаи, когда проверка может ложно завершиться успешно" — здесь обратный риск: ложно неинформативный отказ, маскирующий инфраструктурную проблему.

💡 Предложение: сохранять последнюю ошибку и включать её в setFailed
   for (let attempt = 1; attempt <= POLL_ATTEMPTS; attempt += 1) {
     let codeRabbitRequestAt = 0;
     let codeRabbitReview;
     let codeRabbitStatus;
     let nativeCodexReview;
+    let lastError;

     try {
       ...
     } catch (error) {
+      lastError = error;
       core.warning(
         `Transient GitHub API error (${attempt}/${POLL_ATTEMPTS}): ${error.message}`,
       );
     }
     ...
   }

   core.setFailed(
-    "Require a trusted exact-head `@coderabbitai` review request created after the immutable head-update anchor and native CodeRabbit evidence for that same head: either a submitted active review after the request or a successful bot-authored automatic commit status created in the same head-update window.",
+    "Require a trusted exact-head `@coderabbitai` review request created after the immutable head-update anchor and native CodeRabbit evidence for that same head: either a submitted active review after the request or a successful bot-authored automatic commit status created in the same head-update window." +
+      (lastError ? ` Last API error: ${lastError.message}` : ""),
   );

Also applies to: 191-193

Source: Path instructions


34-53: Утилиты нормализации и построения exact-head паттерна — корректны.

commandLinesOf и containsExactHead дают безопасный от ReDoS и от частичного совпадения SHA механизм (граница по не-hex символам), а использование created_at вместо updated_at во всех местах (createdTimeOf, latestCreatedTime) корректно закрывает сценарий редактирования старого комментария для имитации свежего запроса.


55-61: 🩺 Stability & Availability

Проверьте trigger workflow и permissions токена. Скрипт падает без context.payload.pull_request.head.sha; если job не запускается на pull_request/pull_request_target, это будет постоянный false-negative. Для actions.getWorkflowRun, pulls.listReviews, issues.listComments и repos.listCommitStatusesForRef нужны actions: read, pull-requests: read, issues: read, statuses: read — иначе проверка будет ложно проваливаться.

Comment thread scripts/verify-ai-review-contract.cjs
@safal207 safal207 mentioned this pull request Jul 6, 2026
18 tasks

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/verify-ai-review-contract.cjs`:
- Around line 49-57: `isPermissionError()` is treating every 403 as a fatal
permission failure, which causes valid GitHub API rate-limit responses to stop
polling. Update the logic in `isPermissionError` to distinguish true
access-denied cases from primary rate-limit 403s by checking the response
headers and/or message text, and only return true for actual permission problems
while allowing rate-limit 403s to be retried.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 31dc7af9-7929-4495-ae0b-40877fe03016

📥 Commits

Reviewing files that changed from the base of the PR and between d8992bf and 2b5a661.

📒 Files selected for processing (1)
  • scripts/verify-ai-review-contract.cjs
📜 Review details
⏰ Context from checks skipped due to timeout. (5)
  • GitHub Check: profiles / profile (desktop)
  • GitHub Check: VISUAL-001 screenshot diff
  • GitHub Check: verify
  • GitHub Check: ios-route-webkit
  • GitHub Check: DAST-001 passive web scan
⚠️ CI failures not shown inline (2)

GitHub Actions: Bot review disposition contract / 0_REVIEW-LEDGER exact-head findings.txt: ci: bootstrap trusted AI review verifier

Conclusion: failure

View job details

##[group]Run actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b
 with:
   script: const owner = context.repo.owner;
const repo = context.repo.repo;
const payloadPr = context.payload.pull_request;
const issuePr = context.payload.issue?.pull_request;
const prNumber = payloadPr?.number ?? (issuePr ? context.payload.issue.number : null);
if (!prNumber) {
  core.notice('Event is not associated with a pull request.');
  return;
}
const pr = (await github.rest.pulls.get({ owner, repo, pull_number: prNumber })).data;
const head = pr.head.sha.toLowerCase();
const [reviewComments, reviews, issueComments] = await Promise.all([
  github.paginate(
    github.rest.pulls.listReviewComments,
    { owner, repo, pull_number: prNumber, per_page: 100 },
  ),
  github.paginate(
    github.rest.pulls.listReviews,
    { owner, repo, pull_number: prNumber, per_page: 100 },
  ),
  github.paginate(
    github.rest.issues.listComments,
    { owner, repo, issue_number: prNumber, per_page: 100 },
  ),
]);
const reviewHeads = new Map(
  reviews.map((review) => [review.id, review.commit_id?.toLowerCase()]),
);
const reviewBots = new Set([
  'chatgpt-codex-connector',
  'chatgpt-codex-connector[bot]',
  'coderabbitai',
  'coderabbitai[bot]',
  'github-advanced-security',
  'github-advanced-security[bot]',
]);
const trustedAssociations = new Set(['OWNER', 'MEMBER', 'COLLABORATOR']);
const exactHeadBody = (body) => new RegExp(`(^|[^0-9a-f])${head}([^0-9a-f]|$)`, 'i').test(body ?? '');
const timeOf = (item) => Math.max(
  0,
  ...[item.submitted_at, item.created_at, item.updated_at]
    .map((value) => Date.parse(value ?? 0))
    .filter(Number.isFinite),
);
const isDispositionReply = (reply) => {
  const body = (reply.body ?? '').replaceAll('`', '');
  const disposition = /^Disposition:\s*(accepted|rejected-with-evidence|superseded)\s*$/im.test(body);
  const exactHead = new RegExp(`^Head:\\s*${head}\\s*$`, 'im').test(body);
  const maintainer = trustedAssociations.has(reply.author_as...

GitHub Actions: Bot review disposition contract / REVIEW-LEDGER exact-head findings: ci: bootstrap trusted AI review verifier

Conclusion: failure

View job details

##[group]Run actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b
 with:
   script: const owner = context.repo.owner;
const repo = context.repo.repo;
const payloadPr = context.payload.pull_request;
const issuePr = context.payload.issue?.pull_request;
const prNumber = payloadPr?.number ?? (issuePr ? context.payload.issue.number : null);
if (!prNumber) {
  core.notice('Event is not associated with a pull request.');
  return;
}
const pr = (await github.rest.pulls.get({ owner, repo, pull_number: prNumber })).data;
const head = pr.head.sha.toLowerCase();
const [reviewComments, reviews, issueComments] = await Promise.all([
  github.paginate(
    github.rest.pulls.listReviewComments,
    { owner, repo, pull_number: prNumber, per_page: 100 },
  ),
  github.paginate(
    github.rest.pulls.listReviews,
    { owner, repo, pull_number: prNumber, per_page: 100 },
  ),
  github.paginate(
    github.rest.issues.listComments,
    { owner, repo, issue_number: prNumber, per_page: 100 },
  ),
]);
const reviewHeads = new Map(
  reviews.map((review) => [review.id, review.commit_id?.toLowerCase()]),
);
const reviewBots = new Set([
  'chatgpt-codex-connector',
  'chatgpt-codex-connector[bot]',
  'coderabbitai',
  'coderabbitai[bot]',
  'github-advanced-security',
  'github-advanced-security[bot]',
]);
const trustedAssociations = new Set(['OWNER', 'MEMBER', 'COLLABORATOR']);
const exactHeadBody = (body) => new RegExp(`(^|[^0-9a-f])${head}([^0-9a-f]|$)`, 'i').test(body ?? '');
const timeOf = (item) => Math.max(
  0,
  ...[item.submitted_at, item.created_at, item.updated_at]
    .map((value) => Date.parse(value ?? 0))
    .filter(Number.isFinite),
);
const isDispositionReply = (reply) => {
  const body = (reply.body ?? '').replaceAll('`', '');
  const disposition = /^Disposition:\s*(accepted|rejected-with-evidence|superseded)\s*$/im.test(body);
  const exactHead = new RegExp(`^Head:\\s*${head}\\s*$`, 'im').test(body);
  const maintainer = trustedAssociations.has(reply.author_as...
🧰 Additional context used
📓 Path-based instructions (1)
scripts/**

⚙️ CodeRabbit configuration file

scripts/**: Ищи недетерминированность, небезопасную работу с путями и shell quoting,
скрытое ослабление проверок, сетевые зависимости, утечки секретов и случаи,
когда проверка может ложно завершиться успешно.

Files:

  • scripts/verify-ai-review-contract.cjs
🔇 Additional comments (1)
scripts/verify-ai-review-contract.cjs (1)

95-95: LGTM!

Also applies to: 112-149

Comment thread scripts/verify-ai-review-contract.cjs

safal207 commented Jul 6, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

safal207 commented Jul 6, 2026

Copy link
Copy Markdown
Owner Author

Disposition: accepted
Head: 2b5a661
Findings: PRRT_kwDOS-8ZNM6OjDmn, PRRT_kwDOS-8ZNM6OjDmq, PRRT_kwDOS-8ZNM6OjDmr, PRRT_kwDOS-8ZNM6OjEc1

Corrections: permission failures now fail immediately with required scopes; comments are filtered with since while immutable created_at remains authoritative; fresh requests follow the existing command-only repository format and are bounded by the immutable head-update anchor; native status evidence must be created after the trusted request; the final failure preserves the latest transient API error.

safal207 commented Jul 6, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/verify-ai-review-contract.cjs`:
- Around line 53-56: Treat all non-rate-limit 403 responses as fatal permission
failures in the 403 classifier. Update the logic in the 403 handling path around
the existing message match so that only rate limit, secondary rate, and abuse
detection messages are treated as transient, while any other 403 seen by the
helper is considered a hard auth/permission error and causes immediate failure
instead of retrying.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 9a356f3c-3e9e-45f6-85b1-ded44a82f4c3

📥 Commits

Reviewing files that changed from the base of the PR and between 2b5a661 and 48a838f.

📒 Files selected for processing (1)
  • scripts/verify-ai-review-contract.cjs
📜 Review details
⏰ Context from checks skipped due to timeout. (5)
  • GitHub Check: profiles / profile (desktop)
  • GitHub Check: profiles / profile (mobile)
  • GitHub Check: verify
  • GitHub Check: VISUAL-001 screenshot diff
  • GitHub Check: DAST-001 passive web scan
⚠️ CI failures not shown inline (2)

GitHub Actions: Bot review disposition contract / REVIEW-LEDGER exact-head findings: ci: bootstrap trusted AI review verifier

Conclusion: failure

View job details

##[group]Run actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b
 with:
   script: const owner = context.repo.owner;
const repo = context.repo.repo;
const payloadPr = context.payload.pull_request;
const issuePr = context.payload.issue?.pull_request;
const prNumber = payloadPr?.number ?? (issuePr ? context.payload.issue.number : null);
if (!prNumber) {
  core.notice('Event is not associated with a pull request.');
  return;
}
const pr = (await github.rest.pulls.get({ owner, repo, pull_number: prNumber })).data;
const head = pr.head.sha.toLowerCase();
const [reviewComments, reviews, issueComments] = await Promise.all([
  github.paginate(
    github.rest.pulls.listReviewComments,
    { owner, repo, pull_number: prNumber, per_page: 100 },
  ),
  github.paginate(
    github.rest.pulls.listReviews,
    { owner, repo, pull_number: prNumber, per_page: 100 },
  ),
  github.paginate(
    github.rest.issues.listComments,
    { owner, repo, issue_number: prNumber, per_page: 100 },
  ),
]);
const reviewHeads = new Map(
  reviews.map((review) => [review.id, review.commit_id?.toLowerCase()]),
);
const reviewBots = new Set([
  'chatgpt-codex-connector',
  'chatgpt-codex-connector[bot]',
  'coderabbitai',
  'coderabbitai[bot]',
  'github-advanced-security',
  'github-advanced-security[bot]',
]);
const trustedAssociations = new Set(['OWNER', 'MEMBER', 'COLLABORATOR']);
const exactHeadBody = (body) => new RegExp(`(^|[^0-9a-f])${head}([^0-9a-f]|$)`, 'i').test(body ?? '');
const timeOf = (item) => Math.max(
  0,
  ...[item.submitted_at, item.created_at, item.updated_at]
    .map((value) => Date.parse(value ?? 0))
    .filter(Number.isFinite),
);
const isDispositionReply = (reply) => {
  const body = (reply.body ?? '').replaceAll('`', '');
  const disposition = /^Disposition:\s*(accepted|rejected-with-evidence|superseded)\s*$/im.test(body);
  const exactHead = new RegExp(`^Head:\\s*${head}\\s*$`, 'im').test(body);
  const maintainer = trustedAssociations.has(reply.author_as...

GitHub Actions: Bot review disposition contract / 0_REVIEW-LEDGER exact-head findings.txt: ci: bootstrap trusted AI review verifier

Conclusion: failure

View job details

##[group]Run actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b
 with:
   script: const owner = context.repo.owner;
const repo = context.repo.repo;
const payloadPr = context.payload.pull_request;
const issuePr = context.payload.issue?.pull_request;
const prNumber = payloadPr?.number ?? (issuePr ? context.payload.issue.number : null);
if (!prNumber) {
  core.notice('Event is not associated with a pull request.');
  return;
}
const pr = (await github.rest.pulls.get({ owner, repo, pull_number: prNumber })).data;
const head = pr.head.sha.toLowerCase();
const [reviewComments, reviews, issueComments] = await Promise.all([
  github.paginate(
    github.rest.pulls.listReviewComments,
    { owner, repo, pull_number: prNumber, per_page: 100 },
  ),
  github.paginate(
    github.rest.pulls.listReviews,
    { owner, repo, pull_number: prNumber, per_page: 100 },
  ),
  github.paginate(
    github.rest.issues.listComments,
    { owner, repo, issue_number: prNumber, per_page: 100 },
  ),
]);
const reviewHeads = new Map(
  reviews.map((review) => [review.id, review.commit_id?.toLowerCase()]),
);
const reviewBots = new Set([
  'chatgpt-codex-connector',
  'chatgpt-codex-connector[bot]',
  'coderabbitai',
  'coderabbitai[bot]',
  'github-advanced-security',
  'github-advanced-security[bot]',
]);
const trustedAssociations = new Set(['OWNER', 'MEMBER', 'COLLABORATOR']);
const exactHeadBody = (body) => new RegExp(`(^|[^0-9a-f])${head}([^0-9a-f]|$)`, 'i').test(body ?? '');
const timeOf = (item) => Math.max(
  0,
  ...[item.submitted_at, item.created_at, item.updated_at]
    .map((value) => Date.parse(value ?? 0))
    .filter(Number.isFinite),
);
const isDispositionReply = (reply) => {
  const body = (reply.body ?? '').replaceAll('`', '');
  const disposition = /^Disposition:\s*(accepted|rejected-with-evidence|superseded)\s*$/im.test(body);
  const exactHead = new RegExp(`^Head:\\s*${head}\\s*$`, 'im').test(body);
  const maintainer = trustedAssociations.has(reply.author_as...
🧰 Additional context used
📓 Path-based instructions (1)
scripts/**

⚙️ CodeRabbit configuration file

scripts/**: Ищи недетерминированность, небезопасную работу с путями и shell quoting,
скрытое ослабление проверок, сетевые зависимости, утечки секретов и случаи,
когда проверка может ложно завершиться успешно.

Files:

  • scripts/verify-ai-review-contract.cjs

Comment thread scripts/verify-ai-review-contract.cjs Outdated

safal207 commented Jul 6, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

safal207 commented Jul 6, 2026

Copy link
Copy Markdown
Owner Author

Bootstrap-Exception: BOOTSTRAP_NOT_ON_DEFAULT_BRANCH
Head: f9d99f6

The current default-branch AI workflow cannot securely validate a verifier that does not yet exist on the default branch. Repository policy explicitly requires merging the bootstrap first and testing it on the next PR. Scope is one verifier file; exact-head CodeRabbit status is successful; every review thread is resolved; security, CodeQL, ZAP, visual, WebKit, Lighthouse, runtime, traceability, reviewdog, and route-preflight checks are green. PR #173 remains blocked until it executes this verifier from its immutable base SHA.

safal207 commented Jul 6, 2026

Copy link
Copy Markdown
Owner Author

/merge-ready f9d99f6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant