ci: pass the Vercel automation bypass to the assistant LLM smoke - #1359
Merged
Conversation
|
E2E results (preview) Smoke
|
tyhonchik
force-pushed
the
fix/assistant-llm-smoke-vercel-bypass
branch
from
September 1, 2026 13:13
74b9817 to
3c95a31
Compare
tyhonchik
force-pushed
the
fix/assistant-llm-smoke-vercel-bypass
branch
from
September 1, 2026 13:14
3c95a31 to
52949fd
Compare
Every LLM smoke run has been red since mid-July for two stacked reasons. Access: the assistant domains sit behind the project firewall — Bot Protection challenges non-browser sources and the assistant-origin-guard rule denies POSTs without an app Origin — so the script's plain fetches died with 403/429 before reaching the service. The smoke now sends x-vercel-protection-bypass, matched by the assistant-smoke-bypass firewall rule (first in the chain, action Bypass); the workflow loads the secret from 1Password as VERCEL_AUTOMATION_BYPASS_SECRET with continue-on-error, so a missing vault item degrades to today's failure instead of a new one. Scenarios: the 403 masked that the script still tested the retired deterministic pipeline (POST /issues/preview, POST /issues) — those routes are gone since the v2 assistant-ui migration, and their 404 text surfaced as a JSON parse error. The scenarios now exercise the real v2 flow: the agent drafts a createLinearTicket tool call, the stream pauses on a tool-approval-request, and an approval resume (the history re-sent ending on the assistant message whose tool part carries approval) executes the creation. Off-topic asserts no draft is offered; the bug report approves the draft and checks the created ticket's identifier/url; the feedback scenario checks the draft intent without approving, so it creates nothing. A one-turn clarifying question from the model is tolerated before failing. Verified against dev: health and draft/approval chunks live; the resume wire format mirrors the chat route's unit-test fixture. The full creation path needs a fresh IP (the dev per-IP new-session budget was exhausted while testing) — first CI run will confirm.
tyhonchik
force-pushed
the
fix/assistant-llm-smoke-vercel-bypass
branch
from
September 1, 2026 14:20
52949fd to
0813be2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every Assistant LLM Smoke run has been red since mid-July — two stacked breakages, the first masking the second.
Access. Both assistant domains sit behind the project firewall: Bot Protection (Challenge) stops non-browser sources and the
assistant-origin-guardrule denies POSTs without an app Origin, so the script's plainfetchcalls died with 403/429 before reaching the service — even/health. Real browsers pass, so the production widget was never affected; only automation was locked out.Scenarios. Once past the firewall, the script still tested the retired deterministic pipeline (
POST /issues/preview,POST /issues) — those routes are gone since the v2 assistant-ui migration, and their404 Not Foundtext surfaced as a cryptic JSON parse error.Changes
llmSmoke.mjssends the Vercel bypass — every request carriesx-vercel-protection-bypasswhenVERCEL_AUTOMATION_BYPASS_SECRETis set. It is matched by the newassistant-smoke-bypassfirewall rule (already created, first in the chain, action Bypass — also exempts the origin guard and the edge rate limits for smoke traffic). Without the secret the header is omitted and behavior is unchanged.op://kv_assistant_infra/VERCEL_AUTOMATION_BYPASS_SECRET/credential) withcontinue-on-error, so a missing vault item degrades to today's 403s instead of failing the job with a new error.createLinearTickettool call, the stream pauses ontool-approval-request, and an approval resume (history re-sent ending on the assistant message whose tool part carriesapproval: { approved: true }, mirroring the chat route's unit-test fixture) executes the creation. Off-topic asserts no draft is offered; the bug report approves the draft and checks the created ticket'sidentifier/url; feedback checks the draftintentwithout approving, so it creates nothing. One clarifying question from the model is tolerated before failing.Verified against dev with the real secret: health passes, draft + approval-request chunks confirmed live. The full creation path could not be finished from my IP — probing exhausted the dev per-IP new-session budget (10/day) — the first CI run (fresh runner IP) will confirm it.
Remaining manual step
Store the firewall rule's secret in 1Password: vault
kv_assistant_infra, itemVERCEL_AUTOMATION_BYPASS_SECRET, fieldcredential(Andrii has the value). Then merge andworkflow_dispatchthe smoke.🤖 Generated with Claude Code