Large influx of 500 internal server error

[cblach]

We are experiencing intermittent 500 internal server errors with actions--access-token. The problem tends to come in bulk (as in when it happens it typically happens to multiple GHA jobs), and it happens often enough that it's somewhat of a problem.
Could it be related to a large number of requests hitting the server at once?

[qoomon]

I'll investigate tomorrow

[qoomon]

Does it still happen?

[nils-borrmann-tacto]

We are seeing similar problems, where the Github Api is seemingly just flaky. Sometimes it's a 500, sometimes other errors. I would appreciate if the gh-action would have an option to retry with exponential backoff. (This is not a persistent problem, but it happens a few times per day, so much so that it's the biggest source of flakyness in our pipeline right now)

  result: {
    requestId: '1c8ed77b-d323-4857-b5a2-1faa6fe8761f',
    status: 403,
    error: 'Forbidden',
    message: 'Issues:\n' +
      '- tacto-ai:\n' +
      '  - Access policy not found\n' +
      'Effective OIDC token subjects:\n' +
      '- repo:tacto-ai/tacto:ref:refs/heads/gh-readonly-queue/main/pr-22068-701ec745037f4cd406e842d12be7b1bfd140cfeb\n' +
      '- repo:tacto-ai/tacto:workflow_ref:tacto-ai/tacto/.github/workflows/backend.yaml@refs/heads/gh-readonly-queue/main/pr-22068-701ec745037f4cd406e842d12be7b1bfd140cfeb\n' +
      '- repo:tacto-ai/tacto:job_workflow_ref:tacto-ai/tacto/.github/workflows/setup-deployment-vars.yaml@refs/heads/gh-readonly-queue/main/pr-22068-701ec745037f4cd406e842d12be7b1bfd140cfeb'
  }

Get access token...
Error: HttpClientError: error code: 1101
    at file:///home/runner/_work/_actions/qoomon/actions--access-token/v3/action/dist/main/index.js:48930:31
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async httpRequest (file:///home/runner/_work/_actions/qoomon/actions--access-token/v3/action/dist/main/index.js:48918:12)
    at async getAccessToken (file:///home/runner/_work/_actions/qoomon/actions--access-token/v3/action/dist/main/index.js:48894:12)
    at async file:///home/runner/_work/_actions/qoomon/actions--access-token/v3/action/dist/main/index.js:48848:25 {
  statusCode: 500,
  result: 'error code: 1101'
}

[cblach]

Hi @qoomon
Yes it's still happening. And we also intermittently observe similar Access policy not found errors like @nils-borrmann-tacto are reporting

[qoomon]

Thanks, probably the server should handle those errors and do some retries.

actions--access-token/server/src/access-token-manager.ts

Lines 597 to 626 in fceb5e4

	async function getAccessPolicy<T extends typeof GitHubAccessPolicySchema>(client: Octokit, {
	owner, repo, paths, schema, preprocessor,
	}: {
	owner: string,
	repo: string,
	paths: string[],
	schema: T,
	preprocessor: (value: unknown) => unknown,
	}): Promise<z.infer<T>> {
	const policyValue = await findFirstNotNull(paths, (path) =>
	getRepositoryFileContent(client, {owner, repo, path, maxSize: ACCESS_POLICY_MAX_SIZE}));
	if (!policyValue) {
	throw new GithubAccessPolicyError(`Access policy not found`);
	}
	const policyParseResult = YamlTransformer
	.transform(preprocessor)
	.pipe(schema)
	.safeParse(policyValue);
	if (policyParseResult.error) {
	const issues = policyParseResult.error.issues.map(formatZodIssue);
	throw new GithubAccessPolicyError(`Invalid access policy`, issues);
	}
	const policy = policyParseResult.data;
	const expectedPolicyOrigin = `${owner}/${repo}`;
	if (policy.origin.toLowerCase() !== expectedPolicyOrigin.toLowerCase()) {
	const issues = [`Policy origin '${policy.origin}' does not match repository '${expectedPolicyOrigin}'`];
	throw new GithubAccessPolicyError(`Invalid access policy`, issues);
	}

[qoomon]

I hopefully fixed a bug regarding the returning requestId. could you provide me a request id of a failing request again any request since now

[qoomon]

another reason could be that cloudflare is kill the function after hitting max cpu usage, because its hosted at the free plan. If you have any other free hosters in mind please let me know. I'm open for sponsoring to run the server side on a paid backend.

[nils-borrmann-tacto]

Here's another failure from about an hour ago:

  statusCode: 403,
  result: {
    requestId: '9d5f81d1af653778',
    status: 403,
    error: 'Forbidden',
    message: 'Issues:\n' +
      '- tacto-ai:\n' +
      '  - Access policy not found\n' +
      'Effective OIDC token subjects:\n' +
      '- repo:tacto-ai/tacto:ref:refs/heads/gh-readonly-queue/main/pr-22223-5856483234b728806a757e8ff2ec7890caf450e8\n' +
      '- repo:tacto-ai/tacto:workflow_ref:tacto-ai/tacto/.github/workflows/dagster.yaml@refs/heads/gh-readonly-queue/main/pr-22223-5856483234b728806a757e8ff2ec7890caf450e8\n' +
      '- repo:tacto-ai/tacto:job_workflow_ref:tacto-ai/tacto/.github/workflows/setup-deployment-vars.yaml@refs/heads/gh-readonly-queue/main/pr-22223-5856483234b728806a757e8ff2ec7890caf450e8'
  }
}

[cblach]

If it helps, I also have one here from a few minutes ago at the point of writing:

Run qoomon/actions--access-token@f2165838fcde60d33f3bc4bb1e586aa6c8b665d3
Get access token...
Error: HttpClientError: Issues:
- cellpointdigital:
  - Access policy not found
Effective OIDC token subjects:
- repo:cellpointdigital/phalanx-applications:pull_request
    at file:///home/runner/work/_actions/qoomon/actions--access-token/f2165838fcde60d33f3bc4bb1e586aa6c8b665d3/action/dist/main/index.js:51123:31
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async httpRequest (file:///home/runner/work/_actions/qoomon/actions--access-token/f2165838fcde60d33f3bc4bb1e586aa6c8b665d3/action/dist/main/index.js:51111:12)
    at async getAccessToken (file:///home/runner/work/_actions/qoomon/actions--access-token/f2165838fcde60d33f3bc4bb1e586aa6c8b665d3/action/dist/main/index.js:51087:12)
    at async file:///home/runner/work/_actions/qoomon/actions--access-token/f2165838fcde60d33f3bc4bb1e586aa6c8b665d3/action/dist/main/index.js:51041:25 {
  statusCode: 403,
  result: {
    requestId: '9d6173df1ac47e56',
    status: 403,
    error: 'Forbidden',
    message: 'Issues:\n' +
      '- cellpointdigital:\n' +
      '  - Access policy not found\n' +
      'Effective OIDC token subjects:\n' +
      '- repo:cellpointdigital/phalanx-applications:pull_request'
Error: Issues:
- cellpointdigital:
  - Access policy not found
Effective OIDC token subjects:
- repo:cellpointdigital/phalanx-applications:pull_request
  }
}

[qoomon]

@cblach btw could you please reduce your amount of requests. currently you are requesting 22k tokens per week.
Or would you like to be a sponsor of this project/service?

[qoomon]

I added some more debug logs. Could you provide another failing request id?

[cblach]

Hi @qoomon. I am sorry about that, I'll push the team to significantly reduce the tokens requested.

[qoomon]

@cblach BTW if you or the team have any feature request let me know