chore: retry the whole test with fixtures in case of error (#684) #601
accuracy-tests.yml
on: push
Run Accuracy Tests
10m 30s
Annotations
3 errors and 3 warnings
|
Run Accuracy Tests
Process completed with exit code 1.
|
|
tests/accuracy/export.test.ts > 'Azure - gpt-4o' > 'Export all the movies in \'mflix.movi…':
tests/accuracy/sdk/agent.ts#L68
AI_RetryError: Failed after 3 attempts. Last error: Your requests to gpt-4o for gpt-4o in South Central US have exceeded the token rate limit for your current OpenAI S0 pricing tier. This request was for ChatCompletions_Create under Azure OpenAI API version 2024-12-01-preview. Please retry after 1 second. To increase your default rate limit, visit: https://aka.ms/oai/quotaincrease.
❯ _retryWithExponentialBackoff node_modules/ai/src/util/retry-with-exponential-backoff.ts:111:13
❯ fn node_modules/ai/src/generate-text/generate-text.ts:345:34
❯ node_modules/ai/src/telemetry/record-span.ts:18:22
❯ generateText node_modules/ai/src/generate-text/generate-text.ts:281:12
❯ Object.prompt tests/accuracy/sdk/agent.ts:68:44
❯ tests/accuracy/sdk/describeAccuracyTests.ts:124:28
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { reason: 'maxRetriesExceeded', errors: [ { stack: 'AI_APICallError: Your requests to gpt-4o for gpt-4o in South Central US have exceeded the token rate limit for your current OpenAI S0 pricing tier. This request was for ChatCompletions_Create under Azure OpenAI API version 2024-12-01-preview. Please retry after 1 second. To increase your default rate limit, visit: https://aka.ms/oai/quotaincrease.\n at file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/@ai-sdk/provider-utils/src/response-handler.ts:57:16\n at processTicksAndRejections (node:internal/process/task_queues:105:5)\n at postToApi (file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/@ai-sdk/provider-utils/src/post-to-api.ts:118:28)\n at OpenAIChatLanguageModel.doGenerate (file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/@ai-sdk/openai/src/chat/openai-chat-language-model.ts:321:9)\n at fn (file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/ai/src/generate-text/generate-text.ts:389:32)\n at file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/ai/src/telemetry/record-span.ts:18:22\n at _retryWithExponentialBackoff (file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/ai/src/util/retry-with-exponential-backoff.ts:96:12)\n at fn (file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/ai/src/generate-text/generate-text.ts:345:34)\n at file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/ai/src/telemetry/record-span.ts:18:22\n at generateText (file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/ai/src/generate-text/generate-text.ts:281:12)', message: 'Your requests to gpt-4o for gpt-4o in South Central US have exceeded the token rate limit for your current OpenAI S0 pricing tier. This request was for ChatCompletions_Create under Azure OpenAI API version 2024-12-01-preview. Please retry after 1 second. To increase your default rate limit, visit: https://aka.ms/oai/quotaincrease.', name: 'AI_APICallError', cause: undefined, url: 'https://gai-405.openai.azure.com/openai//deployments/gpt-4o/chat/completions?api-version=2024-12-01-preview', requestBodyValues: { model: 'gpt-4o', logit_bias: undefined, logprobs: undefined, top_logprobs: undefined, user: undefined, parallel_tool_calls: undefined, max_tokens: undefined, temperature: undefined, top_p: undefined, frequency_penalty: undefined, presence_penalty: undefined, response_format: undefined, stop: undefined, seed: undefined, verbosity: undefined, max_completion_tokens: undefined, store: undefined, metadata: undefined, prediction: undefined, reasoning_effort: undefined, service_tier: undefined, prompt_cache_key: undefined, safety_identifier: undefined, messages: [ { role: 'system', content: 'The keywords "MUST", "MUST NOT", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119\nYou are an expert AI assistant with access to a set of tools for MongoDB database operations.\nYou MUST use the most relevant tool to answer the user\'s request\nWhen calling a tool, you MUST strictly f
|
|
tests/accuracy/dropIndex.test.ts > 'Azure - gpt-4o' > 'First create a text index on field \'…':
tests/accuracy/sdk/agent.ts#L68
AI_RetryError: Failed after 3 attempts. Last error: Your requests to gpt-4o for gpt-4o in South Central US have exceeded the token rate limit for your current OpenAI S0 pricing tier. This request was for ChatCompletions_Create under Azure OpenAI API version 2024-12-01-preview. Please retry after 2 seconds. To increase your default rate limit, visit: https://aka.ms/oai/quotaincrease.
❯ _retryWithExponentialBackoff node_modules/ai/src/util/retry-with-exponential-backoff.ts:111:13
❯ fn node_modules/ai/src/generate-text/generate-text.ts:345:34
❯ node_modules/ai/src/telemetry/record-span.ts:18:22
❯ generateText node_modules/ai/src/generate-text/generate-text.ts:281:12
❯ Object.prompt tests/accuracy/sdk/agent.ts:68:44
❯ tests/accuracy/sdk/describeAccuracyTests.ts:124:28
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { reason: 'maxRetriesExceeded', errors: [ { stack: 'AI_APICallError: Your requests to gpt-4o for gpt-4o in South Central US have exceeded the token rate limit for your current OpenAI S0 pricing tier. This request was for ChatCompletions_Create under Azure OpenAI API version 2024-12-01-preview. Please retry after 1 second. To increase your default rate limit, visit: https://aka.ms/oai/quotaincrease.\n at file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/@ai-sdk/provider-utils/src/response-handler.ts:57:16\n at processTicksAndRejections (node:internal/process/task_queues:105:5)\n at postToApi (file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/@ai-sdk/provider-utils/src/post-to-api.ts:118:28)\n at OpenAIChatLanguageModel.doGenerate (file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/@ai-sdk/openai/src/chat/openai-chat-language-model.ts:321:9)\n at fn (file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/ai/src/generate-text/generate-text.ts:389:32)\n at file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/ai/src/telemetry/record-span.ts:18:22\n at _retryWithExponentialBackoff (file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/ai/src/util/retry-with-exponential-backoff.ts:96:12)\n at fn (file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/ai/src/generate-text/generate-text.ts:345:34)\n at file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/ai/src/telemetry/record-span.ts:18:22\n at generateText (file:///home/runner/work/mongodb-mcp-server/mongodb-mcp-server/node_modules/ai/src/generate-text/generate-text.ts:281:12)', message: 'Your requests to gpt-4o for gpt-4o in South Central US have exceeded the token rate limit for your current OpenAI S0 pricing tier. This request was for ChatCompletions_Create under Azure OpenAI API version 2024-12-01-preview. Please retry after 1 second. To increase your default rate limit, visit: https://aka.ms/oai/quotaincrease.', name: 'AI_APICallError', cause: undefined, url: 'https://gai-405.openai.azure.com/openai//deployments/gpt-4o/chat/completions?api-version=2024-12-01-preview', requestBodyValues: { model: 'gpt-4o', logit_bias: undefined, logprobs: undefined, top_logprobs: undefined, user: undefined, parallel_tool_calls: undefined, max_tokens: undefined, temperature: undefined, top_p: undefined, frequency_penalty: undefined, presence_penalty: undefined, response_format: undefined, stop: undefined, seed: undefined, verbosity: undefined, max_completion_tokens: undefined, store: undefined, metadata: undefined, prediction: undefined, reasoning_effort: undefined, service_tier: undefined, prompt_cache_key: undefined, safety_identifier: undefined, messages: [ { role: 'system', content: 'The keywords "MUST", "MUST NOT", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119\nYou are an expert AI assistant with access to a set of tools for MongoDB database operations.\nYou MUST use the most relevant tool to answer the user\'s request\nWhen calling a tool, you MUST strictly
|
|
Run Accuracy Tests
1 added, 0 removed; done.
|
|
Run Accuracy Tests
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
|
|
Run Accuracy Tests
Updating certificates in /etc/ssl/certs...
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
accuracy-test-summary
|
46.4 KB |
sha256:aef7edbd7e98cc2608ddada972dbaf8b90ea1624c1739e86965bb0fde7b7e486
|
|
|
run-accuracy-tests-permissions-22e58b36b9291daee5ec76021a983f89
|
140 Bytes |
sha256:bb4b5bc93d9c90223a4297a7e4d57096ebe499ccb784abfe40d47b08f850df15
|
|