feat:OpenAPI eval update/status oneOf results score fields nullable/error #148
Conversation
WalkthroughThe OpenAPI spec for evaluation endpoints was updated to use explicit oneOf-typed result schemas (Classify, Score, Compare) in update and status operations. EvaluationScoreResults replaced aggregated_scores with mean_score, pass_percentage, and std_score. EvaluationJob and status responses now allow nullable results and include an inline error object option. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant C as Client
participant API as Evaluation API
participant Store as Job Store
rect rgb(240,248,255)
note over C,API: Update evaluation job results
C->>API: PATCH /evaluation/{id}/update<br/>body.results: oneOf(Classify|Score|Compare)
API->>Store: Persist status + results
Store-->>API: OK
API-->>C: 200 Updated
end
rect rgb(245,255,250)
note over C,API: Get evaluation status and results
C->>API: GET /evaluation/{id}/status
API->>Store: Fetch job
Store-->>API: Job { status, results|null }
API-->>C: 200 { results: oneOf(Classify|Score|Compare|error) nullable }
end
note right of API: Score results fields:<br/>mean_score, pass_percentage, std_score
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
coderabbitai
bot
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
src/libs/Together/openapi.yaml (3)
1035-1056: Make update payload schema state-safe (require results on completed; require error on error states; forbid extras).Today only
statusis required andresults/errorare optional and can co-exist. Tighten validation to prevent invalid state transitions and bad writes to storage.Apply this diff within the requestBody.schema:
- required: - - status - type: object - properties: - error: - type: string - description: Error message - results: - oneOf: - - $ref: '#/components/schemas/EvaluationClassifyResults' - - $ref: '#/components/schemas/EvaluationScoreResults' - - $ref: '#/components/schemas/EvaluationCompareResults' - status: - enum: - - completed - - error - - running - - queued - - user_error - - inference_error - type: string - description: The new status for the job + oneOf: + # Completed requires results, and must not include error + - allOf: + - type: object + properties: + status: + enum: [completed] + type: string + results: + oneOf: + - $ref: '#/components/schemas/EvaluationClassifyResults' + - $ref: '#/components/schemas/EvaluationScoreResults' + - $ref: '#/components/schemas/EvaluationCompareResults' + required: [status, results] + - not: + required: [error] + # Error-like states require error message, and must not include results + - allOf: + - type: object + properties: + status: + enum: [error, user_error, inference_error] + type: string + error: + type: string + description: Error message + required: [status, error] + - not: + required: [results] + # Running/queued carry status only + - type: object + properties: + status: + enum: [running, queued] + type: string + required: [status]
3776-3781: Fix typo in description ("Percentage").“Pecentage” → “Percentage”.
pass_percentage: type: number - description: Pecentage of pass labels. + description: Percentage of pass labels. format: integer nullable: true example: 10
171-173: cURL sample points to the wrong endpoint.Under /audio/translations, the cURL uses /audio/transcriptions.
- source: "curl -X POST \"https://api.together.xyz/v1/audio/transcriptions\" \ + source: "curl -X POST \"https://api.together.xyz/v1/audio/translations\" \ -H \"Authorization: Bearer $TOGETHER_API_KEY\" \ -F \"[email protected]\" \ -F \"model=openai/whisper-large-v3\" \ -F \"language=es\" "
🧹 Nitpick comments (4)
src/libs/Together/openapi.yaml (4)
983-1005: Makestatusrequired in GET /evaluation/{id}/status.The 200 response object lacks
required: ["status"], allowing empty objects to validate.schema: type: object + required: [status] properties: results: oneOf: - $ref: '#/components/schemas/EvaluationClassifyResults' - $ref: '#/components/schemas/EvaluationScoreResults' - $ref: '#/components/schemas/EvaluationCompareResults' - type: object properties: error: type: string nullable: true status:
990-994: Requireerrorin inline error objects.Both the status response and EvaluationJob allow an empty object as an “error”. Make the
errorfield required.- - type: object - properties: - error: - type: string + - type: object + required: [error] + properties: + error: + type: string- - type: object - properties: - error: - type: string + - type: object + required: [error] + properties: + error: + type: stringAlso applies to: 3850-3853
4003-4014: Confirm intended shape of EvaluationScoreResults; consider flattening and deprecating nested aggregated_scores.Code keeps
mean_score,pass_percentage,std_scorenested underaggregated_scores. If SDKs/docs expect top-level fields, add them at top-level and deprecate the nested object to maintain backward compatibility.EvaluationScoreResults: type: object properties: - aggregated_scores: - type: object - properties: - mean_score: - type: number - format: float - pass_percentage: - type: number - format: float - std_score: - type: number - format: float + mean_score: + type: number + format: float + pass_percentage: + type: number + format: float + std_score: + type: number + format: float + aggregated_scores: + deprecated: true + type: object + properties: + mean_score: + type: number + format: float + pass_percentage: + type: number + format: float + std_score: + type: number + format: float
63-69: SDK sample path likely wrong for speech (TS/JS).Python sample uses
client.audio.speech.create, but TS/JS useclient.audio.create. Adjust for parity unless the SDK purposely differs.-const response = await client.audio.create({ +const response = await client.audio.speech.create({ model: "cartesia/sonic-2", input: "The quick brown fox jumps over the lazy dog.", voice: "laidback woman", });
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
⛔ Files ignored due to path filters (9)
src/libs/Together/Generated/Together..JsonSerializerContext.g.csis excluded by!**/generated/**src/libs/Together/Generated/Together.EvaluationsClient.EvaluationUpdate.g.csis excluded by!**/generated/**src/libs/Together/Generated/Together.IEvaluationsClient.EvaluationUpdate.g.csis excluded by!**/generated/**src/libs/Together/Generated/Together.JsonSerializerContextTypes.g.csis excluded by!**/generated/**src/libs/Together/Generated/Together.Models.EvaluationScoreResults.g.csis excluded by!**/generated/**src/libs/Together/Generated/Together.Models.EvaluationScoreResultsAggregatedScores.g.csis excluded by!**/generated/**src/libs/Together/Generated/Together.Models.EvaluationUpdateRequest.g.csis excluded by!**/generated/**src/libs/Together/Generated/Together.Models.EvaluationUpdateRequestResults.Json.g.csis excluded by!**/generated/**src/libs/Together/Generated/Together.Models.EvaluationUpdateRequestResults.g.csis excluded by!**/generated/**
📒 Files selected for processing (1)
src/libs/Together/openapi.yaml(2 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Test / Build, test and publish
🔇 Additional comments (1)
src/libs/Together/openapi.yaml (1)
996-1003: Unify status enums across responses and models (missing inference_error).
inference_errorexists on the update payload but not on GET /status nor EvaluationJob schema. Align enums across all three to avoid client-side enum drift.I can generate a follow-up diff once you confirm whether
inference_errorshould be public (GET/status + EvaluationJob) or internal-only (remove from update).Also applies to: 1048-1055, 3856-3863
Summary by CodeRabbit
New Features
Refactor