[worker] Remove legacy upload code, add retries, auto-waive builds#3848
[worker] Remove legacy upload code, add retries, auto-waive builds#3848sjchmiela wants to merge 9 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3848 +/- ##
==========================================
+ Coverage 58.26% 58.42% +0.17%
==========================================
Files 917 917
Lines 39827 39977 +150
Branches 8350 8416 +66
==========================================
+ Hits 23200 23353 +153
- Misses 15193 16529 +1336
+ Partials 1434 95 -1339 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Subscribed to pull request
Generated by CodeMention |
| Authorization: `Bearer ${robotAccessToken}`, | ||
| }, | ||
| shouldThrowOnNotOk: false, | ||
| retries: UPLOAD_API_REQUEST_RETRIES, |
There was a problem hiding this comment.
https://github.com/expo/eas-cli/blob/main/packages/worker/src/utils/turtleFetch.ts#L53
so before we explicitly did not retry on POST, are we sure we can now without some form of ensuring idempotency?
There was a problem hiding this comment.
we did not retry on POST _by default).
i think so:
- creating upload sessions
- in builds appends to the array (limit of 5 but i don't think we're going to hit that)
- in job runs creates workflow artifacts (unique index on name, but if the request failed then i'd rather fail trying again than no trying again?)
- saving artifacts ensures a given value is set for a given key so it's idempotent
next time can you please try to answer this question yourself and give a conclusive review to unblock?
There was a problem hiding this comment.
Took a deeper look since the retry thread touches the server side too. This looks good to me.
One suggestion and one question:
Suggestion: SystemError.metadata may leak into user-visible build logs
[findAndUploadBuildArtifacts.ts#L29](
), [#L36]( ), and [uploadArtifact.ts#L131]( ) uselogger.error('...', err), which inspect-prints enumerable properties and now includes metadata (presigned upload URLs and response payloads from the cause chain).
The practical risk seems low since the URLs are PUT-only and short-lived, but it does add credential-shaped noise to user-visible logs. Using logger.error({ err }, '...') would route through Bunyan's serializers and avoid logging the metadata.
Question: does "auto-waive builds" need a www-side companion?
I may be missing something here, but from my reading of the code, the current SERVER_ERROR auto-waive logic appears to exist only for job runs ([TurtleJobRunsRouter.ts#L183-L186](https://github.com/expo/universe/blob/81a0f3a38f2498c56b4360ca9d6ba3fe944dcb08/server/www/src/apiv2/TurtleJobRunsRouter.ts#L183-L186)). The orchestrator intentionally never sends waivedAt for builds ([update_www.go#L54](https://github.com/expo/workflow-orchestration/blob/9241d2cef9a8a11b29436253c23605e995a43718/internal/utils/update_www.go#L54)), and I couldn't find equivalent handling on the build-update path.
If I'm understanding that correctly, builds failing these uploads would still be recorded as SERVER_ERROR and remain billable. Is there another piece I'm missing, or should the scope here be clarified to job runs only?
|
⏩ The changelog entry check has been skipped since the "no changelog" label is present. |
|
Fixed Adding waiving system error builds in https://github.com/expo/universe/pull/28145. |
What changed
gcsSignedUploadUrlForApplicationArchiveandgcsSignedUploadUrlForBuildArtifactsfrom worker runtime config and runtime config typing.SystemErrors with internal tracking codes while keeping the public error code asSERVER_ERROR.turtleFetchcalls for upload-session creation and artifact metadata saving.Why
These signed upload URL fields are no longer expected, so keeping fallback paths made the worker handle an obsolete runtime contract. Upload failures in these managed paths are infrastructure/API/storage failures rather than user-actionable project issues.
Validation
yarn oxfmt packages/worker/src/config.ts packages/worker/src/upload.ts packages/worker/src/external/turtle.ts packages/worker/src/__unit__/upload.test.tsyarn workspace @expo/worker test:unit src/__unit__/upload.test.ts --runInBandyarn workspace @expo/worker typecheck