Add S3 backend to cloudsync#18815
Conversation
* Ensure we're using C89 standards and other libretro standards * Use more appropriate logging levels so we don't spam the log * Fix incorrect S3 part size
Similar to how the WebDAV driver uses async tasks to do network operations I'm moving the MVP of S3 to do this also.
warmenhoven
left a comment
warmenhoven
left a comment
There was a problem hiding this comment.
Looks good, just a few minor things. I haven't tried compiling or running it yet.
libretro-common/net/net_http.c
Outdated
| } | ||
| } | ||
|
|
||
| fprintf(stderr, |
There was a problem hiding this comment.
I feel like this is better done as additional state on http_t that can then be logged by the caller.
There was a problem hiding this comment.
I added this while troubleshooting the -1 http status. I could definitely move it to http_t or even remove it if you prefer, thoughts?
There was a problem hiding this comment.
I like the idea of being able to introspect what's going on inside of net_http but verbosity/log isn't available and printf/fprintf is not a good answer. So I'd be more in favor of moving it so callers can do the logging. If that ends up being too big or too big for this pr I'd prefer it's just removed entirely (and possibly added in a new pr). Up to you if you want to do that as part of this pr, or just remove it for this one.
There was a problem hiding this comment.
Moved this to http_t. Lmk how it looks.
tasks/tasks_internal.h
Outdated
| const void *content, size_t content_len, const char *content_type, bool mute, | ||
| const char *headers, retro_task_callback_t cb, void *user_data); | ||
|
|
||
| void *task_push_http_transfer_with_content_ex(const char *url, const char *method, |
There was a problem hiding this comment.
tasks_internal.h isn't public api, it's fine to change it and avoid the _ex suffix.
| SETTING_ARRAY("cheevos_leaderboards_enable", settings->arrays.cheevos_leaderboards_enable, true, "", true); /* deprecated */ | ||
| #endif | ||
|
|
||
| #ifdef HAVE_NETWORKING |
There was a problem hiding this comment.
Why move this down a line?
There was a problem hiding this comment.
I was having a problem when I was finding the right place for some of the config and #ifdefs and had forced myself into a faux problem with netplay_mitm_server. Looks like I didn't revert that pushdown, I'll move it back up.
* Support extra prefix in URL * Cloudflare should support both virtual-host style and path-based URLs
|
I've merged everything except for the final commit, c289bd1, because I'm still indecisive on what I think a debug API for net_http should look like, and I didn't want to hold up the rest of it waiting on that. |
2 participants
Description
This PR adds an S3-compatible Cloud Sync backend to RetroArch.
S3-compatible object storage is now widely available, and the existing Cloud Sync manifest flow (
manifest.server/manifest.local) lets us scope uploads/downloads without requiring bucket list operations. That makes S3 a practical backend while keeping behavior consistent with existing cloud sync architecture.What this PR adds
CreateMultipartUploadUploadPartCompleteMultipartUploadAbortMultipartUploadHTTP/task-layer integration changes required for correctness
During integration/testing, multipart support required a few targeted internal changes:
net_httpnow accepts valid zero-length POST requests whenContent-Length == 0ETagheadersScope and platform enablement
Notes on multipart threshold strategy
Multipart thresholds are currently baseline-oriented and platform-conditioned in code. This is intended as a functional starting point. Future tuning could include:
Attribution
This feature was developed with AI-assisted coding/review and human direction for design, troubleshooting, refactoring, and completeness.
Related Issues
No formal issue was opened before implementation.
Integration work in this PR addresses the following discovered runtime issues:
ETagextraction requiring opt-in response header access behaviorRelated Pull Requests
None required in other repositories at this time.
Reviewers