Support HTTP Credential URL for AWS/S3 in Cyberduck

[sannies]

Is your feature request related to a problem? Please describe.
In many AWS workflows today, credentials are not static. Instead, a temporary credential issuance service runs behind a custom HTTP endpoint. The official AWS SDKs (e.g., JavaScript v3’s fromHttp() provider) support this pattern.
Cyberduck currently lacks the ability to fetch AWS credentials via an arbitrary HTTP endpoint and refresh them automatically.
Existing issues such as #8610, #10989, #14862 revolve around limitations in metadata or profile-based credential retrieval—but none address a generic HTTP credential endpoint.

Describe the solution you'd like
I implemented a solution that adds a new bookmark property in the Custom dict:

    <key>Custom</key>
    <dict>
        <key>aws.credentials.http.url</key>
       <string>https://up.content.example.com/api_v1/upload/VMHGTtd0BDYpcZgWU2arm6SwElXsiOCK</string>
    </dict>

When present:

• S3CredentialsConfigurator detects the property and returns placeholder credentials so that validation doesn’t block connection.
• S3Session.configureCredentialsStrategy skips other auth methods and creates an AWSSessionCredentialsRetriever with this URL.
• AWSSessionCredentialsRetriever has been modified to:
  • Use the provided URL literally (preserving path and encoding)
  • Perform HTTP GET and parse AWS STS-style JSON:

{ "AccessKeyId": "...", "SecretAccessKey": "...", "Token": "...", "Expiration": "..." }

If the property is not set, Cyberduck behaves exactly as before (profiles, metadata, etc.).

Describe alternatives you've considered

• Using EC2 Instance Metadata (169.254.169.254) as a proxy
  It is technically possible to run a local service (or SSH tunnel) that mimics the AWS metadata endpoint and returns credentials. However, this requires routing tricks, hosts file changes, root privileges, or VPN manipulation — fragile and environment-specific.
• Manually pasting temporary AWS credentials into Cyberduck
  This works only for short periods until credentials expire. It is error-prone, breaks automation, and is not feasible for non-technical users or expiring STS credentials in CI / upload pipelines.
• Writing credentials into ~/.aws/credentials using AWS CLI or scripts
  This requires storing STS tokens in plaintext on disk and still risks expiration. It also makes Cyberduck pick them up only if profile names and hostnames are aligned, which is not always the case.

Additional context
The implementation is already underway in branch http-credential-provider of the fork:
https://github.com/castlabs/cyberduck-sannies/tree/http-credential-provider
This demonstrates the added classes, modifications to S3CredentialsConfigurator, S3Session, and AWSSessionCredentialsRetriever, along with unit tests covering the new aws.credentials.http.url bookmark property.

[dkocher]

The current support to retrieve credentials from EC2 instance metadata is considered to support this requirement already. You can create a connection profile like the S3 (Credentials from Instance Metadata).cyberduckprofile ¹ to fetch credentials from a compatible IMDSv1 API ².

Footnotes

1. https://github.com/iterate-ch/profiles/blob/main/S3%20(Credentials%20from%20Instance%20Metadata).cyberduckprofile ↩

2. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html ↩

[sannies]

I understand your point - and it's true that Cyberduck can already retrieve credentials from EC2 instance metadata via a connection profile. However, this approach falls short for our scenario:

• The credential URL we use is not static like the IMDS endpoint.
• It contains a temporary, user-specific token, with a limited lifetime.
• Because of this, we cannot put it in a .cyberduckprofile file permanently — users would end up with a growing number of profiles containing expired tokens.
• We also can't reuse the same profile because each upload request generates a different pre-authorized credential URL, valid only for a specific user and target path.

Our application issues such URLs so users can securely upload content to specific S3 locations without static credentials.
In this workflow, it is much more natural to store the URL in the bookmark, not as a permanent profile - and let Cyberduck fetch credentials dynamically from it (similar to IMDS, but configurable).

That's why I've added support for a bookmark-level property as in the PR.

This keeps profiles reusable while allowing per-session HTTP-based STS credentials, without accumulating stale profiles.

Would you be open to discussing this extension at all?

[dkocher]

From your description you might want to look into STS. We support uses cases to connect with temporary credentials using the STS AssumeRole and AssumeRoleWithWebIdentity APIs.

In general we are very hesitant to add patches for specific for internal-use cases that we cannot test. In a generic way we could add support to lookup the property in the bookmark prior the default configuration in the profile as suggested in #17617.