S3-aware proxy mode for transparent P2P S3 downloads

[YQ-Wang]

Feature request:

Add an S3-aware proxy mode to dfdaemon so that authenticated S3 reads (GetObject) can be accelerated through Dragonfly P2P without requiring any application code changes. Existing AWS SDK and CLI clients opt in by setting HTTPS_PROXY and trusting the proxy CA certificate.

• GetObject: accelerated through Dragonfly P2P.
• Ranged GetObject: accelerated through Dragonfly P2P. If the caller's SigV4 signature explicitly covers the Range header, dfdaemon preserves that original signed Range on the source request instead of rewriting it.
• HeadObject and ListObjectsV2: direct passthrough (no P2P).
• Other S3 APIs: passthrough, not accelerated.
• The proxy preserves caller-provided SigV4 Authorization headers, X-Amz-* headers, and presigned URL query parameters end-to-end. dfdaemon does not discover AWS credentials or re-sign origin requests.
• S3 request classification runs before the existing generic proxy rules, controlled by a new proxy.s3 configuration block with an allowlist of hosts/suffixes and optional automatic detection of common AWS S3 endpoint patterns.
• Task ID generation strips per-request SigV4 query parameters (X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-Signature, X-Amz-Security-Token, X-Amz-SignedHeaders, X-Amz-Algorithm) so that different presigned URLs for the same object share the same P2P task.

Use case:

Organizations that store large ML models, datasets, or media files in S3 and distribute them across many hosts (e.g. during fleet-wide deployments or batch inference) currently cannot leverage Dragonfly P2P without rewriting download logic to use Dragonfly's API directly. With S3-aware proxy mode, those workloads can be accelerated transparently:

1. Deploy dfdaemon with proxy.s3.enable: true and a persistent CA cert/key pair.
2. Configure application hosts to trust the proxy CA (via AWS_CA_BUNDLE or the system trust store).
3. Set HTTPS_PROXY=http://127.0.0.1:4001 in the application environment.

All GetObject downloads then automatically go through Dragonfly P2P, reducing origin bandwidth and improving download speed across the fleet. HeadObject, ListObjectsV2, and other S3 operations pass through unmodified, so existing workflows (listing, metadata checks, uploads) are unaffected.

UI Example:

Client configuration (dfdaemon.yaml):

proxy:
  server:
    port: 4001
    caCert: /etc/dragonfly/certs/proxy-ca.pem
    caKey: /etc/dragonfly/certs/proxy-ca-key.pem
  s3:
    enable: true
    detectAwsEndpoints: true
    hosts: []
    hostSuffixes: []

Application environment:

export HTTPS_PROXY=http://127.0.0.1:4001
export HTTP_PROXY=http://127.0.0.1:4001
export AWS_CA_BUNDLE=/etc/dragonfly/certs/proxy-ca.pem

# Standard AWS CLI and SDK calls are now accelerated automatically
aws s3 cp s3://my-bucket/large-model.bin ./large-model.bin

[YQ-Wang]

#1779

[EvanCley]

#1779

@YQ-Wang please new a pr again if needed.

[YQ-Wang]

@EvanCley Thanks, I plan to address the range issue first in #1820. With that, we can just use the proxy support in the different client libraray.