Implement the ability to have 'remote' S3 resources exposed to Agents

[ryancormack]

Summary

Currently, Amazon Q CLI agents only support local file resources using the file:// scheme. This proposal suggests extending resource support to include S3 objects using the s3:// scheme, enabling organizations to centrally store and manage contextual resources for their agents.

Current Behaviour

Supported Resource Types

Currently, agents can only reference local files in their resources configuration:

{
  "resources": [
    "file://README.md",
    "file://docs/**/*.md",
    "file://.amazonq/rules/coding-standards.md"
  ]
}

Implementation Details

• The agent schema restricts resources to file:// URIs only: "pattern": "^(file://)"
• The context manager filters out any resources that don't start with file://
• Resource paths are treated as local filesystem paths after stripping the file:// prefix

Limitations

This approach has several limitations for organizations:
• No centralized resource management. I can't share 'company Terraform conventions' with every IaC Agent. If I copy that into each repo, I can't easily bulk update.
• No guarantee that all team members (and their Agents) have the latest versions
• Large organizations can't efficiently distribute agent resources

Proposed Feature: S3 Resource Support

Enhanced Resource Schema

Extend the resource pattern to support S3 URIs:

{
  "resources": [
    "file://local-config.md",
    "s3://company-agents/shared/coding-standards.md",
    "s3://team-bucket/project-context/**/*.md"
  ]
}

Benefits

• Store shared resources in S3 buckets with proper versioning
• Ensure all team members use the same version of contextual resources
• Support large organizations with distributed teams
• Track resource access and modifications through CloudTrail. These resources could then easily be managed in source control and updated via IaC

Implementation Considerations

AWS Profile Configuration

Support configurable AWS profiles while defaulting to standard credential resolution:

{
  "resourceSettings": {
    "s3_resources": {
      "profile": "company-agents-profile",
      "region": "us-east-1"
    }
  }
}

If not specified, use the standard AWS credential resolution chain (environment variables, IAM roles, default profile, etc.).

Error Handling

When S3 resources are inaccessible:
• Fail agent initialization with clear error messages. The agent can't do its job without the correct information
• Provide specific error details (access denied, object not found, network issues)

Caching Strategy

• Cache downloaded S3 objects locally with TTL, or use an etag and head request.

• Could possibly extend the s3_resource settings to always fetch latest
  • Respect S3 object metadata (ETag, Last-Modified) for efficient updates

Glob Pattern Support

Extend existing glob pattern support to S3:
• s3://bucket/path/**/.md - All markdown files recursively
• s3://bucket/configs/.json - All JSON files in configs directory

This may require limits as there could be loads of files in here and we'd need to do a ListObjects and then GetObject for a potentially large number.

Current Workaround

Organizations currently needing centralized resources must use agent hooks:

{
  "hooks": {
    "agentSpawn": [
      {
        "command": "aws s3 cp s3://company-agents/coding-standards.md ./temp-standards.md"
      },
      {
        "command": "aws s3 sync s3://company-agents/shared-docs/ ./temp-docs/"
      }
    ]
  },
  "resources": [
    "file://./temp-standards.md",
    "file://./temp-docs/**/*.md"
  ]
}

Workaround Limitations

• This works, but is a bit messy and not super obvious/clear what's going on.
• These files likely need excluded from any source control tracking as they're not part of a given repo.

Technical Implementation Notes

The codebase already hints at extensibility in wrapper_types.rs:
rust
// You can extend this list via "|". e.g. r"^(file://|database://)"
#[schemars(regex(pattern = r"^(file://)"))]

Required Changes

1. Schema Update: Modify schemas/agent-v1.json to allow s3:// pattern
2. Resource Handler: Extend context manager to detect and handle S3 URIs
3. AWS Integration: Add S3 client with proper credential handling
4. Error Management: Implement comprehensive error handling for S3 operations
5. Caching Layer: Add local caching for performance and offline capability

Backward Compatibility

This change would be fully backward compatible - existing file:// resources would continue to work unchanged.

Use Cases

Enterprise Development Teams

{
  "resources": [
    "s3://company-standards/coding-guidelines.md",
    "s3://company-standards/security-requirements.md",
    "s3://project-docs/api-specifications/**/*.yaml"
  ]
}

Multi-Environment Configurations

{
  "resources": [
    "s3://dev-configs/database-schemas.sql",
    "s3://shared-knowledge/troubleshooting-guides/**/*.md"
  ]
}

Compliance and Governance

{
  "resources": [
    "s3://compliance-docs/data-handling-policies.md",
    "s3://audit-requirements/logging-standards.md"
  ]
}