Skip to content

Conversation

@mihow
Copy link
Collaborator

@mihow mihow commented Oct 1, 2025

Summary

Previously we omitted the region parameter for simplicity because the S3 implementations in the university compute clouds do not use regions and the default region worked for existing AWS S3 users. Now we have some users in Europe that need to specify the S3 region in order for the storage backend to sync, to display source images in the UI, and to processes source images.

List of Changes

  • Adds a region field to the S3StorageSource model
  • Passes region field to necessary functions
  • Fixes some existing type annotation errors

How to Test the Changes

.

Screenshots

If applicable, add screenshots to help explain this PR (ex. Before and after for UI changes).

Deployment Notes

.

Checklist

  • I have tested these changes appropriately.
  • I have added and/or modified relevant tests.
  • I updated relevant documentation or comments.
  • I have verified that this PR follows the project's coding standards.
  • Any dependent changes have already been merged to main.

Summary by CodeRabbit

  • New Features

    • Optional S3 region support for storage sources, including region-aware bucket handling.
  • Chores

    • Database migration added to store an optional region for S3 storage entries.
    • Configuration updated to expose a test-region setting for S3.
  • Tests

    • Test fixtures and utilities updated to include and propagate the new region setting.

✏️ Tip: You can customize this high-level summary in your review settings.

@mihow mihow marked this pull request as ready for review October 1, 2025 12:16
@netlify
Copy link

netlify bot commented Oct 1, 2025

Deploy Preview for antenna-preview canceled.

Name Link
🔨 Latest commit 05a1c81
🔍 Latest deploy log https://app.netlify.com/projects/antenna-preview/deploys/69689ba1c8fac3000833daa1

Copilot AI review requested due to automatic review settings October 1, 2025 12:16
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for S3 regions to enable users in different AWS regions (particularly Europe) to properly configure their S3 storage backends. Previously, the region parameter was omitted for simplicity as university clouds didn't require it and the default region worked for existing AWS users.

  • Adds optional region field to the S3StorageSource model and corresponding configuration
  • Updates S3 client initialization to properly handle region configuration
  • Fixes existing type annotation issues in S3 utility functions

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
ami/main/models.py Adds nullable region field to S3StorageSource model and includes it in the config property
ami/main/migrations/0075_s3storagesource_region.py Database migration to add the region field to existing S3 storage sources
ami/utils/s3.py Updates S3 client/session initialization to use region and fixes type annotations
ami/tests/fixtures/storage.py Updates test fixture to include region parameter
config/settings/base.py Adds S3_TEST_REGION environment variable configuration

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@mihow mihow force-pushed the fix/add-s3-region branch from aff251a to 15a4edd Compare December 6, 2025 00:59
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Dec 6, 2025

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

📝 Walkthrough

Walkthrough

Added an optional region field to S3 configuration across model, config dataclass, boto3 session/client/resource usage, tests, and settings; included a Django migration to persist the new region field and a new create_bucket helper that is region-aware.

Changes

Cohort / File(s) Summary
S3 utils & API integration
ami/utils/s3.py
Added `region: str
Settings
config/settings/base.py
Added S3_TEST_REGION module-level variable (defaults from MINIO_REGION / None).
Model & Migration
ami/main/models.py, ami/main/migrations/0079_s3storagesource_region.py
Added region = models.CharField(max_length=255, null=True, blank=True, help_text=...) to S3StorageSource; updated S3StorageSource.config to include region=self.region; migration 0079 to add field.
Tests / Fixtures
ami/tests/fixtures/storage.py
Propagated region into S3Config in fixtures and added region default to S3StorageSource test defaults.

Sequence Diagram(s)

(omitted — changes are configuration additions, API client adjustments and a small helper; they do not introduce a new multi-component sequential flow requiring visualization)

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 I found a new field where regions play,
I hopped from model to config on my way,
Boto3 now knows which skies to roam,
Tests and fixtures echo it back home,
A tiny leap — S3 finds its way.

🚥 Pre-merge checks | ✅ 1 | ❌ 2
❌ Failed checks (2 warnings)
Check name Status Explanation Resolution
Description check ⚠️ Warning The description is largely incomplete with minimal content in several required sections and placeholder text ('.') in key deployment and testing sections. Expand 'How to Test the Changes' with specific testing instructions and commands; provide concrete 'Deployment Notes' for database migration and configuration steps; clarify documentation updates made.
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (1 passed)
Check name Status Explanation
Title check ✅ Passed The title 'Add support for S3 regions' directly and clearly summarizes the main change in the pull request - adding region support to S3 storage configuration.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings

🧹 Recent nitpick comments
ami/utils/s3.py (1)

103-128: Consider simplifying the duplicated client creation logic.

Both branches pass identical parameters except for endpoint_url. Since the session already contains credentials and region from get_session(), you could simplify this:

♻️ Suggested simplification
 def get_s3_client(config: S3Config) -> S3Client:
     session = get_session(config)
 
     # Always use signature version 4
     boto_config = botocore.config.Config(signature_version="s3v4")
 
-    if config.endpoint_url:
-        client = session.client(
-            service_name="s3",
-            endpoint_url=config.endpoint_url,
-            aws_access_key_id=config.access_key_id,
-            aws_secret_access_key=config.secret_access_key,
-            region_name=config.region,
-            config=boto_config,
-        )
-    else:
-        client = session.client(
-            service_name="s3",
-            aws_access_key_id=config.access_key_id,
-            aws_secret_access_key=config.secret_access_key,
-            region_name=config.region,
-            config=boto_config,
-        )
+    client = session.client(
+        service_name="s3",
+        endpoint_url=config.endpoint_url,  # None is acceptable here
+        config=boto_config,
+    )
 
     client = typing.cast(S3Client, client)
     return client

Note: Re-passing credentials is unnecessary since the session already holds them, and endpoint_url=None is valid (boto3 ignores it).


📜 Recent review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between fa47749 and 05a1c81.

📒 Files selected for processing (3)
  • ami/main/migrations/0079_s3storagesource_region.py
  • ami/main/models.py
  • ami/utils/s3.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • ami/main/models.py
🧰 Additional context used
🪛 Ruff (0.14.11)
ami/main/migrations/0079_s3storagesource_region.py

7-9: Mutable class attributes should be annotated with typing.ClassVar

(RUF012)


11-22: Mutable class attributes should be annotated with typing.ClassVar

(RUF012)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: test
🔇 Additional comments (7)
ami/main/migrations/0079_s3storagesource_region.py (1)

1-22: LGTM!

The migration is well-structured. The region field is appropriately defined as optional (null=True, blank=True) with clear help text guiding users on when to specify a region. The static analysis warnings about ClassVar are false positives—Django migrations intentionally use mutable class attributes for dependencies and operations.

ami/utils/s3.py (6)

33-41: LGTM!

The region field is correctly added with an appropriate type and default value, maintaining backward compatibility.


94-100: LGTM!

Correctly propagates the region to the boto3 session.


131-141: LGTM!

The region and signature version configuration are correctly applied to the S3 resource.


144-177: LGTM!

The region-aware bucket creation logic correctly handles the AWS requirement for CreateBucketConfiguration on non-us-east-1 regions while maintaining compatibility with Swift/MinIO. The docstring clearly documents this is primarily for testing purposes.


615-629: LGTM!

The comment explains the type ignore rationale well—StreamingBody implements the file-like interface that PIL.Image.open expects.


708-719: LGTM!

Explicitly setting region=None in the test configuration is clear and consistent with the MinIO setup that doesn't require regions.

✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
ami/tests/fixtures/storage.py (1)

25-39: Consider passing region to S3StorageSource defaults.

The S3_TEST_CONFIG now includes region, but create_storage_source doesn't pass it to the S3StorageSource.objects.get_or_create defaults. This could lead to inconsistency where the test config has a region but the created storage source doesn't.

         defaults=dict(
             bucket=S3_TEST_CONFIG.bucket_name,
             prefix=prefix,
             endpoint_url=S3_TEST_CONFIG.endpoint_url,
             access_key=S3_TEST_CONFIG.access_key_id,
             secret_key=S3_TEST_CONFIG.secret_access_key,
             public_base_url=S3_TEST_CONFIG.public_base_url,
+            region=S3_TEST_CONFIG.region,
         ),
ami/utils/s3.py (1)

131-139: Consider passing region_name and config to the resource for consistency.

Unlike get_s3_client, this function doesn't explicitly pass region_name or the botocore.config.Config with signature version. While the session inherits the region from get_session(), explicitly passing these parameters would ensure consistent behavior and make the code more explicit.

 def get_resource(config: S3Config) -> S3ServiceResource:
     session = get_session(config)
+    boto_config = botocore.config.Config(signature_version="s3v4")
     s3 = session.resource(
         "s3",
         endpoint_url=config.endpoint_url,
-        # api_version="s3v4",
+        region_name=config.region,
+        config=boto_config,
     )
     s3 = typing.cast(S3ServiceResource, s3)
     return s3
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f899482 and 15a4edd.

📒 Files selected for processing (5)
  • ami/main/migrations/0075_s3storagesource_region.py (1 hunks)
  • ami/main/models.py (2 hunks)
  • ami/tests/fixtures/storage.py (1 hunks)
  • ami/utils/s3.py (6 hunks)
  • config/settings/base.py (1 hunks)
🧰 Additional context used
🪛 Ruff (0.14.7)
ami/main/migrations/0075_s3storagesource_region.py

7-9: Mutable class attributes should be annotated with typing.ClassVar

(RUF012)


11-17: Mutable class attributes should be annotated with typing.ClassVar

(RUF012)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: test
🔇 Additional comments (9)
config/settings/base.py (1)

448-452: LGTM!

The new S3_TEST_REGION setting follows the established pattern for S3 test configuration and appropriately defaults to None for backward compatibility with existing setups that don't require region specification.

ami/main/models.py (1)

1389-1423: LGTM!

The region field is properly added to S3StorageSource and correctly propagated to S3Config in the config property. The nullable design appropriately maintains backward compatibility for existing storage sources and S3-compatible backends that don't require region specification.

ami/main/migrations/0075_s3storagesource_region.py (1)

1-17: LGTM!

Standard Django migration for adding the nullable region field. The static analysis hints about ClassVar annotations are false positives — Django migrations conventionally define dependencies and operations as class attributes without type annotations.

ami/utils/s3.py (6)

13-13: The boto3.session import is used.

Contrary to the previous review comment, this import is used for the return type annotation on line 94: def get_session(config: S3Config) -> boto3.session.Session. The import should be retained.


33-41: LGTM!

The region field is properly added to the S3Config dataclass with an appropriate None default for backward compatibility.


94-100: LGTM!

The session now correctly receives region_name from the config, ensuring region-aware S3 operations.


103-128: LGTM! The client now consistently uses Signature Version 4 and the region parameter.

Minor note: The credentials are passed to both the session (via get_session) and the client. This redundancy is harmless—the client parameters take precedence—but could be simplified in a future refactor.


599-601: LGTM!

Good improvement extracting Body to a separate variable. The StreamingBody is file-like but type checkers don't recognize it, so the type: ignore comment is appropriate.


684-695: LGTM!

The test function is properly updated with the region=None parameter to match the updated S3Config signature.

@netlify
Copy link

netlify bot commented Dec 24, 2025

Deploy Preview for antenna-ssec canceled.

Name Link
🔨 Latest commit 05a1c81
🔍 Latest deploy log https://app.netlify.com/projects/antenna-ssec/deploys/69689ba1c0806a0008c89e92

mihow and others added 5 commits January 14, 2026 23:30
Ensure consistency between S3_TEST_CONFIG and created S3StorageSource
instances by passing the region parameter to the defaults dict. This
prevents potential inconsistencies where the test config has a region
but the created storage source doesn't.

Co-Authored-By: Claude <[email protected]>
Update get_resource() to explicitly pass region_name and botocore
Config with signature version 4, matching the pattern used in
get_s3_client(). This ensures consistent S3 behavior across both
client and resource interfaces.

Co-Authored-By: Claude <[email protected]>
AWS requires CreateBucketConfiguration with LocationConstraint for
buckets created outside us-east-1. Updated create_bucket() to handle
this properly while maintaining compatibility with Swift/MinIO.

Added docstring noting this function is primarily used for testing,
as production users create their own buckets.

Co-Authored-By: Claude <[email protected]>
Add guidance that Swift/MinIO users should leave the region field
blank, while AWS users should specify their region (e.g., 'us-east-1',
'eu-west-1').

Co-Authored-By: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants