feat(docker): read package manifests from s3 (#259)

* feat(docker): read package manifests from s3

* fix(docker): add type assertions for manifest data

Fix Pyright type error in _load_manifest_data() by adding explicit
type casting for jsonlines.Reader return values. Also run Black
formatter to fix code formatting issues.

- Add cast() import from typing
- Cast manifest_meta to Dict
- Cast entries to List[Dict]
- Run Black formatter on Python files

Fixes CI type checking failures in PR #259.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: update CHANGELOG for v0.8.7

Document all changes in v0.8.7:

Added:
- Browse buttons for linked packages
- Read-only package manifest browsing from S3

Changed:
- PackageFileFetcher reads manifests from S3 (not quilt3.Package.browse())
- Added role_arn and region parameters for cross-account access

Fixed:
- Docker filesystem writes eliminated (closes #258)
- Pyright type error with explicit type casting
- Python code formatting (Black)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(deps): migrate to uv dependency-groups and refactor Makefile

- Migrate from [project.optional-dependencies] to [dependency-groups]
  for PEP 735 compliance and proper uv group support
- Add UVDEV variable to Makefile for improved readability
- Update install target to use --group dev --group docs
- Update lint and test-unit targets to use $(UVDEV)
- Fix pyright not being available in uv run context

This resolves the "Failed to spawn: pyright" error by ensuring
dev dependencies (pytest, black, isort, pyright) are properly
configured as a dependency group rather than optional dependencies.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore(deps): remove unused docs dependency group

Remove sphinx and sphinx-rtd-theme from dependency-groups as the
project uses markdown documentation instead of Sphinx-generated docs.

Updates:
- Remove docs group from pyproject.toml
- Update Makefile install target to only sync dev group
- Update uv.lock to reflect removed dependencies

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Update uv.lock

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: drernie

CHANGELOG.md

@@ -8,17 +8,23 @@ All notable changes to this project will be documented in this file.
 ### Added
 
 - Browse buttons for linked packages in Benchling Canvas
+- Read-only package manifest browsing directly from S3 (no local filesystem writes)
 
 ### Changed
 
 - **BREAKING**: Removed `--config` alias, use only `--profile` for all commands
+- `PackageFileFetcher` now reads manifests directly from S3 instead of using `quilt3.Package.browse()`
+- Added `role_arn` and `region` parameters to `PackageFileFetcher` for cross-account access
 
 ### Fixed
 
 - **CRITICAL**: `--profile` flag now works correctly for all commands
 - Configuration validation allows additional properties for backward compatibility
 - Status command works with any profile containing stackArn (not just integrated stacks)
 - Python package license corrected to Apache-2.0
+- Docker filesystem writes eliminated by reading package manifests directly from S3 (closes #258)
+- Pyright type error in `_load_manifest_data()` with explicit type casting
+- Python code formatting issues (Black formatter)
 
 ## [0.8.3] - 2025-11-18
 

docker/Makefile

@@ -23,6 +23,9 @@ PORT_NATIVE := 5001
 PORT_DOCKER_DEV := 5002
 PORT_DOCKER_PROD := 5003
 
+# UV command with dev group (for linting and testing)
+UVDEV = uv run --group dev
+
 .PHONY: help build test clean install lint ngrok kill \
         check-xdg check-ngrok run-native run-native-verbose run-dev run run-native-ngrok run-ecr \
         test test-unit test-integration test-dev test-docker-dev test-docker-prod test-deployed-dev test-deployed-dev-direct test-deployed-prod test-ecr test-native test-benchling test-query \
@@ -117,7 +120,7 @@ check-ngrok:
 
 # Install dependencies
 install:
-	uv sync --all-extras
+	uv sync --group dev
 
 # Test Benchling OAuth credentials from AWS Secrets Manager
 test-benchling: check-xdg
@@ -212,7 +215,7 @@ test-all: lint test-unit test-dev test-integration
 
 # Run unit tests with pytest
 test-unit:
-	uv run pytest -v
+	$(UVDEV) pytest -v
 
 # Run integration tests with execution monitoring (requires real Benchling credentials)
 test-integration: test-benchling
@@ -371,9 +374,9 @@ test-deployed-prod: check-xdg
 
 # Auto-fix code formatting (black + isort) and type check (pyright)
 lint:
-	uv run black src/ tests/ scripts/
-	uv run isort src/ tests/ scripts/
-	uv run pyright src/ tests/ scripts/
+	$(UVDEV) black src/ tests/ scripts/
+	$(UVDEV) isort src/ tests/ scripts/
+	$(UVDEV) pyright src/ tests/ scripts/
 
 # Deployment
 

docker/pyproject.toml

@@ -36,7 +36,7 @@ dependencies = [
     "click>=8.1.0",
 ]
 
-[project.optional-dependencies]
+[dependency-groups]
 dev = [
     "pytest==9.0.1",
     "pytest-mock==3.15.1",
@@ -47,10 +47,6 @@ dev = [
     "isort==7.0.0",
     "pyright==1.1.407",
 ]
-docs = [
-    "sphinx==8.2.3",
-    "sphinx-rtd-theme==3.0.2",
-]
 
 [project.urls]
 Homepage = "https://github.com/quiltdata/benchling-webhook"

docker/src/canvas.py

@@ -76,6 +76,8 @@ def __init__(
         self._package_file_fetcher = package_file_fetcher or PackageFileFetcher(
             catalog_url=config.quilt_catalog,
             bucket=config.s3_bucket_name,
+            role_arn=config.quilt_write_role_arn,
+            region=config.aws_region,
         )
 
     @property

docker/src/package_files.py

@@ -16,11 +16,18 @@
 - Package creation (see entry_packager.py)
 """
 
-from typing import List, Optional
+import io
+import json
+import os
+from typing import Dict, List, Optional, Tuple, cast
 
-import quilt3
+import jsonlines
 import structlog
+from quilt3.backends import get_package_registry
+from quilt3.packages import ManifestJSONDecoder
+from quilt3.util import PhysicalKey
 
+from .auth.role_manager import RoleManager
 from .packages import Package
 
 logger = structlog.get_logger(__name__)
@@ -82,25 +89,100 @@ class PackageFileFetcher:
     """Fetches file lists and metadata from Quilt packages.
 
     Responsibilities:
-    - Browse package contents via quilt3.Package.browse()
+    - Browse package manifests directly from S3 (no local writes)
     - List all files in a package with metadata
     - Fetch package-level metadata
-    - Create PackageFile instances from browse results
+    - Create PackageFile instances from manifest entries
 
     Designed to be reusable and cacheable to avoid repeated API calls.
     Does not modify packages, only reads their contents.
     """
 
-    def __init__(self, catalog_url: str, bucket: str):
+    def __init__(
+        self,
+        catalog_url: str,
+        bucket: str,
+        role_arn: Optional[str] = None,
+        region: Optional[str] = None,
+        role_manager: Optional[RoleManager] = None,
+    ):
         """Initialize fetcher.
 
         Args:
             catalog_url: Quilt catalog URL (e.g., "nightly.quilttest.com")
             bucket: S3 bucket name
+            role_arn: Optional IAM role ARN for cross-account access
+            region: Optional AWS region (defaults to AWS_REGION env or us-east-1)
+            role_manager: Optional RoleManager instance (used in tests)
         """
         self.catalog_url = catalog_url
         self.bucket = bucket
         self.logger = structlog.get_logger(__name__)
+        self.role_manager = role_manager or RoleManager(
+            role_arn=role_arn,
+            region=region or os.getenv("AWS_REGION", "us-east-1"),
+        )
+
+    def _get_registry(self):
+        """Create an S3-backed package registry."""
+        return get_package_registry(f"s3://{self.bucket}")
+
+    def _fetch_physical_key_bytes(self, physical_key: PhysicalKey) -> bytes:
+        """Read bytes from a PhysicalKey without touching the local filesystem."""
+        if physical_key.is_local():
+            with open(physical_key.path, "rb") as file:
+                return file.read()
+
+        s3_client = self.role_manager.get_s3_client()
+        params = {"Bucket": physical_key.bucket, "Key": physical_key.path}
+        if physical_key.version_id:
+            params["VersionId"] = physical_key.version_id
+        response = s3_client.get_object(**params)
+        return response["Body"].read()
+
+    def _load_manifest_data(self, package_name: str) -> Tuple[Dict, List[Dict]]:
+        """Load manifest metadata and entries for the latest package version."""
+        registry = self._get_registry()
+
+        top_hash = self._fetch_physical_key_bytes(registry.pointer_latest_pk(package_name)).decode("utf-8").strip()
+        manifest_bytes = self._fetch_physical_key_bytes(registry.manifest_pk(package_name, top_hash))
+
+        manifest_stream = io.StringIO(manifest_bytes.decode("utf-8"))
+        reader = jsonlines.Reader(manifest_stream, loads=ManifestJSONDecoder().decode)
+
+        manifest_meta = cast(Dict, reader.read())
+        entries = cast(List[Dict], list(reader))
+
+        return manifest_meta, entries
+
+    @staticmethod
+    def _is_valid_file_entry(entry: Dict) -> bool:
+        """Return True if manifest entry represents a real file."""
+        logical_key = entry.get("logical_key")
+        physical_keys = entry.get("physical_keys") or []
+
+        if not logical_key or logical_key.startswith(".quilt/"):
+            return False
+
+        return bool(physical_keys)
+
+    def _parse_entry_json(self, entries: List[Dict]) -> Optional[dict]:
+        """Load entry.json contents if present."""
+        entry_json = next((entry for entry in entries if entry.get("logical_key") == "entry.json"), None)
+        if not entry_json:
+            return None
+
+        physical_keys = entry_json.get("physical_keys") or []
+        if not physical_keys:
+            return None
+
+        try:
+            physical_key = PhysicalKey.from_url(physical_keys[0])
+            data = self._fetch_physical_key_bytes(physical_key)
+            return json.loads(data.decode("utf-8"))
+        except Exception as exc:  # pragma: no cover - defensive logging
+            self.logger.warning("Failed to load entry.json", error=str(exc))
+            return None
 
     def get_package_files(
         self,
@@ -124,18 +206,18 @@ def get_package_files(
         self.logger.info("Fetching package files", package_name=package_name)
 
         try:
-            # Browse package
-            pkg = quilt3.Package.browse(package_name, registry=f"s3://{self.bucket}")
-
-            # Collect all files (not directories)
-            files = []
-            for logical_key, entry in pkg.walk():
-                # Skip metadata files (internal to Quilt)
-                if logical_key.startswith(".quilt/"):
+            _, entries = self._load_manifest_data(package_name)
+
+            files: List[PackageFile] = []
+            for entry in entries:
+                if not self._is_valid_file_entry(entry):
                     continue
 
-                # Get file size
-                size = entry.size if hasattr(entry, "size") else 0
+                logical_key = entry["logical_key"]
+                try:
+                    size = int(entry.get("size", 0) or 0)
+                except (TypeError, ValueError):
+                    size = 0
 
                 files.append(
                     PackageFile(
@@ -147,7 +229,6 @@ def get_package_files(
                     )
                 )
 
-                # Respect max_files limit
                 if max_files and len(files) >= max_files:
                     break
 
@@ -183,16 +264,14 @@ def get_package_metadata(self, package_name: str) -> dict:
         self.logger.info("Fetching package metadata", package_name=package_name)
 
         try:
-            pkg = quilt3.Package.browse(package_name, registry=f"s3://{self.bucket}")
+            manifest_meta, entries = self._load_manifest_data(package_name)
 
-            # Try to read entry.json if it exists
-            if "entry.json" in pkg:
-                metadata = pkg["entry.json"]()  # Fetch and deserialize
+            entry_metadata = self._parse_entry_json(entries)
+            if entry_metadata is not None:
                 self.logger.info("Fetched entry.json metadata", package_name=package_name)
-                return metadata
+                return entry_metadata
 
-            # Otherwise return package-level metadata
-            metadata = pkg.meta or {}
+            metadata = manifest_meta.get("user_meta") or {}
             self.logger.info("Fetched package metadata", package_name=package_name)
             return metadata