Support mixed-media manifests for items with both images and videos

[Copilot]

IIIF manifest generation assumed homogeneous media types per item. Items like 2025-highland-house-walkthrough-ma containing both images and videos only rendered one type in the manifest.

Changes

Added mixed-media detection (check_mixed_media)

• Scans for original images (JPEG, PNG, GIF, TIFF) and videos (MPEG4, h.264, etc.)
• Excludes thumbnails by format and filename patterns
• Returns (has_images, has_videos) tuple

Added canvas creation helpers

• create_image_canvas() - IIIF Image API v3 canvases with Cantaloupe service integration
• create_video_canvas() - Video canvases with derivative choices and VTT subtitle support

Modified create_manifest3()

• Checks for mixed media before mediatype-based routing
• When detected, processes all originals sorted by name
• Creates appropriate canvas types per file format
• Falls through to existing logic for single-mediatype items

Added IMAGE_FORMATS constant

• Eliminates duplicate format lists across detection and processing logic

Example

Before: Video-only manifest despite 30+ images present

{
  "items": [
    {"type": "Canvas", "duration": 123.45, ...}
  ]
}

After: Complete manifest with all media

{
  "items": [
    {"type": "Canvas", "height": 1800, "width": 1200, ...},  // Image
    {"type": "Canvas", "height": 1800, "width": 1200, ...},  // Image
    {"type": "Canvas", "duration": 123.45, ...}              // Video
  ]
}

Backward compatible - only affects items with both image and video originals.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• archive.org
  • Triggering command: /usr/bin/curl curl -s REDACTED (dns block)
  • Triggering command: /usr/bin/python3 python3 -m unittest tests.test_mixed_media -v (dns block)
  • Triggering command: /usr/bin/python3 python3 -m unittest tests.test_mixed_media tests.test_resolver -v (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Support Mixed-Media Manifests for Items with both Images and Video</issue_title>
<issue_description># Issue: Support Multi-media manifests for items containing both Images and Videos

Description

Currently, IIIF manifest generation (specifically in iiif/resolver.py) seems to assume a single mediatype (often video) for items that actually contain both images and videos. As noted by @mekarpeles , multi-media archive.org items don't produce complete manifests that enable access to all relevant files.

The goal is to move away from strictly mirroring the IA "Theater" view and instead present all original files as canvases within the manifest, ensuring viewers like Clover or Theseus can render the full collection.

Proposed Implementation

In iiif/resolver.py, we should extend create_manifest3 to check for the presence of both images and videos within the metadata's file list. Proposed by AI agent:

def create_manifest3(identifier, domain=None, page=None):
    # ... existing setup code ...
    
    mediatype = metadata.get("mediatype")
    
    # NEW: Check if this is a mixed-media item
    has_images = any(f.get('format') == 'JPEG' and f.get('source') == 'original' 
                     for f in metadata.get('files', []))
    has_videos = any(f.get('format') in ['MPEG4', 'h.264 IA'] 
                     for f in metadata.get('files', []))
    
    if has_images and has_videos:
        # Handle mixed media
        for file in sorted(metadata.get('files', []), key=lambda x: x.get('name', '')):
            if file.get('source') != 'original':
                continue
                
            file_format = file.get('format')
            file_name = file.get('name')
            
            if file_format == 'JPEG':
                # Add image canvas
                canvas = create_image_canvas(identifier, file, metadata, domain)
                manifest.add_item(canvas)
                
            elif file_format in ['MPEG4', 'h.264 IA']:
                # Add video canvas
                canvas = create_video_canvas(identifier, file, metadata, domain)
                manifest.add_item(canvas)
    
    elif mediatype.lower() == 'movies':
        # ... existing video-only logic ...
    
    elif mediatype.lower() == 'image':
        # ... existing image-only logic ...

Key Considerations from Discussion

• Thumbnail Case: We should distinguish between primary media and supporting thumbnails. If we ignore the thumbnail case (where a video just has associated frames), we can treat items with non-thumbnail images and videos as "multi-media" and fallback otherwise.
• IA Theater Parity: As discussed with Mark Baggett, this involves a choice to ignore the current IA theater display logic in favor of showing all original files together.
• Viewer Compatibility: David Schober noted that while RAMP may struggle, viewers like Clover handle multiple content types on a canvas well, though video can occasionally hit CORS issues.
• Reference Item: 2025-highland-house-walkthrough-ma
  • This item has ~30 unique files (images and video) but the current manifest defaults to a single item/video view.

Related Issues

• Ref: Representing video thumbnails for frame navigation #56 (Video thumbnails)
• Ref: Issue with thumbnail nav #156 (Thumbnail-nav behavior)</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Support Mixed-Media Manifests for Items with both Images and Video #157

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[Copilot]

Pull request overview

This pull request adds support for mixed-media IIIF manifests that can handle items containing both images and videos, addressing a limitation where the manifest generator assumed homogeneous media types per item.

Changes:

• Introduced check_mixed_media() function to detect items with both original images and videos, excluding thumbnails
• Added create_image_canvas() and create_video_canvas() helper functions to generate appropriate IIIF v3 canvases for each media type
• Modified create_manifest3() to check for mixed-media items before routing to mediatype-specific logic, processing all originals when both types are detected

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File	Description
iiify/resolver.py	Added IMAGE_FORMATS constant, mixed-media detection function, canvas creation helpers, and integrated mixed-media handling into manifest generation
tests/test_mixed_media.py	Added unit tests for mixed-media detection logic and integration test for manifest structure

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Canvas IDs for images in mixed-media items don't include a sequential number (line 783 generates {identifier}/{slugged_id}/canvas), but the existing multi-file image handling includes pageCount in the canvas ID (line 1079: {identifier}${pageCount}/canvas). This inconsistency could cause issues if canvas IDs are expected to follow a specific pattern. Consider using a consistent ID pattern, either including the canvas number or ensuring the slugged filename is sufficient for uniqueness.

[Copilot]

The os module is imported but never used in this test file. Consider removing this unused import.

Suggested change

import os

[Copilot]

The exception handling catches all exceptions broadly (except Exception as e:), which could mask unexpected errors like KeyError or AttributeError from bugs in the code. The existing image handling at lines 1084-1089 catches specific exceptions (requests.exceptions.HTTPError) and creates a fallback canvas with error information. Consider either: (1) catching specific expected exceptions, or (2) creating a fallback canvas similar to the existing pattern, or (3) logging the error more robustly rather than just printing.

[Copilot]

The exception handling catches all exceptions broadly (except Exception as e:), which could mask unexpected errors. Consider catching specific expected exceptions or logging errors more robustly. The existing image handling at lines 1084-1089 catches specific exceptions and creates fallback canvases.

[Copilot]

The canvas_number parameter is defined but never used in the function body. This parameter appears to be unnecessary since the canvas ID is generated from the file name, not from a sequential number.

Suggested change

	def create_image_canvas(identifier, file, metadata, domain, canvas_number):
	def create_image_canvas(identifier, file, metadata, domain, _canvas_number):

[Copilot]

The test is using hasattr(body, 'items') on line 107, but body is a dict (as checked on line 105). The hasattr check will always return True because dicts have an 'items' method. This should check if 'items' is a key in the dict using 'items' in body or check if body is a Choice object. Similarly, line 110 checks hasattr(item, 'type') but item could be a dict from body.items (the dict method), not from the Choice items list.

Suggested change

	# Handle both direct body and Choice bodies
	if isinstance(body, dict):
	canvas_types.add(body.get('type', 'Unknown'))
	elif hasattr(body, 'items'):
	# Choice body
	for item in body.items:
	if hasattr(item, 'type'):
	canvas_types.add(item.type)
	# Handle both direct body and Choice bodies (IIIF v3 JSON is dict-based)
	if isinstance(body, dict):
	body_type = body.get('type')
	if body_type == 'Choice' and 'items' in body:
	# Choice body: iterate through the list of items
	for item in body['items']:
	if isinstance(item, dict) and 'type' in item:
	canvas_types.add(item['type'])
	else:
	# Direct body: use its type
	canvas_types.add(body.get('type', 'Unknown'))

[Copilot]

The thumbnail exclusion logic is duplicated between check_mixed_media (lines 762-764) and the mixed-media processing loop (lines 957-959). This creates a maintenance risk if the criteria for identifying thumbnails changes. Consider extracting this into a helper function like is_thumbnail_image(file_format, file_name) to ensure consistency.