Skip to content

Implement VolumeStorageAdapter to storage-proxy #7840

@jopemachine

Description

@jopemachine

Add VolumeStorageAdapter

Motivation

The current artifact import pipeline accesses storage through the AbstractStorage interface. However, to import artifacts directly into user vfolders, we need a way to use AbstractVolume (the volume backend interface) as AbstractStorage.

In the existing structure, only pre-configured storages like VFSStorage and ObjectStorage can be used, and users cannot specify their own vfolder as an import destination. To solve this, we need an adapter that wraps AbstractVolume into the AbstractStorage interface.

Required Features

  1. VolumeStorageAdapter class implementation
    • Implement AbstractStorage interface (stream_upload, stream_download, delete_file, get_file_info)
    • Delegate to AbstractVolume's native file operations (add_file, read_file, delete_files, mkdir)
    • VFolderID-based path resolution
  2. StorageTarget class implementation
    • Wrap storage name (str) or AbstractStorage instance
    • resolve() method: lookup from StoragePool or return instance directly
    • name property: return storage name
  3. Import API extension
    • Add vfid field to HuggingFaceImportModelsReq and ReservoirImportModelsReq DTOs
    • Add logic in API handlers to create VolumeStorageAdapter when vfid is provided
  4. ImportStepContext update
    • Change storage_step_mappings type to dict[ArtifactStorageImportStep, StorageTarget]

Impact

Component Impact
`ai.backend.common.artifact_storage Add StorageTarget clas
`ai.backend.common.dto.storage.request Add vfid field to import DTO
`ai.backend.storage.storages.volume_adapter New file: VolumeStorageAdapter clas
`ai.backend.storage.api.v1.registries.huggingface Add vfid handling logi
`ai.backend.storage.api.v1.registries.reservoir Add vfid handling logi
`ai.backend.storage.services.artifacts.* Change to use `StorageTarget.resolve()

Testing Scenarios

  1. VolumeStorageAdapter unit tests
    • stream_upload: Verify file exists in vfolder after upload
    • stream_download: Download uploaded file and verify content matches
    • delete_file: Verify file no longer exists after deletion
    • get_file_info: Verify accuracy of file metadata (size, type, modification time)
    • Verify nested directory path handling
  2. StorageTarget unit tests
    • StorageTarget(str): Verify resolve() looks up from StoragePool
    • StorageTarget(AbstractStorage): Verify resolve() returns instance directly
    • name property: Verify correct name returned for both string and instance
  3. Import API integration tests
    • Request without vfid: Verify existing behavior maintained (storage lookup from StoragePool)
    • Request with vfid: Verify VolumeStorageAdapter created and files saved to vfolder
    • Verify correct operation across different volume backends (VFS, XFS, etc.)
  4. Error cases
    • Verify appropriate error returned when requesting with non-existent volume name
    • Verify error handling for invalid vfid format

JIRA Issue: BA-3790

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions