-
Notifications
You must be signed in to change notification settings - Fork 164
Closed
Description
Add VolumeStorageAdapter
Motivation
The current artifact import pipeline accesses storage through the AbstractStorage interface. However, to import artifacts directly into user vfolders, we need a way to use AbstractVolume (the volume backend interface) as AbstractStorage.
In the existing structure, only pre-configured storages like VFSStorage and ObjectStorage can be used, and users cannot specify their own vfolder as an import destination. To solve this, we need an adapter that wraps AbstractVolume into the AbstractStorage interface.
Required Features
- VolumeStorageAdapter class implementation
- Implement
AbstractStorageinterface (stream_upload,stream_download,delete_file,get_file_info) - Delegate to
AbstractVolume's native file operations (add_file,read_file,delete_files,mkdir) - VFolderID-based path resolution
- Implement
- StorageTarget class implementation
- Wrap storage name (str) or AbstractStorage instance
resolve()method: lookup from StoragePool or return instance directlynameproperty: return storage name
- Import API extension
- Add
vfidfield toHuggingFaceImportModelsReqandReservoirImportModelsReqDTOs - Add logic in API handlers to create
VolumeStorageAdapterwhenvfidis provided
- Add
- ImportStepContext update
- Change
storage_step_mappingstype todict[ArtifactStorageImportStep, StorageTarget]
- Change
Impact
| Component | Impact |
|---|---|
| `ai.backend.common.artifact_storage | Add StorageTarget clas |
| `ai.backend.common.dto.storage.request | Add vfid field to import DTO |
| `ai.backend.storage.storages.volume_adapter | New file: VolumeStorageAdapter clas |
| `ai.backend.storage.api.v1.registries.huggingface | Add vfid handling logi |
| `ai.backend.storage.api.v1.registries.reservoir | Add vfid handling logi |
| `ai.backend.storage.services.artifacts.* | Change to use `StorageTarget.resolve() |
Testing Scenarios
- VolumeStorageAdapter unit tests
stream_upload: Verify file exists in vfolder after uploadstream_download: Download uploaded file and verify content matchesdelete_file: Verify file no longer exists after deletionget_file_info: Verify accuracy of file metadata (size, type, modification time)- Verify nested directory path handling
- StorageTarget unit tests
StorageTarget(str): Verifyresolve()looks up from StoragePoolStorageTarget(AbstractStorage): Verifyresolve()returns instance directlynameproperty: Verify correct name returned for both string and instance
- Import API integration tests
- Request without
vfid: Verify existing behavior maintained (storage lookup from StoragePool) - Request with
vfid: Verify VolumeStorageAdapter created and files saved to vfolder - Verify correct operation across different volume backends (VFS, XFS, etc.)
- Request without
- Error cases
- Verify appropriate error returned when requesting with non-existent volume name
- Verify error handling for invalid vfid format
JIRA Issue: BA-3790
Metadata
Metadata
Assignees
Labels
No labels