Add recursive directory support #1871
Description
What is the version of your ORAS CLI
No response
What would you like to be added?
Current Behavior
When using oras push, files are uploaded individually as blobs while directories are bundled into tarballs that are then uploaded as blobs. These blobs are wrapped in an Image Manifest.
The current approach makes it impossible to list the contents for a directory tarball or download a single file from the directory tarball without downloading the tarball in full.
Proposed Solution
Implement hierarchical directory representation using OCI Image Indexes in addition to OCI Image Manifests. To preserve backwards compatibility, this new behavior must be explicitly enabled via parameters/config.
Directory Structure Mapping
- File-only directories: Represented as a single Image Manifest
- Directory-only directories: Represented as an Image Index linking to other Image Indexes or Image Manifests
- Mixed directories: Similar to "directory-only" but with an additional Image Manifest representing the directory files.
Example filesystem:
sample/
├── dirs-only
│ ├── assets
│ │ └── logo.png
│ └── config
│ └── settings.json
├── files-only
│ ├── config.txt
│ ├── data.json
│ └── readme.md
└── mixed
├── package.json
├── README.md
├── src
│ └── main.js
└── tests
└── test.js
Example OCI references:
OCI Image Indexes OCI Image Manifests OCI Blobs
+---------+ +-----------+ +--------+ +----------+
| sample +---+---->| dirs-only +--------+-------->| assets +-------------------->| logo.png |
+---------+ | +-----------+ | +--------+ +----------+
| |
| |
| | +--------+ +---------------+
| +-------->| config +------------------>| settings.json |
| +--------+ +---------------+
|
|
| +------------+ +------------+
+--------------------------------->| files-only +--------+-------->| config.txt |
| +------------+ | +------------+
| |
| |
| | +-----------+
| +--------->| data.json |
| | +-----------+
| |
| |
| | +-----------+
| +--------->| readme.md |
| +-----------+
|
|
| +-------+ +---+ +--------------+
+----->| mixed +-----------+----------->| . +------------+-------->| package.json |
+-------+ | +---+ | +--------------+
| |
| |
| |
| | +-----------+
| +---------->| README.md |
| +-----------+
|
|
| +-----+ +---------+
+---------->| src +----------------------->| main.js |
| +-----+ +---------+
|
|
| +--------+ +-----------+
+--------->| tests +-------------------->| test.json |
+--------+ +-----------+
Why is this needed for ORAS?
Motivation
This enhancement addresses specific requirements for representing YUM repositories as OCI Artifacts. YUM repositories containing hundreds of thousands of files present scalability challenges. For example, I observed that certain registries, Quay, can easily run into timeouts when processing Image Manifests with about 2,000 blobs due to integrity checks implemented by the registry.
Organizing files into subdirectories, some form of "sharding", maintains reasonable blob counts per Image Manifest.
See discussion for additional context.
This could be achieved today with wrapper scripts around oras, but I do think that coming to an agreement on how to represent this data in an OCI registry with the oras community would be valuable for anyone using OCI artifacts.
Are you willing to submit PRs to contribute to this feature?
- Yes, I am willing to implement it.