Skip to content

image/internal: sha512 support for skopeo copy #475

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+146 −20
Open

image/internal: sha512 support for skopeo copy #475

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7f1e5e4
image/directory: store blobs with digest algorithm prefix
lsm5 Nov 25, 2025
d1587a4
image/directory: assign version based on digest algorithm
lsm5 Nov 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 31 additions & 9 deletions image/directory/directory_dest.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,20 +20,30 @@ import (
"go.podman.io/storage/pkg/fileutils"
)

const version = "Directory Transport Version: 1.1\n"
// Write version file based on digest algorithm used
// 1.1 for sha256-only images, 1.2 otherwise.
const (
versionPrefix = "Directory Transport Version: "
version = versionPrefix + "1.2\n"
version1_1 = versionPrefix + "1.1\n"
)

// ErrNotContainerImageDir indicates that the directory doesn't match the expected contents of a directory created
// using the 'dir' transport
var ErrNotContainerImageDir = errors.New("not a containers image directory, don't want to overwrite important data")

// ErrUnsupportedVersion indicates that the directory uses a version newer than we support
var ErrUnsupportedVersion = errors.New("unsupported directory transport version")

type dirImageDestination struct {
impl.Compat
impl.PropertyMethodsInitialize
stubs.IgnoresOriginalOCIConfig
stubs.NoPutBlobPartialInitialize
stubs.AlwaysSupportsSignatures

ref dirReference
ref dirReference
usesNonSHA256Digest bool
}

// newImageDestination returns an ImageDestination for writing to a directory.
Expand Down Expand Up @@ -76,7 +86,11 @@ func newImageDestination(sys *types.SystemContext, ref dirReference) (private.Im
return nil, err
}
// check if contents of version file is what we expect it to be
if string(contents) != version {
versionStr := string(contents)
if versionStr != version && versionStr != version1_1 {
if versionStr > version {
return nil, fmt.Errorf("%w: %q", ErrUnsupportedVersion, versionStr)
}
return nil, ErrNotContainerImageDir
}
} else {
Expand All @@ -94,11 +108,6 @@ func newImageDestination(sys *types.SystemContext, ref dirReference) (private.Im
return nil, fmt.Errorf("unable to create directory %q: %w", ref.resolvedPath, err)
}
}
// create version file
err = os.WriteFile(ref.versionPath(), []byte(version), 0o644)
if err != nil {
return nil, fmt.Errorf("creating version file %q: %w", ref.versionPath(), err)
}

d := &dirImageDestination{
PropertyMethodsInitialize: impl.PropertyMethods(impl.Properties{
Expand Down Expand Up @@ -151,7 +160,8 @@ func (d *dirImageDestination) PutBlobWithOptions(ctx context.Context, stream io.
}
}()

digester, stream := putblobdigest.DigestIfCanonicalUnknown(stream, inputInfo)
digester, stream := putblobdigest.DigestIfUnknown(stream, inputInfo)
Copy link
Contributor

@mtrmac mtrmac Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the destination-transport-side digest computation must be a more complex logic, see in the earlier PR about the interaction with cannotModifyManifestReason.

… and dirReference.layerPath discards the algorithm name; that does not generalize for other algorithms, we need to move towards agility where adding an extra algorithm is a ~parameter change and does not require any more changes to the “code proper”; i.e. discarding algorithm names is no longer much of an option.

(We need to keep the existing file names for sha256, to retain compatibility. And… do we define a new value for versionPath?!)

Copy link
Member Author

@lsm5 lsm5 Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated PutBlob to store blob under provided digest algorithm with the algorithm name prepended (except for Canonical) along with a canonical digest hardlink.

$ /usr/bin/ls
dc518581817f4e75a7dcfd35383e67c3ef85438250c17e10090b5a31ab8f68d4  manifest.json  sha512-2ee373e378345b35e7966a106c5c0a40a005a13bfc87695d89c5bb217f969c351e73810cf78d3d841237098731cf76878f05af8f8d28176c316681f9422ff688  version

$ diff dc518581817f4e75a7dcfd35383e67c3ef85438250c17e10090b5a31ab8f68d4 sha512-2ee373e378345b35e7966a106c5c0a40a005a13bfc87695d89c5bb217f969c351e73810cf78d3d841237098731cf76878f05af8f8d28176c316681f9422ff688
$

do we define a new value for versionPath?!)

Doesn't break existing behaviour but there's new stuff, so maybe we should? I'll defer to you.

Copy link
Contributor

@mtrmac mtrmac Nov 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Hard links are not supported by all file systems (FAT)
  • Symbolic links are not supported by all file systems either, and generally restricted to admin users on Windows

And, anyway, readers of dir: can only start with the manifest, and the values provided in the manifest. So if PutBlob returns sha512, and the manifest is written to include sha512, readers will not know the sha256 value and have no way to use it.

So I don’t think we need to compute both digests at all; just the layerPath changes to the path computation, + some (as-yet-undefined) logic for PutBlob to use “the algorithm the user wanted”, should be sufficient.

do we define a new value for versionPath?!)

I think with the changes to layerPath, we need to. Previously it was, hypothetically, possible to read a complete sha512 image from dir:, and those images will now break. And we will need to update both dir…Dest… and dir…Src…: destinations should refuse to work on future versions, and still assign the existing 1.1 version for sha256 images for maximum compatibility. sources should detect+refuse future versions.

Consider making the dir changes a separate PR.

Copy link
Member Author

@lsm5 lsm5 Nov 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll keep the dir changes here because of the review comments. Separate PR for image/internal at #486


// TODO: This can take quite some time, and should ideally be cancellable using ctx.Done().
size, err := io.Copy(blobFile, stream)
if err != nil {
Expand All @@ -165,6 +175,10 @@ func (d *dirImageDestination) PutBlobWithOptions(ctx context.Context, stream io.
return private.UploadedBlob{}, err
}

if blobDigest.Algorithm() != digest.Canonical {
d.usesNonSHA256Digest = true
}

// On POSIX systems, blobFile was created with mode 0600, so we need to make it readable.
// On Windows, the “permissions of newly created files” argument to syscall.Open is
// ignored and the file is already readable; besides, blobFile.Chmod, i.e. syscall.Fchmod,
Expand Down Expand Up @@ -257,6 +271,14 @@ func (d *dirImageDestination) PutSignaturesWithFormat(ctx context.Context, signa
// - Uploaded data MAY be visible to others before CommitWithOptions() is called
// - Uploaded data MAY be removed or MAY remain around if Close() is called without CommitWithOptions() (i.e. rollback is allowed but not guaranteed)
func (d *dirImageDestination) CommitWithOptions(ctx context.Context, options private.CommitOptions) error {
versionToWrite := version1_1
if d.usesNonSHA256Digest {
versionToWrite = version
}
err := os.WriteFile(d.ref.versionPath(), []byte(versionToWrite), 0o644)
if err != nil {
return fmt.Errorf("writing version file %q: %w", d.ref.versionPath(), err)
}
return nil
}

Expand Down
21 changes: 19 additions & 2 deletions image/directory/directory_src.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,24 @@ type dirImageSource struct {

// newImageSource returns an ImageSource reading from an existing directory.
// The caller must call .Close() on the returned ImageSource.
func newImageSource(ref dirReference) private.ImageSource {
func newImageSource(ref dirReference) (private.ImageSource, error) {
versionPath := ref.versionPath()
contents, err := os.ReadFile(versionPath)
if err != nil {
if !os.IsNotExist(err) {
return nil, fmt.Errorf("reading version file %q: %w", versionPath, err)
}
} else {
versionStr := string(contents)
if versionStr != version && versionStr != version1_1 {
// Check if it's a future version we don't support
if versionStr > version {
return nil, fmt.Errorf("%w: %q", ErrUnsupportedVersion, versionStr)
}
return nil, fmt.Errorf("invalid version file content: %q", versionStr)
}
}

s := &dirImageSource{
PropertyMethodsInitialize: impl.PropertyMethods(impl.Properties{
HasThreadSafeGetBlob: false,
Expand All @@ -36,7 +53,7 @@ func newImageSource(ref dirReference) private.ImageSource {
ref: ref,
}
s.Compat = impl.AddCompat(s)
return s
return s, nil
}

// Reference returns the reference used to set up this source, _as specified by the user_
Expand Down
71 changes: 71 additions & 0 deletions image/directory/directory_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"errors"
"io"
"os"
"path/filepath"
"testing"

"github.com/opencontainers/go-digest"
Expand Down Expand Up @@ -201,8 +202,78 @@ func TestGetPutSignatures(t *testing.T) {
assert.Equal(t, signatures, sigs)
}

func TestVersionAssignment(t *testing.T) {
t.Run("SHA256 gets version 1.1", func(t *testing.T) {
ref, tmpDir := refToTempDir(t)
cache := memory.New()

dest, err := ref.NewImageDestination(context.Background(), nil)
require.NoError(t, err)
defer dest.Close()

blob := []byte("test-blob-sha256")
_, err = dest.PutBlob(context.Background(), bytes.NewReader(blob), types.BlobInfo{Digest: "", Size: int64(len(blob))}, cache, false)
require.NoError(t, err)

err = dest.Commit(context.Background(), nil)
require.NoError(t, err)

versionBytes, err := os.ReadFile(filepath.Join(tmpDir, "version"))
require.NoError(t, err)
assert.Equal(t, "Directory Transport Version: 1.1\n", string(versionBytes))
})

t.Run("Non-SHA256 gets version 1.2", func(t *testing.T) {
ref, tmpDir := refToTempDir(t)
cache := memory.New()

dest, err := ref.NewImageDestination(context.Background(), nil)
require.NoError(t, err)
defer dest.Close()

blob := []byte("test-blob-sha512")
sha512Digest := digest.SHA512.FromBytes(blob)
_, err = dest.PutBlob(context.Background(), bytes.NewReader(blob), types.BlobInfo{Digest: sha512Digest, Size: int64(len(blob))}, cache, false)
require.NoError(t, err)

err = dest.Commit(context.Background(), nil)
require.NoError(t, err)

versionBytes, err := os.ReadFile(filepath.Join(tmpDir, "version"))
require.NoError(t, err)
assert.Equal(t, "Directory Transport Version: 1.2\n", string(versionBytes))
})

t.Run("Mixed digests get version 1.2", func(t *testing.T) {
ref, tmpDir := refToTempDir(t)
cache := memory.New()

dest, err := ref.NewImageDestination(context.Background(), nil)
require.NoError(t, err)
defer dest.Close()

blob1 := []byte("test-blob-sha256")
_, err = dest.PutBlob(context.Background(), bytes.NewReader(blob1), types.BlobInfo{Digest: "", Size: int64(len(blob1))}, cache, false)
require.NoError(t, err)

blob2 := []byte("test-blob-sha512")
sha512Digest := digest.SHA512.FromBytes(blob2)
_, err = dest.PutBlob(context.Background(), bytes.NewReader(blob2), types.BlobInfo{Digest: sha512Digest, Size: int64(len(blob2))}, cache, false)
require.NoError(t, err)

err = dest.Commit(context.Background(), nil)
require.NoError(t, err)

versionBytes, err := os.ReadFile(filepath.Join(tmpDir, "version"))
require.NoError(t, err)
assert.Equal(t, "Directory Transport Version: 1.2\n", string(versionBytes))
})
}

func TestSourceReference(t *testing.T) {
ref, tmpDir := refToTempDir(t)
err := os.WriteFile(filepath.Join(tmpDir, "version"), []byte("Directory Transport Version: 1.1\n"), 0o644)
require.NoError(t, err)

src, err := ref.NewImageSource(context.Background(), nil)
require.NoError(t, err)
Expand Down
17 changes: 12 additions & 5 deletions image/directory/directory_transport.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ func (ref dirReference) NewImage(ctx context.Context, sys *types.SystemContext)
// NewImageSource returns a types.ImageSource for this reference.
// The caller must call .Close() on the returned ImageSource.
func (ref dirReference) NewImageSource(ctx context.Context, sys *types.SystemContext) (types.ImageSource, error) {
return newImageSource(ref), nil
return newImageSource(ref)
}

// NewImageDestination returns a types.ImageDestination for this reference.
Expand All @@ -172,12 +172,19 @@ func (ref dirReference) manifestPath(instanceDigest *digest.Digest) (string, err
}

// layerPath returns a path for a layer tarball within a directory using our conventions.
func (ref dirReference) layerPath(digest digest.Digest) (string, error) {
if err := digest.Validate(); err != nil { // digest.Digest.Encoded() panics on failure, and could possibly result in a path with ../, so validate explicitly.
func (ref dirReference) layerPath(d digest.Digest) (string, error) {
if err := d.Validate(); err != nil { // digest.Digest.Encoded() panics on failure, and could possibly result in a path with ../, so validate explicitly.
return "", err
}
// FIXME: Should we keep the digest identification?
return filepath.Join(ref.path, digest.Encoded()), nil

var filename string
if d.Algorithm() == digest.Canonical {
filename = d.Encoded()
} else {
filename = d.Algorithm().String() + "-" + d.Encoded()
Copy link
Contributor

@mtrmac mtrmac Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder whether we need to do a version check and conditionally use the old logic here. I guess that doesn’t matter in practice.

}

return filepath.Join(ref.path, filename), nil
}

// signaturePath returns a path for a signature within a directory using our conventions.
Expand Down
17 changes: 13 additions & 4 deletions image/directory/directory_transport_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,9 @@ func TestReferenceNewImageNoValidManifest(t *testing.T) {
}

func TestReferenceNewImageSource(t *testing.T) {
ref, _ := refToTempDir(t)
ref, tmpDir := refToTempDir(t)
err := os.WriteFile(filepath.Join(tmpDir, "version"), []byte("Directory Transport Version: 1.1\n"), 0o644)
require.NoError(t, err)
src, err := ref.NewImageSource(context.Background(), nil)
assert.NoError(t, err)
defer src.Close()
Expand Down Expand Up @@ -209,14 +211,21 @@ func TestReferenceManifestPath(t *testing.T) {
}

func TestReferenceLayerPath(t *testing.T) {
const hex = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
const hex256 = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
const hex512 = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"

ref, tmpDir := refToTempDir(t)
dirRef, ok := ref.(dirReference)
require.True(t, ok)
res, err := dirRef.layerPath("sha256:" + hex)

res, err := dirRef.layerPath("sha256:" + hex256)
require.NoError(t, err)
assert.Equal(t, tmpDir+"/"+hex, res)
assert.Equal(t, tmpDir+"/"+hex256, res)

res, err = dirRef.layerPath("sha512:" + hex512)
require.NoError(t, err)
assert.Equal(t, tmpDir+"/sha512-"+hex512, res)

_, err = dirRef.layerPath(digest.Digest("sha256:../hello"))
assert.Error(t, err)
}
Expand Down
Loading