Add hardlink support by ChengyuZhu6 · Pull Request #1954 · containerd/stargz-snapshotter

ChengyuZhu6 · 2025-01-24T02:30:41Z

Propose the implementation of a hardlink feature in the caching mechanism to optimize memory usage, improve performance and save disk space.

Fixes: #1953

cache/cache.go

docs/hardlink.md

AkihiroSuda · 2025-02-13T02:28:25Z

Needs rebase

ChengyuZhu6 · 2025-02-13T03:19:39Z

Needs rebase

Done.

ChengyuZhu6 · 2025-02-13T03:41:16Z

Just retrigger ci, no code change.

ChengyuZhu6 · 2025-04-01T02:29:58Z

@ktock I conducted experiments with several basic images, converting them to the estargz format and running them in containers with a simple 'echo "hello"' command. These tests used only background threads of stargz to pull images to the local machine. By measuring the overall memory and disk usage, I observed that implementing hardlinks resulted in a 20-30% reduction in both memory consumption and disk space requirements.

ChengyuZhu6 · 2025-04-01T06:31:37Z

When working with different versions of an image, implementing hardlinks can achieve a memory and disk deduplication effect of nearly 50%. I verified this by conducting tests with various development versions of Golang images.

ChengyuZhu6 · 2025-04-01T06:31:48Z

kindly ping @ktock

ChengyuZhu6 · 2025-06-24T06:24:14Z

cc @ktock

JulienBalestra · 2025-07-08T16:12:23Z

We're very interested by this PR, is there any plan for review @ktock ?

AkihiroSuda · 2025-07-09T05:56:48Z

Needs rebase

cache/cache.go

ktock · 2025-07-16T16:57:21Z

cache/cache.go

 }

+// ChunkDigest option allows specifying a chunk digest for the cache
+func ChunkDigest(digest string) Option {


why not use digest.Digest

cache/cache.go

fs/remote/blob.go

cmd/containerd-stargz-grpc/db/reader.go

fs/layer/layer.go

ChengyuZhu6 · 2025-07-28T06:03:18Z

@ktock

metadata/metadata.go

fs/layer/layer.go

cache/utils.go

cache/cache.go

cache/hardlink.go

ktock · 2025-07-28T07:38:33Z

CI failure in Optimize will be fixed in #2094.

ChengyuZhu6 · 2025-08-18T03:06:55Z

ping @ktock

hardlink/hardlink.go

fs/reader/reader.go

cache/cache.go

cmd/containerd-stargz-grpc/db/reader.go

metadata/memory/reader.go

Implement hardlink management for Stargz Snapshotter cache to reduce storage usage by deduplicating identical content chunks. Implementation: - Store canonical files in {root}/hardlinks/ directory - Use nlink-based garbage collection (remove when nlink == 1) - Track chunk digests with LRU cache for efficient lookups - Create hardlinks from canonical files to cache locations Configuration: - Add HardlinkRoot option for hardlink storage directory API: - Enroll(): Register canonical files by digest - Get(): Retrieve canonical file paths - Add(): Create hardlinks to target locations Signed-off-by: ChengyuZhu6 <hudson@cyzhu.com>

ChengyuZhu6 · 2025-12-03T09:16:56Z

ping @ktock

nibble0x2 · 2026-01-06T04:31:34Z

hardlink/hardlink.go

+	}
+
+	if sys, ok := stat.Sys().(*unix.Stat_t); ok {
+		if sys.Nlink == 1 {


Hi @ChengyuZhu6 ,

Really like the idea and see lots of benefits from this... have few Qs around the implementation -

when Nlink becomes 1? as far as I understand - stargz never deletes its temp dir cache so Nlink will never become 1? https://github.com/containerd/stargz-snapshotter/blob/main/fs/layer/layer.go#L231

while evicting at hardlink_manager side - what if cleanup never goes through i.e Nlink > 1 then these files will never get cleaned up as we are clearing LRU entries?

what's the purpose of DS digestToKeys and keyToDigest ? as I see #Get completely relies on chunkDigest to retrieve the link

does hardlink_manager recover the existing state during start up? let's say stargz was running fine and it had to restart - should it populate digestToFile during initialization for existing hardlinks?

Thanks!

ChengyuZhu6 force-pushed the hardlink branch 3 times, most recently from ff5ecf3 to 3e921b6 Compare January 24, 2025 02:44

ktock reviewed Feb 3, 2025

View reviewed changes

cache/cache.go Outdated Show resolved Hide resolved

cache/cache.go Outdated Show resolved Hide resolved

docs/hardlink.md Outdated Show resolved Hide resolved

ChengyuZhu6 force-pushed the hardlink branch 2 times, most recently from b879c95 to 457b45b Compare February 5, 2025 11:17

ChengyuZhu6 requested a review from ktock February 5, 2025 11:18

ChengyuZhu6 force-pushed the hardlink branch 2 times, most recently from 8829e66 to 92702af Compare February 10, 2025 06:20

ChengyuZhu6 force-pushed the hardlink branch from 92702af to 1d5fd08 Compare February 13, 2025 03:19

ChengyuZhu6 force-pushed the hardlink branch from 1d5fd08 to 8694c9a Compare February 13, 2025 03:41

ChengyuZhu6 force-pushed the hardlink branch from 8694c9a to a386406 Compare April 1, 2025 01:58

ChengyuZhu6 force-pushed the hardlink branch from a386406 to ef122fa Compare May 27, 2025 06:44

ChengyuZhu6 force-pushed the hardlink branch 3 times, most recently from efbcc75 to 01b70ae Compare June 4, 2025 09:55

ChengyuZhu6 force-pushed the hardlink branch 3 times, most recently from 1d0858b to 756bb86 Compare June 24, 2025 03:19

ChengyuZhu6 force-pushed the hardlink branch from 756bb86 to b3e9c8b Compare July 16, 2025 08:58

ktock requested changes Jul 16, 2025

View reviewed changes

ChengyuZhu6 force-pushed the hardlink branch 2 times, most recently from 9669477 to 349ad77 Compare July 28, 2025 03:21

ktock reviewed Jul 28, 2025

View reviewed changes

ChengyuZhu6 force-pushed the hardlink branch 5 times, most recently from 170ded5 to d62a0aa Compare August 15, 2025 08:12

ChengyuZhu6 requested a review from ktock August 15, 2025 08:12

ChengyuZhu6 force-pushed the hardlink branch 6 times, most recently from e85935b to f39d6a2 Compare August 15, 2025 09:23

ktock reviewed Aug 21, 2025

View reviewed changes

ChengyuZhu6 force-pushed the hardlink branch from f39d6a2 to b2a92e2 Compare October 14, 2025 10:57

ChengyuZhu6 force-pushed the hardlink branch 2 times, most recently from 3351d93 to e90577f Compare October 30, 2025 06:20

ChengyuZhu6 force-pushed the hardlink branch 2 times, most recently from 6d9d2bb to c3fa985 Compare November 18, 2025 11:38

ChengyuZhu6 requested a review from ktock November 18, 2025 11:39

ChengyuZhu6 force-pushed the hardlink branch from c3fa985 to e376da9 Compare November 18, 2025 11:51

ChengyuZhu6 force-pushed the hardlink branch from e376da9 to c0aac13 Compare November 18, 2025 12:01

nibble0x2 reviewed Jan 6, 2026

View reviewed changes

Conversation

ChengyuZhu6 commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AkihiroSuda commented Feb 13, 2025

Uh oh!

ChengyuZhu6 commented Feb 13, 2025

Uh oh!

ChengyuZhu6 commented Feb 13, 2025

Uh oh!

ChengyuZhu6 commented Apr 1, 2025

Uh oh!

ChengyuZhu6 commented Apr 1, 2025

Uh oh!

ChengyuZhu6 commented Apr 1, 2025

Uh oh!

ChengyuZhu6 commented Jun 24, 2025

Uh oh!

JulienBalestra commented Jul 8, 2025

Uh oh!

AkihiroSuda commented Jul 9, 2025

Uh oh!

Uh oh!

ktock Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChengyuZhu6 commented Jul 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ktock commented Jul 28, 2025

Uh oh!

ChengyuZhu6 commented Aug 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChengyuZhu6 commented Dec 3, 2025

Uh oh!

nibble0x2 Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ChengyuZhu6 commented Jan 24, 2025 •

edited

Loading