Adopt COPY --link in Dockerfiles for improved build performance

[coderabbitai]

Summary

This issue tracks the adoption of COPY --link in all Dockerfiles across the repository to improve build performance and cache stability.

Background

The COPY --link flag (available in BuildKit with Dockerfile frontend v1.4+) creates independent layers for copied files, decoupling them from earlier commands in the build stage. This approach offers several advantages over standard COPY instructions.

Reference: Suggested in PR #1166, comment #1166 (comment) by @junhaoliao

Benefits

1. Stronger cache stability: COPY steps won't be invalidated when earlier commands in the same stage change, leading to more reliable cache hits
2. Faster parallel builds: BuildKit can execute COPY operations without waiting for earlier commands to complete
3. Reduced I/O and disk usage: BuildKit avoids duplicating file blobs when possible through hard-link-like behavior
4. Improved CI/CD performance: Works well with remote cache and enables easier image rebasing

Implementation Requirements

1. Add syntax header to all Dockerfiles

Each Dockerfile must start with:

# syntax=docker/dockerfile:1.4

2. Update COPY instructions

Replace COPY with COPY --link where appropriate:

COPY --link ./source ./destination

3. Dockerfiles to update

The following Dockerfiles should be evaluated and updated:

• tools/docker-images/clp-package/Dockerfile
• tools/docker-images/clp-execution-base-ubuntu-jammy/Dockerfile
• components/core/tools/docker-images/clp-core-ubuntu-jammy/Dockerfile
• components/core/tools/docker-images/clp-env-base-centos-stream-9/Dockerfile
• components/core/tools/docker-images/clp-env-base-manylinux_2_28-aarch64/Dockerfile
• components/core/tools/docker-images/clp-env-base-manylinux_2_28-x86_64/Dockerfile
• components/core/tools/docker-images/clp-env-base-musllinux_1_2-aarch64/Dockerfile
• components/core/tools/docker-images/clp-env-base-musllinux_1_2-x86_64/Dockerfile
• components/core/tools/docker-images/clp-env-base-ubuntu-jammy/Dockerfile

Important Caveats

1. Destination semantics: COPY --link creates layers starting from an empty filesystem, which can change behavior when copying to ambiguous destinations. Use explicit trailing slashes for directories.

2. Symlink resolution: Avoid copying into or relying on symlink resolution from earlier layers, as linked layers don't inherit the previous layer's filesystem state.

3. Testing required: Before production deployment:

   • Validate image behavior and file layouts
   • Test across different platforms (Linux, macOS, Windows)
   • Verify CI/CD pipelines work correctly
   • Check interactions with --chown flags

4. BuildKit requirement: Ensure CI/CD environments use BuildKit (enabled by default in Docker 23.0+, or set DOCKER_BUILDKIT=1).

Testing Plan

1. Update one Dockerfile as a proof-of-concept
2. Verify build times and cache behavior improvements
3. Test resulting images for correct runtime behavior
4. Roll out to remaining Dockerfiles incrementally
5. Update CI/CD workflows if needed to ensure BuildKit is enabled

References

• Docker BuildKit COPY --link documentation
• Docker blog: Image rebase and improved remote cache support
• How-To Geek: Docker COPY --link explanation

Success Criteria

• All Dockerfiles updated with syntax header and COPY --link
• Build times improved or maintained
• Cache hit rates improved
• All images validated for correct runtime behavior
• CI/CD pipelines functioning correctly