Using cache-image with private AWS ECR

[a1flecke]

Hello,

I am trying to use --cache-image with a private AWS ECR. I keep getting a 401 Unauthorized when running pack:

docker run \
    --platform linux/amd64 \
    --privileged \
    -u root \
    -v /var/run/docker.sock:/var/run/docker.sock  \
    -v $PWD:/workspace \
    -w /workspace buildpacksio/pack build test-pack \
    --builder paketobuildpacks/builder:tiny \
    --buildpack paketo-buildpacks/amazon-corretto \
    --buildpack paketo-buildpacks/java \
    --cache-image ***REDACTED_DOCKER_REGISTRY***/***REDACTED_DOCKER_REPOSITORY***:1 \
    --env BP_MAVEN_SETTINGS_PATH=/platform/bindings/settings.xml \
    --env ARTIFACTORY_USERNAME=$ARTIFACTORY_USERNAME \
    --env ARTIFACTORY_TOKEN=$ARTIFACTORY_TOKEN \
    --env BP_MAVEN_BUILD_ARGUMENTS="-Dmaven.repo.local=/platform/bindings/java/maven/binding/repository -Dmaven.test.skip=true --no-transfer-progress package --batch-mode" \
    --env BP_JVM_JLINK_ENABLED=true \
    --volume $HOME/.m2/settings.xml:/platform/bindings/settings.xml \
    --volume $HOME/.m2/repository:/platform/bindings/java/maven/binding/repository \
    --publish

Error:

failed to create image cache: accessing cache image "REDACTED*": connect to repo store "REDACTED": GET https://REDACTED_DOCKER_REGISTRY/v2/REDACTED_REPOISOTRY_NAME/manifests/1: unexpected status code 401 Unauthorized: Not Authorized

Outside of pack I confirmed that I can both pull and push images from the docker URI that I specified in the pack command. I have two questions:

1. Where is the code that takes the --cache-image docker uri value and translate it to the HTTP url used in the GET request specified in the error message? I looked and could not find it
2. Does pack use the underlying, already authenticated docker process to perform caching or does pack perform the calls directly to the registry?

Thank you for your help,

Aaron

[natalieparellano]

@a1flecke that is interesting - are you able to do a successful build without the use of --cache-image? Just trying to determine if pack is failing to resolve creds for the cache image specifically, or in general.

The place where pack constructs creds for images that the lifecycle needs to access is about here: https://github.com/buildpacks/pack/blob/52902b093e04f3c37479def536cfaa54cc24ce17/internal/build/lifecycle_execution.go#L581 - it uses authn.DefaultKeychain that reads from the docker config file

[a1flecke]

@natalieparellano Thanks for this info. It prompted me to dig deeper. I first simplified my setup by running pack directly on my system instead of in a devcontainer. This revealed that I had some docker configuration issues that were impacting pack (but somehow did not impact my day to day use of docker). Pack now works fine when running without the cache-image setting.

I am still running into some weirdness when using the cache-image setting. The following errors:

pack build test-pack \
    --builder paketobuildpacks/builder:tiny \
    --buildpack paketo-buildpacks/amazon-corretto \
    --buildpack paketo-buildpacks/java \
    --cache-image ***DOCKER REGISTRY***/***DOCKER REPOSITORY***/cache:1 \
    --env BP_MAVEN_SETTINGS_PATH=/platform/bindings/settings.xml \
    --env ARTIFACTORY_USERNAME=$ARTIFACTORY_USERNAME \
    --env ARTIFACTORY_TOKEN=$ARTIFACTORY_TOKEN \
    --env BP_MAVEN_BUILD_ARGUMENTS="-Dmaven.repo.local=/platform/bindings/java/maven/binding/repository -Dmaven.test.skip=true --no-transfer-progress package --batch-mode" \
    --env BP_JVM_JLINK_ENABLED=true \
    --volume $HOME/.m2/settings.xml:/platform/bindings/settings.xml \
    --volume $HOME/.m2/repository:/platform/bindings/java/maven/binding/repository \
    --publish

output:

tiny: Pulling from paketobuildpacks/builder
Digest: sha256:33f370b0ba68dc2e4c71b753e3eae9de61e21a14cbb57921a8eb59985943a3e8
Status: Image is up to date for paketobuildpacks/builder:tiny
===> ANALYZING
ERROR: failed to initialize analyzer: validating registry read access: ensure registry read access to test-pack
ERROR: failed to build: executing lifecycle: failed with status code: 1

If I change the command to:

        pack build ***DOCKER REGISTRY***/**DOCKER REPOSITORY***/test-pack \
            --builder paketobuildpacks/builder:tiny \
            --buildpack paketo-buildpacks/amazon-corretto \
            --buildpack paketo-buildpacks/java \
            --cache-image ***DOCKER REGISTRY***/***DOCKER REPOSITORY***/cache:1 \
            --env BP_MAVEN_SETTINGS_PATH=/platform/bindings/settings.xml \
            --env ARTIFACTORY_USERNAME=$ARTIFACTORY_USERNAME \
            --env ARTIFACTORY_TOKEN=$ARTIFACTORY_TOKEN \
            --env BP_MAVEN_BUILD_ARGUMENTS="-Dmaven.repo.local=/platform/bindings/java/maven/binding/repository -Dmaven.test.skip=true --no-transfer-progress package --batch-mode" \
            --env BP_JVM_JLINK_ENABLED=true \
            --volume $HOME/.m2/settings.xml:/platform/bindings/settings.xml \
            --volume $HOME/.m2/repository:/platform/bindings/java/maven/binding/repository \
            --publish

then pack successfully builds and uses the cache. Any idea why pack is running into issues with just the docker image being "locally tagged"?

[natalieparellano]

Ah - I think that makes sense. When you use --publish you are telling the lifecycle to export to an OCI registry. When you provide simply test-pack in that scenario then we infer the "default" registry which is index.docker.io. test-pack should work fine without --publish (but then you may have issues providing the --cache-image).