ACR Containers Not Updating

[LanceMcCarthy]

Note

This is a branch of this initial discussion where the solution is to define a custom base image. This new discussion is to focus on azd deploy problem where this new image is not being used in Azure.

When an update to a .NET Aspire service container is made, Azure Container Registry (ACR) is not picking up the changes when deploying. Here's an example lifecycle to explain:

Deploy 1

• apiservice (v1)
• webfrontend (v1)

Now, let's say I changed the base container for apiservice, and rebuilt and redeploy to Azure

Deploy 2

• apiservice (v2)
• webfrontend (v1)

Problem

When visiting the application, it is still using apiservice v1 image, verified by exec into the running container and confirming the changes for v2 are not there.

To eliminate if this was a cache optimization side effect (not re-pulling image with the same tag), I started deploying images with explicit versioning in the tag (apiservice:1, apiservice:2, etc) to force the relationship and confirm the correct image should always be there.

Unfortunately, explicit version tags did not resolve the problem.

Workaround

I was able to workaround this by manually editing the container for the API service in Azure and choosing the GitHub registry version, instead of the ACR one.

Here's a screenshot of where I edit the running container and chose my GitHub image instead of the ACR one. This effectively bypasses the part of the lifecycle where I suspect the problem lies (the latest image is not being pulled by ACR).

Once the container's image was changed to the GitHub copy, it was working as expected:

Moving Forward

Unfortunately, this is short-lived solution, because on the very next azd deploy, the incorrect image from ACR is used again. After every deploy, I need to go into Azure and edit the apiservice's container to use the GitHub copy of the image.

So, you can see why there is suspicion that ACR is not pulling the latest image, but I do not have any evidence other than the above described behavior of the image in ACR is always the old version.

[davidfowl]

cc @ellismg @vhvb1989

[vhvb1989]

@LanceMcCarthy , when you say:

let's say I changed the base container for apiservice

what do you mean? Is it just making changes to a project? (project.v0) or do you mean a container.v0 and you are updating the img for it?

[LanceMcCarthy]

Hi @vhvb1989 Thanks for the prompt follow up!

In this situation, there are no .NET code changes. Only the ContainerBaseImage for the build is different.

To avoid making this more complex than it needs to be, let me show the real app that is experiencing the prblem right now

Quick Backstory

I build an Aspire app and everything is working great locally, so I publish it to Azure. Unfortunately, when in Azure, I realize it is not working because mcr.microsoft.com/dotnet/aspnet:8.0 is missing a critical dependency.

Fix

Not a problem, after this discussion with Tim Heuer, Rich Lander and Chet, I have a solution to that problem. I simply use this Dockerfile to build a new base image that has the dependencies I need. that base image is published as: lancemccarthy/aspirebase:8.0

Okay, now I have a new base image to build the apiservice project with, and I use -p ContainerBaseImage="ghcr.io/lancemccarthy/aspirebase:8.0".

Here's the complete command:

dotnet publish 'ApiService.csproj' /t:PublishContainer \

           # HERE IS THE UPDATED BASE IMAGE BEING USED
  -p ContainerBaseImage="ghcr.io/lancemccarthy/aspirebase:8.0" \

         # the other stuff for the apiservice container
  -p ContainerRepository="lancemccarthy/aspireapiservice:8.0" \
  -p ContainerImageTag="8.0" \
  -p ContainerRegistry="ghcr.io"

If you want to see the actual build and publish, here's the couple workflow steps

Success!

That build produces the apiService project's image: lancemccarthy/aspireapiservice:8.0 with the new base image. This is working beautifully and was the final solution to my problems.

Azure Problem

However, now comes the time when I use azd deploy, when I publish the updated version of apiservice it is still behaving like it did after that first deploy with only the Microsoft base image.

It is as if all future deploys are not pushing/pulling the updated version of apiservice that has the new base image.

Does that help? The full repo used is here https://github.com/LanceMcCarthy/DevCraftAspire

[vhvb1989]

by design, azd does not support passing -p ContainerBaseImage when running dotnet publish.
You need to set the base image into the project, like: https://learn.microsoft.com/dotnet/core/docker/publish-as-container?pivots=dotnet-8-0#containerbaseimage

Then when azd runs dotnet publish, dotnet will use that base image you set :)

[LanceMcCarthy]

aha!

The funny thing is, I did originally have that set in the csproj... but got an error about the image and took it out. I never added it back. I'll do another build now.

-removed question, there's no longer a need for any of the other container info now that azd is doing all the work -

P.S. If you have any control over azd's future, I would petition for the ability to pass some params to it that allow me to pick which projects not to build again ;)

[LanceMcCarthy]

Okay, @vhvb1989 I do get the same error as before, it seems that azd doesn't know how to use a public repo for the ContainerBaseImage directly:

I think this was the whole reason I went with dotnet build in the first place.

[EDIT - UPDATE]

To avoid this extra headache, I just grabbed the credentials of the underlying ACR and pushed the base image to ACR instead ;)

[timheuer]

@LanceMcCarthy the ghcr.io are actually NOT public (open to all) as you still need to auth on those. So you likely in your CI could add an additional docker/login to the ghcr.io and the pull would succeed (wanna try? :-)

RE: don't rebuild certain projects...I assume you're in the mindset of 'nothing's changed, so no need for a rebuild/revision'? If so, this is kinda being discussed how/if we should approach this.

[LanceMcCarthy]

@timheuer Indeed, I did realize the access issue with some ghcr.io packages, so I was careful to make sure it was fully public.

• To verify this, I opened the following link in a private browser tab https://github.com/LanceMcCarthy/DevCraftAspire/pkgs/container/aspirebase
• Just because the web page is public, doesn't mean docker can pull it. So I triple checked from an anonymous shell and ran docker pull ghcr.io/lancemccarthy/aspirebase:8.0

So, knowing that it is accessible to an anonymous shell, should you able to repro my error by using <ContainerBaseImage>ghcr.io/lancemccarthy/aspirebase:8.0</ContainerBaseImage> ?

So I think maybe, because azd is authenticated with azurecr.io when it does the build, it could be interfering with the pull? It has to deal with two registries at the same time.

azurecr

All that said, I did just switch everything over to trying it from azurecr.io (the build is happening right now => https://github.com/LanceMcCarthy/DevCraftAspire/actions/runs/9214612357/job/25351162061