Azure Blockchain Workbench container registry throwing Unauthorized

[vermaaditya]

Hi Team,

We have deployed a brand new version of the Azure Blockchain workbench on our Azure Subscription.
On the day of provisioning the services, everything was working fine and the API as well as the Workbench dashboard was working fine.
But after 3-4 days, when visiting back the workbench, the below error is what we encounter -

when looking into the docker logs, below is what I see -

2020-09-15T00:02:39.011Z ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository) 2020-09-15T00:02:39.024Z INFO - Stopping site pocblkchainsept-2gtmxa-api because it failed during startup. 2020-09-15T00:07:38.419Z INFO - Pulling image from Docker hub: blockchainworkbenchprod.azurecr.io/appbuilder.api:1.8.1 2020-09-15T00:07:38.990Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://blockchainworkbenchprod.azurecr.io/v2/appbuilder.api/manifests/1.8.1: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information."}

The similar issue was reported in the link - https://social.msdn.microsoft.com/Forums/en-US/5233adb9-efe8-4f23-94d1-1b6706007beb/azure-blockchain-workbench-container-registry?forum=opensourcedevwithazure but there are no resolutions / workarounds.

Kindly provide inputs.

[vermaaditya]

adding @venbhavip to the loop from my team.

[vermaaditya]

@mmercuri - will it be possible for you to pitch in here?

[caleteeter]

Thanks @vermaaditya , I am looking into this now.

[caleteeter]

@vermaaditya this is working currently, just did a fresh deployment. Is your instance still broken? Maybe restart your VM scale set?

[vermaaditya]

@caleteeter - the old instance for which i have provided the screenshot and logs is deleted as it was unusable, and was being billed against our subscription.
We have created a fresh instance today some 6 hours back.

[caleteeter]

Think we need to investigate the ACR that holds these images to see what was up there, intermittently not available. Keep this issue open for feedback and resolution for this.

[vermaaditya]

Screenshot of the container settings tab on Web as well as API -

Dont know how the app is working right now. Is it something to do with caching ?
Please let me know if we can have a quick call to understand this ?

[caleteeter]

I don't believe this is an caching issue, but rather the fact that an image was attempted to be pulled that should exist but didn't at the time of this screenshot. Following up internal here.

[sudobenny]

Has there been any solution on this issue? I'm getting the same " :( Application Error" after 2 days of the blockchain workbench app working. I'm not sure what to do except delete the resource group and redeploy Blockchain Workbench and try again. Thanks
ERROR:
2020-11-04T01:18:35.088Z ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-11-04T01:18:35.099Z INFO - Stopping site infloblockchain-klr6up because it failed during startup.
2020-11-04T01:19:06.018Z INFO - Pulling image from Docker hub: blockchainworkbenchprod.azurecr.io/webapp:1.8.1
2020-11-04T01:19:06.224Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://blockchainworkbenchprod.azurecr.io/v2/webapp/manifests/1.8.1: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information."}
https://docs.microsoft.com/en-ca/azure/container-registry/container-registry-auth-service-principal

[vermaaditya]

@sudobenny - check comment from @caleteeter on Sept 16th. The actual resolution isnt provided as such. For me it stopped reproducing after it was reported here.