Pool fails - authorization to perform action 'Microsoft.Batch/batchAccounts/pools/write'

[BC89]

Problem Description

Running locally and the code to provisioin a pool and mount a fileshare works great. Once I deploy it to Azure container app serivce I get the following:

The client 'a15f98......' with object id 'a15f98.....' does not have authorization to perform action 'Microsoft.Batch/batchAccounts/pools/write' over scope '/subscriptions/235b2f.../resourceGroups/Br..../providers/Microsoft.Batch/batchAccounts/banx.../pools/626916.....' or the scope is invalid. If access was recently granted, please refresh your credentials.

C# .Net Management SDK. Works fine locally.

Any suggestions?

Steps to Reproduce

Expected Results

Pool would be provisioned, fileshare mounted. Works locally.

Actual Results

Additional Logs

INSERT ADDITIONAL LOGS HERE

Additonal Comments

[staer]

How are you authenticating locally when you interact with Batch API? Given the error from Azure Container App (permission error) it seems like you are probably authenticating with two different methods. Can you verify the permissions on the SP you are using with azure container app service (sorry i'm not very familiar with container app service).

[BC89]

So what I'm doing is explicitly creating a Pool and attaching a UserAssigned identity to it. I'm using DefaultAzureCredential to get a token and supply that to the batchmanagementclien. This works fine locally.

var credential = new DefaultAzureCredential(new DefaultAzureCredentialOptions
            {
                TenantId = environmentConfig.ManagedIdentityTenantId,
                ManagedIdentityClientId = environmentConfig.ManagedIdentityClientId,
                ExcludeAzureCliCredential = true,
                ExcludeAzurePowerShellCredential = true,
                ExcludeInteractiveBrowserCredential = true,
                ExcludeVisualStudioCodeCredential = true
            });

            logger.LogInformation($"Roboflat using credential {credential}");

            var token = await credential.GetTokenAsync(new TokenRequestContext(new string[] { TOKEN_ENDPOINT }));
            logger.LogInformation($"Roboflat token {token.ExpiresOn}");

            using var batchManagementClient = new Microsoft.Azure.Management.Batch.BatchManagementClient(new TokenCredentials(token.Token));

            batchManagementClient.SubscriptionId = environmentConfig.SubscriptionId;

then later...

var poolParameters = new Microsoft.Azure.Management.Batch.Models.Pool()
            {
                TaskSchedulingPolicy = new Microsoft.Azure.Management.Batch.Models.TaskSchedulingPolicy(Microsoft.Azure.Management.Batch.Models.ComputeNodeFillType.Spread),
                TaskSlotsPerNode = modelRuntimeSpec.TaskSlotsPerNode,

                VmSize = modelRuntimeSpec.VmSize,

                ScaleSettings = new Microsoft.Azure.Management.Batch.Models.ScaleSettings
                {
                    FixedScale = new Microsoft.Azure.Management.Batch.Models.FixedScaleSettings
                    {
                        TargetDedicatedNodes = modelRuntimeSpec.TargetDedicatedNodes,
                        TargetLowPriorityNodes = modelRuntimeSpec.TargetLowPriorityNodes
                    },
                },

                DeploymentConfiguration = new Microsoft.Azure.Management.Batch.Models.DeploymentConfiguration
                {
                    VirtualMachineConfiguration = virtualMachineConfiguration
                },

                //NOT SURE WHY BUT THIS DOES NOT WORK IN ACA --- HAD TO ADD CONTRIBUTOR
                Identity = new Microsoft.Azure.Management.Batch.Models.BatchPoolIdentity
                {
                    Type = Microsoft.Azure.Management.Batch.Models.PoolIdentityType.UserAssigned,
                    UserAssignedIdentities = new Dictionary<string, Microsoft.Azure.Management.Batch.Models.UserAssignedIdentities>
                    {
                        [$"/subscriptions/{environmentConfig.SubscriptionId}/resourcegroups/{environmentConfig.ResourceGroup}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{environmentConfig.ManagedIdentityName}"]
                            = new Microsoft.Azure.Management.Batch.Models.UserAssignedIdentities()
                    }
                },

                MountConfiguration = new[] { new Microsoft.Azure.Management.Batch.Models.MountConfiguration(azureFileShareConfiguration: azurefileshareconfiguration) }
            };

The odd thing is if I assign the UserManaged identity CONTRIBUTOR role in the batch service it will at least start creating the pool but then fails on provisioning the nodes with:

Code: NodePreparationError

Message:
An error occurred during node preparation

Error - Hit unexpected error installing containers
Message - 400, message='Bad Request', url=URL('http://169.2..../metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/&mi_res_id=/subscriptions/235b....resourcegroups/Bryx_Test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/minx....') 

Thank you!

[BC89]

So As an update and for anyone else struggling with this issue, I was able to resolve this by creating a custom role in my resource group which inherits from the existing azure batch role. This seems like something MS could easily do so others wouldn't have to muddle through it. Side-note, I had to also add 'Assign User Assigned Identity' to the custom role. Once I had the role setup I granted my existing UserIdentity the role and everything worked. I'll probably work through removing some of the inherited Batch permissions but for now it's working.