Azure function: sporadic and unpredictable failures during code deployment

[andrew-aladev]

Describe the bug

Hello. We are trying to use Azure functions in our project.

I created a pipeline with a script that creates a zip file (size is approximately 10 MB), creates an Azure function, and deploys a zip file into it.

We are running this script using Azure pipelines on an Azure machine. I mean, we are not using self-hosted solutions for this project. So I am sure that zip file is flying from Azure machine to Azure machine in the Azure internal network. Please don't suspect a network issue.

The version of the az tool is the latest possible because we are running it on the Azure DevOps machine as a part of the pipeline script.

We have received an issue in the simplest and most reliable part of the script:

echo 'Deploy application'
az functionapp deployment source config-zip \
  --name '$(FUNCTION_NAME)' \
  --resource-group '$(AZ_RG)' \
  --src "$APP_PATH"

The failing output is the following:

Deploy application
WARNING: Getting scm site credentials for zip deployment
WARNING: Starting zip deployment. This operation can take a while to complete ...
WARNING: Deployment endpoint responded with status code 202 for deployment id "***"
ERROR: Failed to connect to 'https://***.scm.azurewebsites.net/api/deployments/***/log' with status code '400' and reason 'Bad Request'

I made about 50 deployments and received about 3 failed deployments because of this error. So the failure rate for deploying Azure functions is about 6%, which is too high!

For now I made a deployment for development purposes only and asked our developers to rerun the pipeline when it fails because of an Azure issue. But tomorrow I will need to make a production deployment, and the customer will ask me a question: why does the pipeline fail?

Can you please provide me a good reason why the chance of successful deployment to Azure Function is so low? So I will resend it to the customer. Because I need to make things clear: this is not my fault; this is Azure's fault. Thank you.

Related command

echo 'Deploy application'
az functionapp deployment source config-zip \
  --name '$(FUNCTION_NAME)' \
  --resource-group '$(AZ_RG)' \
  --src "$APP_PATH"

Errors

Deploy application
WARNING: Getting scm site credentials for zip deployment
WARNING: Starting zip deployment. This operation can take a while to complete ...
WARNING: Deployment endpoint responded with status code 202 for deployment id "***"
ERROR: Failed to connect to 'https://***.scm.azurewebsites.net/api/deployments/***/log' with status code '400' and reason 'Bad Request'

Issue script & Debug output

Full script:

stages:
- stage: DeployFunction
  displayName: 'Deploy Function'
  jobs:
  - job: DeployFunction
    displayName: 'Deploy Function'
    steps:
    - task: AzureCLI@2
      displayName: 'Deploy Function'
      inputs:
        azureSubscription: '$(AZ_SUB)'
        scriptType: 'bash'
        scriptLocation: 'inlineScript'
        inlineScript: |
          set -euo pipefail

          ...

          echo 'Create application zip'
          APP_PATH='$(Agent.TempDirectory)/.application.zip'
          ...

          echo 'Create app insights'
          az monitor app-insights component create \
            --app '$(APP_INSIGHTS_NAME)' \
            --location '$(AZ_LOC)' \
            --resource-group '$(AZ_RG)' \
            --application-type 'web'

          if ! az functionapp show --name '$(FUNCTION_NAME)' --resource-group '$(AZ_RG)' >/dev/null 2>&1; then
            echo 'Create function'
            az functionapp create \
              --name '$(FUNCTION_NAME)' \
              --resource-group '$(AZ_RG)' \
              --storage-account '$(AZ_STO)' \
              --flexconsumption-location '$(AZ_LOC)' \
              --app-insights '$(APP_INSIGHTS_NAME)' \
              --functions-version '$(FUNCTION_VERSION)' \
              --runtime '$(RUNTIME_NAME)' \
              --runtime-version '$(RUNTIME_VERSION)' \
              --os-type 'Linux'
          fi

          echo 'Set application env'
          az functionapp config appsettings set \
            --name '$(FUNCTION_NAME)' \
            --resource-group '$(AZ_RG)' \
            --settings \
            ...

          echo 'Deploy application'
          az functionapp deployment source config-zip \
            --name '$(FUNCTION_NAME)' \
            --resource-group '$(AZ_RG)' \
            --src "$APP_PATH"

          echo 'Show application info'
          az functionapp function list \
            --name '$(FUNCTION_NAME)' \
            --resource-group '$(AZ_RG)' \
            -o 'table'

Expected behavior

The error may appear, but a failure rate of 6% is too high.

Environment Summary

azure-cli: latest version

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kamperiadis, @amamounelsayed, @khkh-ms.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @yairgil.