|
| 1 | +--- |
| 2 | +title: Troubleshoot prompt flow deployments |
| 3 | +titleSuffix: Azure Machine Learning |
| 4 | +description: This article provides instructions on how to troubleshoot your prompt flow deployments. |
| 5 | +manager: scottpolly |
| 6 | +ms.service: machine-learning |
| 7 | +ms.topic: how-to |
| 8 | +ms.date: 04/01/2024 |
| 9 | +ms.reviewer: lagayhar |
| 10 | +ms.author: keli19 |
| 11 | +author: likebupt |
| 12 | +--- |
| 13 | + |
| 14 | +# Troubleshoot prompt flow deployments |
| 15 | + |
| 16 | +This article provides instructions on how to troubleshoot your deployments from prompt flow. |
| 17 | + |
| 18 | +## Lack authorization to perform action "Microsoft.MachineLearningService/workspaces/datastores/read" |
| 19 | + |
| 20 | +If your flow contains Index Look Up tool, after deploying the flow, the endpoint needs to access workspace datastore to read MLIndex yaml file or FAISS folder containing chunks and embeddings. Hence, you need to manually grant the endpoint identity permission to do so. |
| 21 | + |
| 22 | +You can either grant the endpoint identity **AzureML Data Scientist** on workspace scope, or a custom role which contains "MachineLearningService/workspace/datastore/reader" action. |
| 23 | + |
| 24 | +## Upstream request timeout issue when consuming the endpoint |
| 25 | + |
| 26 | +If you use CLI or SDK to deploy the flow, you may encounter timeout error. By default the `request_timeout_ms` is 5000. You can specify at max to 5 minutes, which is 300000 ms. Following is example showing how to specify request time out in the deployment yaml file. Learn more about the deployment schema [here](../reference-yaml-deployment-managed-online.md). |
| 27 | + |
| 28 | +```yaml |
| 29 | +request_settings: |
| 30 | + request_timeout_ms: 300000 |
| 31 | +``` |
| 32 | +
|
| 33 | +## OpenAI API hits Authentication Error |
| 34 | +
|
| 35 | +If you regenerate your Azure OpenAI key and manually update the connection used in prompt flow, you may encounter errors like "Unauthorized. Access token is missing, invalid, audience is incorrect or have expired." when invoking an exissting endpoint created before key regenerating. |
| 36 | +
|
| 37 | +This is because the connections used in the endpoints/deployments will not be automatically updated. Any change for key or secrets in deployments should be done by manual update, which aims to avoid impacting online production deployment due to unintentional offline operation. |
| 38 | +
|
| 39 | +- If the endpoint was deployed in the studio UI, you can just redeploy the flow to the existing endpoint using the same deployment name. |
| 40 | +- If the endpoint was deployed using SDK or CLI, you need to make some modification to the deployment definition such as adding a dummy environment variable, and then use `az ml online-deployment update` to update your deployment. |
| 41 | + |
| 42 | + |
| 43 | +## Vulnerability issues in prompt flow deployments |
| 44 | + |
| 45 | +For prompt flow runtime related vulnerabilities, following are approaches which can help mitigate: |
| 46 | + |
| 47 | +- Update the dependency pacakages in your requirements.txt in your flow folder. |
| 48 | +- If you are using customized base image for your flow, you need to update the prompt flow runtime to latest version and rebuild your base image, then re-deploy the flow. |
| 49 | + |
| 50 | +For any other vulnerabilities of managed online deployments, Azure Machine Learning will fix the issues in a monthly manner. |
| 51 | + |
| 52 | +## "MissingDriverProgram Error" or "Could not find driver program in the request" |
| 53 | + |
| 54 | +If you deploy your flow encounter the following error, it might be related to the deployment environment. |
| 55 | + |
| 56 | +```text |
| 57 | +'error': |
| 58 | +{ |
| 59 | + 'code': 'BadRequest', |
| 60 | + 'message': 'The request is invalid.', |
| 61 | + 'details': |
| 62 | + {'code': 'MissingDriverProgram', |
| 63 | + 'message': 'Could not find driver program in the request.', |
| 64 | + 'details': [], |
| 65 | + 'additionalInfo': [] |
| 66 | + } |
| 67 | +} |
| 68 | +``` |
| 69 | + |
| 70 | +```text |
| 71 | +Could not find driver program in the request |
| 72 | +``` |
| 73 | + |
| 74 | +There are 2 ways to fix this error. |
| 75 | + |
| 76 | +1. (Recommended) You can find the container image uri in your custom environment detail page, and set it as the flow base image in the flow.dag.yaml file. When you deploy the flow in UI, you just select **Use environment of current flow definition**, and the backend service will create the customized environment based on this base image and `requirement.txt` for your deployment. Learn more about [the environment specified in the flow definition](#use-environment-of-current-flow-definition). |
| 77 | + |
| 78 | + :::image type="content" source="./media/how-to-deploy-for-real-time-inference/custom-environment-image-uri.png" alt-text="Screenshot of custom environment detail page. " lightbox = "./media/how-to-deploy-for-real-time-inference/custom-environment-image-uri.png"::: |
| 79 | + |
| 80 | + :::image type="content" source="./media/how-to-deploy-for-real-time-inference/flow-environment-image.png" alt-text="Screenshot of specifying base image in raw yaml file of the flow. " lightbox = "./media/how-to-deploy-for-real-time-inference/flow-environment-image.png"::: |
| 81 | + |
| 82 | +1. You can fix this error by adding `inference_config` in your custom environment definition. |
| 83 | + |
| 84 | + Following is an example of customized environment definition. |
| 85 | + |
| 86 | +```yaml |
| 87 | +$schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json |
| 88 | +name: pf-customized-test |
| 89 | +build: |
| 90 | + path: ./image_build |
| 91 | + dockerfile_path: Dockerfile |
| 92 | +description: promptflow customized runtime |
| 93 | +inference_config: |
| 94 | + liveness_route: |
| 95 | + port: 8080 |
| 96 | + path: /health |
| 97 | + readiness_route: |
| 98 | + port: 8080 |
| 99 | + path: /health |
| 100 | + scoring_route: |
| 101 | + port: 8080 |
| 102 | + path: /score |
| 103 | +``` |
| 104 | + |
| 105 | +## Model response taking too long |
| 106 | + |
| 107 | +Sometimes, you might notice that the deployment is taking too long to respond. There are several potential factors for this to occur. |
| 108 | + |
| 109 | +- Model is not powerful enough (ex. use gpt over text-ada) |
| 110 | +- Index query is not optimized and taking too long |
| 111 | +- Flow has many steps to process |
| 112 | + |
| 113 | +Consider optimizing the endpoint with above considerations to improve the performance of the model. |
| 114 | + |
| 115 | +## Unable to fetch deployment schema |
| 116 | + |
| 117 | +After you deploy the endpoint and want to test it in the **Test tab** in the endpoint detail page, if the **Test tab** shows **Unable to fetch deployment schema** like following, you can try the following 2 methods to mitigate this issue: |
| 118 | + |
| 119 | +:::image type="content" source="./media/how-to-deploy-for-real-time-inference/unable-to-fetch-deployment-schema.png" alt-text="Screenshot of the error unable to fetch deployment schema in Test tab in endpoint detail page. " lightbox = "./media/how-to-deploy-for-real-time-inference/unable-to-fetch-deployment-schema.png"::: |
| 120 | + |
| 121 | +- Make sure you have granted the correct permission to the endpoint identity. Learn more about [how to grant permission to the endpoint identity](#grant-permissions-to-the-endpoint). |
| 122 | +- It might be because you ran your flow in an old version runtime and then deployed the flow, the deployment used the environment of the runtime which was in old version as well. Update the runtime following [this guidance](./how-to-create-manage-runtime.md#update-a-runtime-on-the-ui) and rerun the flow in the latest runtime and then deploy the flow again. |
| 123 | + |
| 124 | +## Access denied to list workspace secret |
| 125 | + |
| 126 | +If you encounter an error like "Access denied to list workspace secret", check whether you have granted the correct permission to the endpoint identity. Learn more about [how to grant permission to the endpoint identity](#grant-permissions-to-the-endpoint). |
| 127 | + |
| 128 | +## Next steps |
| 129 | + |
| 130 | +- Learn more about [managed online endpoint schema](../reference-yaml-endpoint-online.md) and [managed online deployment schema](../reference-yaml-deployment-managed-online.md). |
| 131 | +- - Learn more about how to [troubleshoot managed online endpoints](../how-to-troubleshoot-online-endpoints.md). |
0 commit comments