You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Learn how to troubleshoot and solve, or work around, common errors you may come across when using [batch endpoints](how-to-use-batch-endpoint.md) for batch scoring. In this article you'll learn:
20
+
Learn how to troubleshoot and solvecommon errors you may come across when using [batch endpoints](how-to-use-batch-endpoint.md) for batch scoring. In this article you learn:
21
21
22
22
> [!div class="checklist"]
23
23
> * How [logs of a batch scoring job are organized](#understanding-logs-of-a-batch-scoring-job).
@@ -32,7 +32,7 @@ After you invoke a batch endpoint using the Azure CLI or REST, the batch scoring
32
32
33
33
Option 1: Stream logs to local console
34
34
35
-
You can run the following command to stream system-generated logs to your console. Only logs in the `azureml-logs` folder will be streamed.
35
+
You can run the following command to stream system-generated logs to your console. Only logs in the `azureml-logs` folder are streamed.
36
36
37
37
```azurecli
38
38
az ml job stream --name <job_name>
@@ -43,13 +43,13 @@ Option 2: View logs in studio
43
43
To get the link to the run in studio, run:
44
44
45
45
```azurecli
46
-
az ml job show --name <job_name> --query interaction_endpoints.Studio.endpoint -o tsv
46
+
az ml job show --name <job_name> --query services.Studio.endpoint -o tsv
47
47
```
48
48
49
49
1. Open the job in studio using the value returned by the above command.
50
50
1. Choose __batchscoring__
51
51
1. Open the __Outputs + logs__ tab
52
-
1. Choose the log(s) you wish to review
52
+
1. Choose one or more logs you wish to review
53
53
54
54
### Understand log structure
55
55
@@ -154,9 +154,9 @@ __Reason__: The compute cluster where the deployment is running can't mount the
154
154
155
155
__Solutions__: Ensure the identity associated with the compute cluster where your deployment is running has at least has at least [Storage Blob Data Reader](../role-based-access-control/built-in-roles.md#storage-blob-data-reader) access to the storage account. Only storage account owners can [change your access level via the Azure portal](../storage/blobs/assign-azure-role-data-access.md).
156
156
157
-
### Data set node [code] references parameter dataset_param which doesn't have a specified value or a default value
157
+
### Data set node [code] references parameter `dataset_param` which doesn't have a specified value or a default value
158
158
159
-
__Message logged__: Data set node [code] references parameter dataset_param which doesn't have a specified value or a default value.
159
+
__Message logged__: Data set node [code] references parameter `dataset_param` which doesn't have a specified value or a default value.
160
160
161
161
__Reason__: The input data asset provided to the batch endpoint isn't supported.
162
162
@@ -176,13 +176,13 @@ __Message logged__: ValueError: No objects to concatenate.
176
176
177
177
__Reason__: All the files in the generated mini-batch are either corrupted or unsupported file types. Remember that MLflow models support a subset of file types as documented at [Considerations when deploying to batch inference](how-to-mlflow-batch.md?#considerations-when-deploying-to-batch-inference).
178
178
179
-
__Solution__: Go to the file `logs/usr/stdout/<process-number>/process000.stdout.txt` and look for entries like `ERROR:azureml:Error processing input file`. If the file type isn't supported, please review the list of supported files. You may need to change the file type of the input data or customize the deployment by providing a scoring script as indicated at [Using MLflow models with a scoring script](how-to-mlflow-batch.md?#customizing-mlflow-models-deployments-with-a-scoring-script).
179
+
__Solution__: Go to the file `logs/usr/stdout/<process-number>/process000.stdout.txt` and look for entries like `ERROR:azureml:Error processing input file`. If the file type isn't supported, review the list of supported files. You may need to change the file type of the input data, or customize the deployment by providing a scoring script as indicated at [Using MLflow models with a scoring script](how-to-mlflow-batch.md?#customizing-mlflow-models-deployments-with-a-scoring-script).
180
180
181
181
### There is no succeeded mini batch item returned from run()
182
182
183
183
__Message logged__: There is no succeeded mini batch item returned from run(). Please check 'response: run()' in https://aka.ms/batch-inference-documentation.
184
184
185
-
__Reason__: The batch endpoint failed to provide data in the expected format to the `run()` method. This may be due to corrupted files being read or incompatibility of the input data with the signature of the model (MLflow).
185
+
__Reason__: The batch endpoint failed to provide data in the expected format to the `run()` method. It can be due to corrupted files being read or incompatibility of the input data with the signature of the model (MLflow).
186
186
187
187
__Solution__: To understand what may be happening, go to __Outputs + Logs__ and open the file at `logs > user > stdout > 10.0.0.X > process000.stdout.txt`. Look for error entries like `Error processing input file`. You should find there details about why the input file can't be correctly read.
188
188
@@ -192,7 +192,7 @@ __Context__: When invoking a batch endpoint using its REST APIs.
192
192
193
193
__Reason__: The access token used to invoke the REST API for the endpoint/deployment is indicating a token that is issued for a different audience/service. Microsoft Entra tokens are issued for specific actions.
194
194
195
-
__Solution__: When generating an authentication token to be used with the Batch Endpoint REST API, ensure the `resource` parameter is set to `https://ml.azure.com`. Please notice that this resource is different from the resource you need to indicate to manage the endpoint using the REST API. All Azure resources (including batch endpoints) use the resource `https://management.azure.com` for managing them. Ensure you use the right resource URI on each case. Notice that if you want to use the management API and the job invocation API at the same time, you'll need two tokens. For details see: [Authentication on batch endpoints (REST)](how-to-authenticate-batch-endpoint.md?tabs=rest).
195
+
__Solution__: When generating an authentication token to be used with the Batch Endpoint REST API, ensure the `resource` parameter is set to `https://ml.azure.com`. Notice that this resource is different from the resource you need to indicate to manage the endpoint using the REST API. All Azure resources (including batch endpoints) use the resource `https://management.azure.com` for managing them. Ensure you use the right resource URI on each case. Notice that if you want to use the management API and the job invocation API at the same time, you'll need two tokens. For details see: [Authentication on batch endpoints (REST)](how-to-authenticate-batch-endpoint.md?tabs=rest).
196
196
197
197
### No valid deployments to route to. Please check that the endpoint has at least one deployment with positive weight values or use a deployment specific header to route.
0 commit comments