Skip to content

Commit 7fb8da6

Browse files
v-regandownerv-regandowner
authored
Merge pull request #212488 from vadthyavath/rvadthyavath/mlflow_schema
mlflow input and output schema v2
2 parents cd76bdb + fa4679c commit 7fb8da6

File tree

1 file changed

+157
-128
lines changed

1 file changed

+157
-128
lines changed

‎articles/machine-learning/reference-automl-images-schema.md

Lines changed: 157 additions & 128 deletions
Original file line numberDiff line numberDiff line change
@@ -235,170 +235,199 @@ Example of a JSONL file for Instance Segmentation:
235235

236236
![Image example for instance segmentation.](media/reference-automl-images-schema/instance-segmentation-predictions.jpg)
237237

238-
## Data format for inference
238+
## Data schema for online scoring
239239

240-
In this section, we document the input data format required to make predictions when using a deployed model. Any aforementioned image format is accepted with content type `application/octet-stream`.
240+
In this section, we document the input data format required to make predictions using a deployed model.
241241

242242
### Input format
243243

244-
The following is the input format needed to generate predictions on any task using task-specific model endpoint. After we [deploy the model](how-to-auto-train-image-models.md#register-and-deploy-model), we can use the following code snippet to get predictions for all tasks.
244+
The following is the input format needed to generate predictions on any task using task-specific model endpoint.
245245

246-
```python
247-
# input image for inference
248-
sample_image = './test_image.jpg'
249-
# load image data
250-
data = open(sample_image, 'rb').read()
251-
# set the content type
252-
headers = {'Content-Type': 'application/octet-stream'}
253-
# if authentication is enabled, set the authorization header
254-
headers['Authorization'] = f'Bearer {key}'
255-
# make the request and display the response
256-
response = requests.post(scoring_uri, data, headers=headers)
246+
```json
247+
{
248+
"input_data": {
249+
"columns": [
250+
"image"
251+
],
252+
"data": [
253+
"image_in_base64_string_format"
254+
]
255+
}
256+
}
257257
```
258+
259+
This json is a dictionary with outer key `input_data` and inner keys `columns`, `data` as described in the following table. The endpoint accepts a json string in the above format and converts it into a dataframe of samples required by the scoring script. Each input image in the `request_json["input_data"]["data"]` section of the json is a [base64 encoded string](https://docs.python.org/3/library/base64.html#base64.encodebytes).
260+
261+
262+
| Key | Description |
263+
| -------- |----------|
264+
| `input_data`<br> (outer key) | It is an outer key in json request. `input_data` is a dictionary that accepts input image samples <br>`Required, Dictionary` |
265+
| `columns`<br> (inner key) | Column names to use to create dataframe. It accepts only one column with `image` as column name.<br>`Required, List` |
266+
| `data`<br> (inner key) | List of base64 encoded images <br>`Required, List`|
267+
268+
269+
After we [deploy the mlflow model](how-to-auto-train-image-models.md#register-and-deploy-model), we can use the following code snippet to get predictions for all tasks.
270+
271+
[!Notebook-python[] (~/azureml-examples-v2samplesreorg/sdk/python/jobs/automl-standalone-jobs/automl-image-object-detection-task-fridge-items/automl-image-object-detection-task-fridge-items.ipynb?name=create_inference_request)]
272+
273+
[!Notebook-python[] (~/azureml-examples-v2samplesreorg/sdk/python/jobs/automl-standalone-jobs/automl-image-object-detection-task-fridge-items/automl-image-object-detection-task-fridge-items.ipynb?name=dump_inference_request)]
274+
275+
[!Notebook-python[] (~/azureml-examples-v2samplesreorg/sdk/python/jobs/automl-standalone-jobs/automl-image-object-detection-task-fridge-items/automl-image-object-detection-task-fridge-items.ipynb?name=invoke_inference)]
276+
258277
### Output format
259278

260-
Predictions made on model endpoints follow different structure depending on the task type. This section explores the output data formats for multi-class, multi-label image classification, object detection, and instance segmentation tasks.
279+
Predictions made on model endpoints follow different structure depending on the task type. This section explores the output data formats for multi-class, multi-label image classification, object detection, and instance segmentation tasks.
280+
281+
The following schemas are applicable when the input request contains one image.
261282

262283
#### Image classification
263284

264285
Endpoint for image classification returns all the labels in the dataset and their probability scores for the input image in the following format.
265286

266287
```json
267-
{
268-
"filename":"/tmp/tmppjr4et28",
269-
"probs":[
270-
2.098e-06,
271-
4.783e-08,
272-
0.999,
273-
8.637e-06
274-
],
275-
"labels":[
276-
"can",
277-
"carton",
278-
"milk_bottle",
279-
"water_bottle"
280-
]
281-
}
288+
[
289+
{
290+
"filename": "/tmp/tmppjr4et28",
291+
"probs": [
292+
2.098e-06,
293+
4.783e-08,
294+
0.999,
295+
8.637e-06
296+
],
297+
"labels": [
298+
"can",
299+
"carton",
300+
"milk_bottle",
301+
"water_bottle"
302+
]
303+
}
304+
]
282305
```
283306

284307
#### Image classification multi-label
285308

286309
For image classification multi-label, model endpoint returns labels and their probabilities.
287310

288311
```json
289-
{
290-
"filename":"/tmp/tmpsdzxlmlm",
291-
"probs":[
292-
0.997,
293-
0.960,
294-
0.982,
295-
0.025
296-
],
297-
"labels":[
298-
"can",
299-
"carton",
300-
"milk_bottle",
301-
"water_bottle"
302-
]
303-
}
312+
[
313+
{
314+
"filename": "/tmp/tmpsdzxlmlm",
315+
"probs": [
316+
0.997,
317+
0.960,
318+
0.982,
319+
0.025
320+
],
321+
"labels": [
322+
"can",
323+
"carton",
324+
"milk_bottle",
325+
"water_bottle"
326+
]
327+
}
328+
]
304329
```
305330

306331
#### Object detection
307332

308333
Object detection model returns multiple boxes with their scaled top-left and bottom-right coordinates along with box label and confidence score.
309334

310335
```json
311-
{
312-
"filename":"/tmp/tmpdkg2wkdy",
313-
"boxes":[
314-
{
315-
"box":{
316-
"topX":0.224,
317-
"topY":0.285,
318-
"bottomX":0.399,
319-
"bottomY":0.620
320-
},
321-
"label":"milk_bottle",
322-
"score":0.937
323-
},
324-
{
325-
"box":{
326-
"topX":0.664,
327-
"topY":0.484,
328-
"bottomX":0.959,
329-
"bottomY":0.812
336+
[
337+
{
338+
"filename": "/tmp/tmpdkg2wkdy",
339+
"boxes": [
340+
{
341+
"box": {
342+
"topX": 0.224,
343+
"topY": 0.285,
344+
"bottomX": 0.399,
345+
"bottomY": 0.620
346+
},
347+
"label": "milk_bottle",
348+
"score": 0.937
330349
},
331-
"label":"can",
332-
"score":0.891
333-
},
334-
{
335-
"box":{
336-
"topX":0.423,
337-
"topY":0.253,
338-
"bottomX":0.632,
339-
"bottomY":0.725
350+
{
351+
"box": {
352+
"topX": 0.664,
353+
"topY": 0.484,
354+
"bottomX": 0.959,
355+
"bottomY": 0.812
356+
},
357+
"label": "can",
358+
"score": 0.891
340359
},
341-
"label":"water_bottle",
342-
"score":0.876
343-
}
344-
]
345-
}
360+
{
361+
"box": {
362+
"topX": 0.423,
363+
"topY": 0.253,
364+
"bottomX": 0.632,
365+
"bottomY": 0.725
366+
},
367+
"label": "water_bottle",
368+
"score": 0.876
369+
}
370+
]
371+
}
372+
]
346373
```
347374
#### Instance segmentation
348375

349376
In instance segmentation, output consists of multiple boxes with their scaled top-left and bottom-right coordinates, labels, confidence scores, and polygons (not masks). Here, the polygon values are in the same format that we discussed in the schema section.
350377

351378
```json
352-
{
353-
"filename":"/tmp/tmpi8604s0h",
354-
"boxes":[
355-
{
356-
"box":{
357-
"topX":0.679,
358-
"topY":0.491,
359-
"bottomX":0.926,
360-
"bottomY":0.810
361-
},
362-
"label":"can",
363-
"score":0.992,
364-
"polygon":[
365-
[
366-
0.82, 0.811, 0.771, 0.810, 0.758, 0.805, 0.741, 0.797, 0.735, 0.791, 0.718, 0.785, 0.715, 0.778, 0.706, 0.775, 0.696, 0.758, 0.695, 0.717, 0.698, 0.567, 0.705, 0.552, 0.706, 0.540, 0.725, 0.520, 0.735, 0.505, 0.745, 0.502, 0.755, 0.493
367-
]
368-
]
369-
},
370-
{
371-
"box":{
372-
"topX":0.220,
373-
"topY":0.298,
374-
"bottomX":0.397,
375-
"bottomY":0.601
376-
},
377-
"label":"milk_bottle",
378-
"score":0.989,
379-
"polygon":[
380-
[
381-
0.365, 0.602, 0.273, 0.602, 0.26, 0.595, 0.263, 0.588, 0.251, 0.546, 0.248, 0.501, 0.25, 0.485, 0.246, 0.478, 0.245, 0.463, 0.233, 0.442, 0.231, 0.43, 0.226, 0.423, 0.226, 0.408, 0.234, 0.385, 0.241, 0.371, 0.238, 0.345, 0.234, 0.335, 0.233, 0.325, 0.24, 0.305, 0.586, 0.38, 0.592, 0.375, 0.598, 0.365
382-
]
383-
]
384-
},
385-
{
386-
"box":{
387-
"topX":0.433,
388-
"topY":0.280,
389-
"bottomX":0.621,
390-
"bottomY":0.679
391-
},
392-
"label":"water_bottle",
393-
"score":0.988,
394-
"polygon":[
395-
[
396-
0.576, 0.680, 0.501, 0.680, 0.475, 0.675, 0.460, 0.625, 0.445, 0.630, 0.443, 0.572, 0.440, 0.560, 0.435, 0.515, 0.431, 0.501, 0.431, 0.433, 0.433, 0.426, 0.445, 0.417, 0.456, 0.407, 0.465, 0.381, 0.468, 0.327, 0.471, 0.318
397-
]
398-
]
399-
}
400-
]
401-
}
379+
[
380+
{
381+
"filename": "/tmp/tmpi8604s0h",
382+
"boxes": [
383+
{
384+
"box": {
385+
"topX": 0.679,
386+
"topY": 0.491,
387+
"bottomX": 0.926,
388+
"bottomY": 0.810
389+
},
390+
"label": "can",
391+
"score": 0.992,
392+
"polygon": [
393+
[
394+
0.82, 0.811, 0.771, 0.810, 0.758, 0.805, 0.741, 0.797, 0.735, 0.791, 0.718, 0.785, 0.715, 0.778, 0.706, 0.775, 0.696, 0.758, 0.695, 0.717, 0.698, 0.567, 0.705, 0.552, 0.706, 0.540, 0.725, 0.520, 0.735, 0.505, 0.745, 0.502, 0.755, 0.493
395+
]
396+
]
397+
},
398+
{
399+
"box": {
400+
"topX": 0.220,
401+
"topY": 0.298,
402+
"bottomX": 0.397,
403+
"bottomY": 0.601
404+
},
405+
"label": "milk_bottle",
406+
"score": 0.989,
407+
"polygon": [
408+
[
409+
0.365, 0.602, 0.273, 0.602, 0.26, 0.595, 0.263, 0.588, 0.251, 0.546, 0.248, 0.501, 0.25, 0.485, 0.246, 0.478, 0.245, 0.463, 0.233, 0.442, 0.231, 0.43, 0.226, 0.423, 0.226, 0.408, 0.234, 0.385, 0.241, 0.371, 0.238, 0.345, 0.234, 0.335, 0.233, 0.325, 0.24, 0.305, 0.586, 0.38, 0.592, 0.375, 0.598, 0.365
410+
]
411+
]
412+
},
413+
{
414+
"box": {
415+
"topX": 0.433,
416+
"topY": 0.280,
417+
"bottomX": 0.621,
418+
"bottomY": 0.679
419+
},
420+
"label": "water_bottle",
421+
"score": 0.988,
422+
"polygon": [
423+
[
424+
0.576, 0.680, 0.501, 0.680, 0.475, 0.675, 0.460, 0.625, 0.445, 0.630, 0.443, 0.572, 0.440, 0.560, 0.435, 0.515, 0.431, 0.501, 0.431, 0.433, 0.433, 0.426, 0.445, 0.417, 0.456, 0.407, 0.465, 0.381, 0.468, 0.327, 0.471, 0.318
425+
]
426+
]
427+
}
428+
]
429+
}
430+
]
402431
```
403432

404433
> [!NOTE]

0 commit comments

Comments
 (0)