You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/search/cognitive-search-output-field-mapping.md
+19-17Lines changed: 19 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,5 @@
1
1
---
2
-
title: Map skill output fields
2
+
title: Map enrichments in indexers
3
3
titleSuffix: Azure Cognitive Search
4
4
description: Export the enriched content created by a skillset by mapping its output fields to fields in a search index.
5
5
@@ -30,15 +30,15 @@ In contrast with a [`fieldMappings`](search-indexer-field-mappings.md) definitio
30
30
31
31
Output field mappings are required if your indexer has an attached [skillset](cognitive-search-working-with-skillsets.md) that creates new information, such as text translation or key phrase extraction. During indexer execution, AI-generated information exists in memory only. To persist this information in a search index, you'll need to tell the indexer where to send the data.
32
32
33
-
Output field mappings can also be used to [flatten nested data structures]() during indexing. A regular [fieldMapping definition](search-indexer-field-mappings.md) doesn't support target fields of a complex type. If you need to set up field associations for hierarchical or nested data structures, you can use a skillset and an output field mapping to create the data path.
33
+
Output field mappings can also be used to retrieve specific nodes in a source document's complex type. If you don't need the full complex structure, you can [flatten individual nodes in a nested data structures](#flattening-information-from-complex-types) into a string collection.
34
34
35
35
Output field mappings apply to:
36
36
37
37
+ Content that's created by skills or extracted by an indexer. The source field is a node in an enriched document residing in memory.
38
38
39
39
+ Search indexes. If you're populating a [knowledge store](knowledge-store-concept-intro.md), use [projections](knowledge-store-projections-examples.md) for data path configuration.
40
40
41
-
Output field mappings always occur after [skillset execution](cognitive-search-working-with-skillsets.md), although it's possible to processing an output field mapping if no skillset is defined. See
41
+
Output field mappings are applied after [skillset execution](cognitive-search-working-with-skillsets.md) or after document cracking if there's no associated skillset.
42
42
43
43
## Define an output field mapping
44
44
@@ -48,7 +48,7 @@ Output field mappings are added to the `outputFieldMappings` array in an indexer
@@ -57,14 +57,14 @@ Output field mappings are added to the `outputFieldMappings` array in an indexer
57
57
58
58
| Property | Description |
59
59
|----------|-------------|
60
-
|"sourceFieldName"| Required. Specifies a path to enriched content. See [Reference annotations in an Azure Cognitive Search skillset](cognitive-search-concept-annotations-syntax.md) for path syntax. |
61
-
| "targetFieldName"| Optional. Specifies the search field that receives the enriched content. This is always a single top-level field or a collection. Target fields must be top-level simple fields or collections. It can't be a path to a subfield in a complex type. Although a target field can't resolve to subfield, you can [flatten a nested source structure into a string collection](#flatten-information-from-complex-types) in memory, and then send it to a string collection in your index. |
62
-
|"mappingFunction"| Optional. Adds extra processing provided by [mapping functions](search-indexer-field-mappings.md#mappingFunctions) supported by indexers. In the case of enrichment nodes, encoding and decoding are the most commonly used functions. |
60
+
| sourceFieldName | Required. Specifies a path to enriched content. An example might be `/document/content`. See [Reference annotations in an Azure Cognitive Search skillset](cognitive-search-concept-annotations-syntax.md) for path syntax and examples. |
61
+
| targetFieldName | Optional. Specifies the search field that receives the enriched content. Target fields must be top-level simple fields or collections. It can't be a path to a subfield in a complex type. If you want to retrieve specific nodes in a complex structure, you can [flatten individual nodes](#flatten-information-from-complex-types) in memory, and then send the output to a string collection in your index. |
62
+
| mappingFunction | Optional. Adds extra processing provided by [mapping functions](search-indexer-field-mappings.md#mappingFunctions) supported by indexers. In the case of enrichment nodes, encoding and decoding are the most commonly used functions. |
63
63
64
64
You can use the REST API or an Azure SDK to define output field mappings.
65
65
66
66
> [!TIP]
67
-
> Indexers created by the [Import data wizard](search-import-data-portal.md) include output field mappings generated by the wizard. If you need examples, run the wizard over your data source to see the full definition.
67
+
> Indexers created by the [Import data wizard](search-import-data-portal.md) include output field mappings generated by the wizard. If you need examples, run the wizard over your data source to see the rendered definition.
68
68
69
69
### [**REST APIs**](#tab/rest)
70
70
@@ -102,7 +102,7 @@ api-key: [admin key]
102
102
}
103
103
```
104
104
105
-
For each output field mapping, set the location of the data in the enriched document tree (sourceFieldName), and the name of the field as referenced in the index (targetFieldName). Assign any [mapping functions](search-indexer-field-mappings.md#field-mapping-functions-and-examples) that you require to transform the content of a field before it's stored in the index.
105
+
For each output field mapping, set the location of the data in the enriched document tree (sourceFieldName), and the name of the field as referenced in the index (targetFieldName). Assign any [mapping functions](search-indexer-field-mappings.md#mappingFunctions) that you require to transform the content of a field before it's stored in the index.
## Flatten complex structures into a string collection
138
138
139
-
If your source data is composed of nested or hierarchical JSON, you can't use field mappings to set up the data paths. Instead, your search index must mirror the source data structure for at each level for a full import.
139
+
If your source data is composed of nested or hierarchical JSON, you can't use field mappings to set up the data paths. Instead, your search index must mirror the source data structure for at each level for a full import.
140
+
141
+
This section walks you through an import process that produces a one-to-one reflection of a complex document on both the source and target sides. Next, it uses the same source document to illustrate the retrieval and flattening of individual nodes into string collections.
140
142
141
-
Sample source JSON document in Cosmos DB with nested JSON:
143
+
Here's an example of a document in Cosmos DB with nested JSON:
142
144
143
145
```json
144
146
{
@@ -171,7 +173,7 @@ Sample source JSON document in Cosmos DB with nested JSON:
171
173
}
172
174
```
173
175
174
-
Sample index definition in Cognitive Search, where names, levels, and types are reflected as a complex type:
176
+
If you wanted to fully index the above source document, you'd create an index definition where the field names, levels, and types are reflected as a complex type. Because field mappings aren't supported for complex types in the search index, your index definition must mirror the source document.
175
177
176
178
```json
177
179
{
@@ -200,7 +202,7 @@ Sample index definition in Cognitive Search, where names, levels, and types are
200
202
}
201
203
```
202
204
203
-
Sample indexer definition that runs the import (notice there are no field mappings and no skillset):
205
+
Here's a sample indexer definition that executes the import (notice there are no field mappings and no skillset).
204
206
205
207
```json
206
208
{
@@ -214,7 +216,7 @@ Sample indexer definition that runs the import (notice there are no field mappin
214
216
}
215
217
```
216
218
217
-
Sample search document, post-import, is similar to the original in Cosmos DB:
219
+
The result is the following sample search document, similar to the original in Cosmos DB.
218
220
219
221
```json
220
222
{
@@ -253,9 +255,9 @@ Sample search document, post-import, is similar to the original in Cosmos DB:
253
255
}
254
256
```
255
257
256
-
An alternative rendering in a search index is to flatten the source's nested structure to a string collection in a search index.
258
+
An alternative rendering in a search index is to flatten individual nodes in the source's nested structure into a string collection in a search index.
257
259
258
-
To accomplish this task, you'll need an `outputFieldMapping` that maps an in-memory node to a string collection in the index. Although output field mappings primarily apply to skill outputs, you can also use them to address nodes after "document cracking" where the indexer opens a source document and reads it into memory.
260
+
To accomplish this task, you'll need an `outputFieldMapping` that maps an in-memory node to a string collection in the index. Although output field mappings primarily apply to skill outputs, you can also use them to address nodes after ["document cracking"](search-indexer-overview.md#stage-1-document-cracking) where the indexer opens a source document and reads it into memory.
259
261
260
262
Below is a sample index definition in Cognitive Search, using string collections to receive flattened output:
When using an [Azure Cognitive Search indexer](search-indexer-overview.md) to push content into a search index, the indexer can often detect and assign source-to-destination field mappings. Implicit field mappings occur when field names and data types are compatible.
19
+
When using an [Azure Cognitive Search indexer](search-indexer-overview.md) to pull content into a search index, the indexer can often implicitly detect and assign source-to-destination field mappings. Implicit field mappings occur when field names and data types are compatible.
20
20
21
-
If inputs and outputs don't match, you can define explicit *field mappings* to set up the data path, as described in this article. Field mappings can also be used to provide light-weight data conversion, such as encoding or decoding, through [mapping functions](#mappingFunctions). If more processing is required, consider [Azure Data Factory](../data-factory/index.yml) to bridge the gap.
21
+
If inputs and outputs don't match, you can define explicit *field mappings* to set up the data path, as described in this article. Field mappings can also be used to introduce light-weight data conversion, such as encoding or decoding, through [mapping functions](#mappingFunctions). If more processing is required, consider [Azure Data Factory](../data-factory/index.yml) to bridge the gap.
22
22
23
23
Field mappings apply to:
24
24
25
-
+ Physical data structures on both sides of the data stream (that is, between a [supported data source](search-indexer-overview.md#supported-data-sources) and a [search index](search-what-is-an-index.md)). If you're importing enriched content created by skills, use [outputFieldMappings](cognitive-search-output-field-mapping.md) instead.
25
+
+ Physical data structures on both sides of the data stream (between a [supported data source](search-indexer-overview.md#supported-data-sources) and a [search index](search-what-is-an-index.md)). If you're importing skill enriched content that resides in memory, use [outputFieldMappings](cognitive-search-output-field-mapping.md) instead.
26
26
27
27
+ Search indexes only. If you're populating a [knowledge store](knowledge-store-concept-intro.md), use [projections](knowledge-store-projections-examples.md) for data path configuration.
28
28
29
29
+ Top-level search fields only, where the "targetFieldName" is either a simple field or a collection. A target field can't be a complex type.
30
30
31
-
If you're working with complex data (nested or hierarchical structures), and you'd like to mirror that data structure in your search index, your search index must match the source structure exactly (same field names, levels, and types) so that the default mappings will work. Optionally, you can flatten incoming data into a string collection (see [outputFieldMappings](cognitive-search-output-field-mapping.md) for this workaround).
31
+
> [!NOTE]
32
+
> If you're working with complex data (nested or hierarchical structures), and you'd like to mirror that data structure in your search index, your search index must match the source structure exactly (same field names, levels, and types) so that the default mappings will work. Optionally, you might want just a few nodes in the complex structure. To get individual nodes, you can flatten incoming data into a string collection (see [outputFieldMappings](cognitive-search-output-field-mapping.md#flatten-complex-structures-into-a-string-collection) for this workaround).
32
33
33
34
## Supported scenarios
34
35
@@ -56,9 +57,9 @@ Field mappings are added to the "fieldMappings" array of an indexer definition.
56
57
57
58
| Property | Description |
58
59
|----------|-------------|
59
-
|"sourceFieldName"| Required. Represents a field in your data source. |
60
-
| "targetFieldName"| Optional. Represents a field in your search index. If omitted, the value of "sourceFieldName" is assumed for the target. Target fields must be top-level simple fields or collections. It can't be a complex type or collection. |
61
-
|"mappingFunction"| Optional. Consists of [predefined functions](#mappingFunctions) that transform data. You can apply functions to both source and target field mappings.|
60
+
| sourceFieldName | Required. Represents a field in your data source. |
61
+
| targetFieldName | Optional. Represents a field in your search index. If omitted, the value of "sourceFieldName" is assumed for the target. Target fields must be top-level simple fields or collections. It can't be a complex type or collection. If you're handling a data type issue, a field's data type is specified in the index definition. The field mapping just needs to have the field's name.|
62
+
| mappingFunction | Optional. Consists of [predefined functions](#mappingFunctions) that transform data. |
62
63
63
64
Azure Cognitive Search uses case-insensitive comparison to resolve the field and function names in field mappings. This is convenient (you don't have to get all the casing right), but it means that your data source or index can't have fields that differ only by case.
64
65
@@ -84,7 +85,7 @@ api-key: [admin key]
84
85
}
85
86
```
86
87
87
-
Map a single source field to multiple target fields ("one-to-many" mappings). The following example shows how to "fork" a field, copying the same source field content to two different index fields that will be analyzed or attributed differently in the index.
88
+
This example maps a single source field to multiple target fields ("one-to-many" mappings). You can "fork" a field, copying the same source field content to two different index fields that will be analyzed or attributed differently in the index.
0 commit comments