You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This article explains how to set explicit field mappings that establish the data path between source fields in a supported data source and target fields in a search index.
22
+
23
+
## When to set a field mapping
24
+
21
25
When an [Azure AI Search indexer](search-indexer-overview.md) loads a search index, it determines the data path using source-to-destination field mappings. Implicit field mappings are internal and occur when field names and data types are compatible between the source and destination. If inputs and outputs don't match, you can define explicit *field mappings* to set up the data path, as described in this article.
22
26
23
27
Field mappings can also be used for light-weight data conversions, such as encoding or decoding, through [mapping functions](#mappingFunctions). If more processing is required, consider [Azure Data Factory](../data-factory/index.yml) to bridge the gap.
24
28
25
29
Field mappings apply to:
26
30
27
-
+ Physical data structures on both sides of the data stream (between fields in a [supported data source](search-indexer-overview.md#supported-data-sources) and fields in a [search index](search-what-is-an-index.md)). If you're importing skill-enriched content that resides in memory, use[outputFieldMappings](cognitive-search-output-field-mapping.md) to map in-memory nodes to output fields in a search index.
31
+
+ Physical data structures on both sides of the data path. Logical data structures created by skills reside only in memory. Use[outputFieldMappings](cognitive-search-output-field-mapping.md) to map in-memory nodes to output fields in a search index.
28
32
29
-
+ Search indexes only. If you're populating a [knowledge store](knowledge-store-concept-intro.md), use [projections](knowledge-store-projections-examples.md) for data path configuration.
33
+
+ Search indexes only. To populate a [knowledge store](knowledge-store-concept-intro.md), use [projections](knowledge-store-projections-examples.md) for data path configuration.
30
34
31
35
+ Top-level search fields only, where the `targetFieldName` is either a simple field or a collection. A target field can't be a complex type.
32
36
33
-
> [!NOTE]
34
-
> If you're working with complex data (nested or hierarchical structures), and you'd like to mirror that data structure in your search index, your search index must match the source structure exactly (same field names, levels, and types) so that the default mappings will work. Optionally, you might want just a few nodes in the complex structure. To get individual nodes, you can flatten incoming data into a string collection (see [outputFieldMappings](cognitive-search-output-field-mapping.md#flatten-complex-structures-into-a-string-collection) for this workaround).
35
-
36
37
## Supported scenarios
37
38
39
+
Make sure you're using a [supported data source](search-indexer-overview.md#supported-data-sources) for indexer-driving indexing.
40
+
38
41
| Use-case | Description |
39
42
|----------|-------------|
40
43
| Name discrepancy | Suppose your data source has a field named `_city`. Given that Azure AI Search doesn't allow field names that start with an underscore, a field mapping lets you effectively map "_city" to "city". </p>If your indexing requirements include retrieving content from multiple data sources, where field names vary among the sources, you could use a field mapping to clarify the path.|
@@ -43,40 +46,58 @@ Field mappings apply to:
43
46
| Encoding and decoding | You can apply [mapping functions](#mappingFunctions) to support Base64 encoding or decoding of data during indexing. |
44
47
| Split strings or recast arrays into collections | You can apply [mapping functions](#mappingFunctions) to split a string that includes a delimiter, or to send a JSON array to a search field of type `Collection(Edm.String)`.
45
48
46
-
## Define a field mapping
49
+
> [!NOTE]
50
+
> If no field mappings are present, indexers assume data source fields should be mapped to index fields with the same name. Adding a field mapping overrides the default field mappings for the source and target field. Some indexers, such as the [blob storage indexer](search-howto-indexing-azure-blob-storage.md), add default field mappings for the index key field automatically.
47
51
48
-
Field mappings are added to the `fieldMappings` array of an indexer definition. A field mapping consists of three parts.
52
+
Complex fields aren't supported in a field mapping. Your source structure (nested or hierarchical structures) must exactly match the complex type in the index so that the default mappings work. For more information, see [Tutorial: Index nested JSON blobs](search-semi-structured-data.md) for an example. If you get an error similar to `"Field mapping specifies target field 'Address/city' that doesn't exist in the index"`, it's because target field mappings can't be a complex type.
49
53
50
-
```json
51
-
"fieldMappings": [
52
-
{
53
-
"sourceFieldName": "_city",
54
-
"targetFieldName": "city",
55
-
"mappingFunction": null
56
-
}
57
-
]
58
-
```
54
+
Optionally, you might want just a few nodes in the complex structure. To get individual nodes, you can flatten incoming data into a string collection (see [outputFieldMappings](cognitive-search-output-field-mapping.md#flatten-complex-structures-into-a-string-collection) for this workaround).
59
55
60
-
| Property | Description |
61
-
|----------|-------------|
62
-
| sourceFieldName | Required. Represents a field in your data source. |
63
-
| targetFieldName | Optional. Represents a field in your search index. If omitted, the value of `sourceFieldName` is assumed for the target. Target fields must be top-level simple fields or collections. It can't be a complex type or collection. If you're handling a data type issue, a field's data type is specified in the index definition. The field mapping just needs to have the field's name.|
64
-
| mappingFunction | Optional. Consists of [predefined functions](#mappingFunctions) that transform data. |
56
+
## Define a field mapping
65
57
66
-
If you get an error similar to `"Field mapping specifies target field 'Address/city' that doesn't exist in the index"`, it's because target field mappings can't be a complex type. The workaround is to create an index schema that's identical to the raw content for field names and data types. See [Tutorial: Index nested JSON blobs](search-semi-structured-data.md) for an example.
58
+
This section explains the steps for setting up field mappings.
67
59
68
-
Azure AI Search uses case-insensitive comparison to resolve the field and function names in field mappings. This is convenient (you don't have to get all the casing right), but it means that your data source or index can't have fields that differ only by case.
60
+
### [**REST APIs**](#tab/rest)
69
61
70
-
> [!NOTE]
71
-
> If no field mappings are present, indexers assume data source fields should be mapped to index fields with the same name. Adding a field mapping overrides the default field mappings for the source and target field. Some indexers, such as the [blob storage indexer](search-howto-indexing-azure-blob-storage.md), add default field mappings for the index key field.
62
+
1. Use [Create Indexer](/rest/api/searchservice/indexers/create) or [Create or Update Indexer](/rest/api/searchservice/indexers/create-or-update) or an equivalent method in an Azure SDK. Here's an example of an indexer definition.
63
+
64
+
```json
65
+
{
66
+
"name": "myindexer",
67
+
"description": null,
68
+
"dataSourceName": "mydatasource",
69
+
"targetIndexName": "myindex",
70
+
"schedule": { },
71
+
"parameters": { },
72
+
"fieldMappings": [],
73
+
"disabled": false,
74
+
"encryptionKey": { }
75
+
}
76
+
```
72
77
73
-
You can use the REST API or an Azure SDK to define field mappings.
78
+
1. Fill out the `fieldMappings` array to specify the mappings. A field mapping consists of three parts.
74
79
75
-
### [**REST APIs**](#tab/rest)
80
+
```json
81
+
"fieldMappings": [
82
+
{
83
+
"sourceFieldName": "_city",
84
+
"targetFieldName": "city",
85
+
"mappingFunction": null
86
+
}
87
+
]
88
+
```
89
+
90
+
| Property | Description |
91
+
|----------|-------------|
92
+
| sourceFieldName | Required. Represents a field in your data source. |
93
+
| targetFieldName | Optional. Represents a field in your search index. If omitted, the value of `sourceFieldName` is assumed for the target. Target fields must be top-level simple fields or collections. It can't be a complex type or collection. If you're handling a data type issue, a field's data type is specified in the index definition. The field mapping just needs to have the field's name.|
94
+
| mappingFunction | Optional. Consists of [predefined functions](#mappingFunctions) that transform data. |
95
+
96
+
#### Example: Name or type discrepancy
76
97
77
-
Use [Create Indexer (REST)](/rest/api/searchservice/indexers/create) or [Update Indexer (REST)](/rest/api/searchservice/indexers/create-or-update), any API version.
98
+
An explicit field mapping establishes a data path for cases where name and type aren't identical.
78
99
79
-
This example handles a field name discrepancy.
100
+
Azure AI Search uses case-insensitive comparison to resolve the field and function names in field mappings. This is convenient (you don't have to get all the casing right), but it means that your data source or index can't have fields that differ only by case.
80
101
81
102
```JSON
82
103
PUT https://[service name].search.windows.net/indexers/myindexer?api-version=[api-version]
@@ -89,6 +110,8 @@ api-key: [admin key]
89
110
}
90
111
```
91
112
113
+
#### Example: One-to-many or forked data paths
114
+
92
115
This example maps a single source field to multiple target fields ("one-to-many" mappings). You can "fork" a field, copying the same source field content to two different index fields that will be analyzed or attributed differently in the index.
93
116
94
117
```JSON
@@ -99,6 +122,8 @@ This example maps a single source field to multiple target fields ("one-to-many"
99
122
]
100
123
```
101
124
125
+
You can use a similar approach for [skills-generated content](cognitive-search-output-field-mapping.md).
126
+
102
127
### [**.NET SDK (C#)**](#tab/csharp)
103
128
104
129
In the Azure SDK for .NET, use the [FieldMapping](/dotnet/api/azure.search.documents.indexes.models.fieldmapping) class that provides `SourceFieldName` and `TargetFieldName` properties and an optional `MappingFunction` reference.
@@ -136,7 +161,7 @@ A field mapping function transforms the contents of a field before it's stored i
136
161
+[urlEncode](#urlEncodeFunction)
137
162
+[urlDecode](#urlDecodeFunction)
138
163
139
-
Note that these functions are exclusively supported for parent indexes at this time. They are not compatible with chunked index mapping, therefore, these functions can't be used for [index projections](index-projections-concept-intro.md).
164
+
Note that these functions are exclusively supported for parent indexes at this time. They aren't compatible with chunked index mapping, therefore, these functions can't be used for [index projections](index-projections-concept-intro.md).
140
165
141
166
<aname="base64EncodeFunction"></a>
142
167
@@ -301,7 +326,7 @@ For example, if the input string is `["red", "white", "blue"]`, then the target
301
326
302
327
#### Example - populate collection from relational data
303
328
304
-
Azure SQL Database doesn't have a built-in data type that naturally maps to `Collection(Edm.String)` fields in Azure AI Search. To populate string collection fields, you can pre-process your source data as a JSON string array and then use the `jsonArrayToStringCollection` mapping function.
329
+
Azure SQL Database doesn't have a built-in data type that naturally maps to `Collection(Edm.String)` fields in Azure AI Search. To populate string collection fields, you can preprocess your source data as a JSON string array and then use the `jsonArrayToStringCollection` mapping function.
![prmerger-automator[bot]](https://avatars.githubusercontent.com/u/22479449?v=4&size=40){kind=avatar}
0 commit comments