You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/search/knowledge-store-concept-intro.md
+16-11Lines changed: 16 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,13 +21,13 @@ Knowledge store is a feature of Azure Search that saves enriched documents and m
21
21
22
22
If you have used cognitive search in the past, you already know that skillsets are used to move a document through a sequence of enrichments. The outcome can be an Azure Search index, or (new in this preview) projections in a knowledge store. The two outputs, search index and knowledge store, are physically distinct from each other. They share the same content, but are stored and used in very different ways.
23
23
24
-
Physically, a knowledge store is created in an Azure Storage account, either as Azure Table storage or Blob storage, depending on how you configure the pipeline. Any tool or process that can connect to Azure Storage can consume the contents of a knowledge store.
24
+
Physically, a knowledge store is an Azure Storage account, either as Azure Table storage, Blob storage or both, depending on how you configure the pipeline. Any tool or process that can connect to Azure Storage can consume the contents of a knowledge store.
25
25
26
26
Projections are your mechanism for structuring data in a knowledge store. For example, through projections, you can choose whether output is saved as a single blob or a collection of related tables. An easy way to view knowledge store contents is through the built-in [Storage Explorer](https://docs.microsoft.com/azure/vs-azure-tools-storage-manage-with-storage-explorer?tabs=windows) for Azure storage.
27
27
28
28

29
29
30
-
To use knowledge store, add a `knowledgeStore` element to a skillset that defines step-wise operations in an indexing pipeline. During execution, Azure Search creates a space in your Azure storage account and fills it with definitions and content created by the pipeline.
30
+
To use knowledge store, add a `knowledgeStore` element to a skillset that defines step-wise operations in an indexing pipeline. During execution, Azure Search creates a space in your Azure storage account and projects the enriched documents with the definition created within the pipeline.
31
31
32
32
## Benefits of knowledge store
33
33
@@ -98,6 +98,13 @@ If you are already familiar with AI-based indexing, the skillset definition dete
@@ -107,7 +114,7 @@ If you are already familiar with AI-based indexing, the skillset definition dete
107
114
"key": "/document/Review/Id"
108
115
}
109
116
]
110
-
}
117
+
}
111
118
]
112
119
}
113
120
}
@@ -138,17 +145,17 @@ Azure Search provides the indexer feature, and indexers are used to drive the en
138
145
| Object | REST API | Description |
139
146
|--------|----------|-------------|
140
147
| data source |[Create Data Source](https://docs.microsoft.com/rest/api/searchservice/create-data-source)| A resource identifying an external Azure data source providing source data used to create enriched documents. |
141
-
| skillset |[Create Skillset (api-version=2019-05-06)](https://docs.microsoft.com/rest/api/searchservice/create-skillset)| A resource coordinating the use of [built-in skills](cognitive-search-predefined-skills.md) and [custom cognitive skills](cognitive-search-custom-skill-interface.md) used in an enrichment pipeline during indexing. |
148
+
| skillset |[Create Skillset (api-version=2019-05-06-Preview)](https://docs.microsoft.com/rest/api/searchservice/create-skillset)| A resource coordinating the use of [built-in skills](cognitive-search-predefined-skills.md) and [custom cognitive skills](cognitive-search-custom-skill-interface.md) used in an enrichment pipeline during indexing. |
142
149
| index |[Create Index](https://docs.microsoft.com/rest/api/searchservice/create-index)| A schema expressing an Azure Search index. Fields in the index map to fields in source data or to fields manufactured during the enrichment phase (for example, a field for organization names created by entity recognition). |
143
150
| indexer |[Create Indexer (api-version=2019-05-06)](https://docs.microsoft.com/rest/api/searchservice/create-skillset)| A resource defining components used during indexing: including a data source, a skillset, field associations from source and intermediary data structures to target index, and the index itself. Running the indexer is the trigger for data ingestion and enrichment. The output is a search index based on the index schema, populated with source data, enriched through skillsets. |
144
151
145
152
### 3 - Cognitive Services
146
153
147
-
Enrichments specified in a skillset are based on the Computer Vision and Language features in Cognitive Services. Cognitive Services functionality is leveraged during indexing through your skillset. A skillset is a composition of skills, and skills are bound to specific Computer Vision and Language features. To integrate Cognitive Services, you'll [attach a Cognitive Services resource](cognitive-search-attach-cognitive-services.md) to a skillset.
154
+
Enrichments specified in a skillset are either custom or based on the Computer Vision and Language features in Cognitive Services. Cognitive Services functionality is leveraged during indexing through your skillset. A skillset is a composition of skills, and skills are bound to specific Computer Vision and Language features. To integrate Cognitive Services, you'll [attach a Cognitive Services resource](cognitive-search-attach-cognitive-services.md) to a skillset.
148
155
149
156
### 4 - Storage account
150
157
151
-
Under your Azure Storage account, Azure Search creates a Blob container or tables, depending on how you configure a skillset. If your data originates from Azure Blob or Table storage, you are already set. Otherwise, you will need to create an Azure storage account. Tables and objects in Azure storage contain the enriched documents created by the AI-based indexing pipeline.
158
+
Under your Azure Storage account, Azure Search creates a Blob container or tables or both, depending on how you configure projections within the skillset. If your data originates from Azure Blob or Table storage, you are already set and can reuse the storage account. Otherwise, you will need to create an Azure storage account. Tables and objects in Azure storage contain the enriched documents created by the AI-based indexing pipeline.
152
159
153
160
The storage account is specified in the skillset. In `api-version=2019-05-06-Preview`, a skillset definition includes a knowledge store definition so that you can provide account information.
154
161
@@ -174,15 +181,13 @@ Within the storage account, the enrichments can be expressed as tables within Az
174
181
175
182
+ Blob storage creates one all-inclusive JSON representation of each document. You can use both storage options in one skillset to get a full range of expressions.
176
183
177
-
+ Azure Search persists content in an index. If your scenario is non-search-related, for example if your objective is analysis in another tool, you can delete the index that the pipeline creates. But you could also keep the index and use a built-in tool like [Search Explorer](search-explorer.md) as a third medium (behind Storage Explorer and your analytics app) for interacting with your content.
178
-
179
-
Along with document contents, enriched documents include the metadata of the skillset version that produced the enrichments.
184
+
+ Azure Search persists content in an index. If your scenario is non-search-related, for example if your objective is analysis in another tool, you can delete the index that the pipeline creates. But you could also keep the index and use a built-in tool like [Search Explorer](search-explorer.md) as a third medium (behind Storage Explorer and your analytics app) for interacting with your content.
180
185
181
186
## Inside a knowledge store
182
187
183
-
The knowledge store consists of an annotation cache and projections. The *cache* is used by the service internally to cache the results from skills and track changes. A *projection* defines the schema and structure of the enrichments that match your intended use. There is one cache per knowledge store, but multiple projections.
188
+
A *projection* defines the schema and structure of the enrichments that match your intended use. You can define multiple projections if you have applications that consume the data in different formats and shapes.
184
189
185
-
The cache is always a blob container, but projections can be articulated as tables or objects:
190
+
Projections can be articulated as objects or tables:
186
191
187
192
+ As an object, the projection maps to Blob storage, where the projection is saved to a container, within which are the objects or hierarchical representations in JSON for scenarios like a data science pipeline.
0 commit comments