Skip to content

Commit ea010ae

Browse files
dereklegenzoffdereklegenzoff
committed
adding tutorial on optimizing indexing with the push api
1 parent 1c2cd5f commit ea010ae

File tree

1 file changed

+255
-0
lines changed

1 file changed

+255
-0
lines changed

‎articles/search/tutorial-optimize-indexing-pushapi.md

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
---
2+
title: 'C# Tutorial: Optimize Indexing with the Push API'
3+
titleSuffix: Azure Cognitive Search
4+
description: Learn how to efficiently index data using Azure Cognitive Search's Push API and an exponential backoff retry strategy. This tutorial and sample code are in C#.
5+
6+
manager: liamca
7+
author: dereklegenzoff
8+
ms.author: delegenz
9+
ms.service: cognitive-search
10+
ms.topic: tutorial
11+
ms.date: 04/20/2020
12+
---
13+
14+
# Tutorial: Optimize Indexing with the Push API
15+
16+
Azure Cognitive Search supports [two basic approaches](https://docs.microsoft.com/en-us/azure/search/search-what-is-data-import) for importing data into a search index: *pushing* your data into the index programmatically, or pointing an [Azure Cognitive Search indexer](https://docs.microsoft.com/en-us/azure/search/search-indexer-overview) at a supported data source to *pull* in the data.
17+
18+
This tutorial describes how to test batch processing speeds and efficiently index data using the [push model](https://docs.microsoft.com/en-us/azure/search/search-what-is-data-import#pushing-data-to-an-index). A .NET Core C# console application has been created so you can [download and run the application](https://github.com/Azure-Samples/azure-search-dotnet-samples/tree/master/optimize-data-indexing). This article explains the key aspects of the application as well as factors to consider when indexing data.
19+
20+
This tutorial uses C# and the [.NET SDK](https://aka.ms/search-sdk) to perform the following tasks:
21+
22+
> [!div class="checklist"]
23+
> * Create an index
24+
> * Test various batch sizes to determine the most efficient size
25+
> * Index data asynchronously
26+
> * Use multiple threads to increase indexing speeds
27+
> * Use an exponential backoff retry strategy to retry failed items
28+
29+
If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) before you begin.
30+
31+
## Prerequisites
32+
33+
The following services and tools are required for this quickstart.
34+
35+
+ [Visual Studio](https://visualstudio.microsoft.com/downloads/), any edition. Sample code and instructions were tested on the free Community edition.
36+
37+
+ [Create an Azure Cognitive Search service](search-create-service-portal.md) or [find an existing service](https://ms.portal.azure.com/#blade/HubsExtension/BrowseResourceBlade/resourceType/Microsoft.Search%2FsearchServices) under your current subscription. You can use a free service for this quickstart.
38+
39+
<a name="get-service-info"></a>
40+
41+
## Download files
42+
43+
Source code for this tutorial is in the [optimzize-data-indexing](https://github.com/Azure-Samples/azure-search-dotnet-samples/tree/master/optimize-data-indexing) folder in the [Azure-Samples/azure-search-dotnet-samples](https://github.com/Azure-Samples/azure-search-dotnet-samples) GitHub repository.
44+
45+
## 1 - Create Azure Cognitive Search service
46+
47+
To complete this tutorial, you'll need an Azure Cognitive Search service, which you can [create in the portal](search-create-service-portal.md). You can use the Free tier to complete this walkthrough, however, we recommend using the same tier you plan to use in production to accurately test and optimize data indexing speeds.
48+
49+
### Get an admin api-key and URL for Azure Cognitive Search
50+
51+
API calls require the service URL and an access key. A search service is created with both, so if you added Azure Cognitive Search to your subscription, follow these steps to get the necessary information:
52+
53+
1. [Sign in to the Azure portal](https://portal.azure.com/), and in your search service **Overview** page, get the URL. An example endpoint might look like `https://mydemo.search.windows.net`.
54+
55+
1. In **Settings** > **Keys**, get an admin key for full rights on the service. There are two interchangeable admin keys, provided for business continuity in case you need to roll one over. You can use either the primary or secondary key on requests for adding, modifying, and deleting objects.
56+
57+
![Get an HTTP endpoint and access key](media/search-get-started-postman/get-url-key.png "Get an HTTP endpoint and access key")
58+
59+
## 2 - Set up your environment
60+
61+
1. Start Visual Studio and open **OptimizeDataIndexing.sln**.
62+
1. In Solution Explorer, open **appsettings.json** to provide connection information.
63+
1. For `searchServiceName`, if the full URL is "https://my-demo-service.search.windows.net", the service name to provide is "my-demo-service".
64+
65+
```json
66+
{
67+
"SearchServiceName": "<YOUR-SEARCH-SERVICE-NAME>",
68+
"SearchServiceAdminApiKey": "<YOUR-ADMIN-API-KEY>",
69+
"SearchIndexName": "optimize-indexing"
70+
}
71+
```
72+
73+
## 3 - Explore the code
74+
75+
Once you update appsettings.json, the sample program in **OptimizeDataIndexing.sln** should be ready to build and run.
76+
77+
This code is derived from the [C# Quickstart](https://docs.microsoft.com/en-us/azure/search/search-get-started-dotnet). You can find more detailed information on creating indexes and the basics of working with the .NET SDK in that article.
78+
79+
This simple C#/.NET console app performs the following tasks:
80+
81+
* Creates a new index based on the data structure of the C# Hotel class (which also references the Address class).
82+
* Test various batch sizes to determine the most efficient size
83+
* Index data asynchronously
84+
* Use multiple threads to increase indexing speeds
85+
* Use an exponential backoff retry strategy to retry failed items
86+
87+
Before running the program, take a minute to study the code and the index definitions for this sample. The relevant code is in several files:
88+
89+
+ **Hotel.cs** and **Address.cs** contains the schema that defines the index
90+
+ **DataGenerator.cs** contains a simple class to make it easy to upload large amounts of hotel data
91+
+ **ExponentialBackoff.cs** contains code to optimize the indexing of data as described below
92+
+ **Program.cs** contains functions that create and delete the Azure Cognitive Search index, index batches of data, and test different batch sizes
93+
94+
### Creating the index
95+
96+
This sample program uses the .NET SDK to define and create an Azure Cognitive Search index. It takes advantage of the [FieldBuilder](https://docs.microsoft.com/dotnet/api/microsoft.azure.search.fieldbuilder) class to generate an index structure from a C# data model class.
97+
98+
The data model is defined by the Hotel class, which also contains references to the Address class. The FieldBuilder drills down through multiple class definitions to generate a complex data structure for the index. Metadata tags are used to define the attributes of each field, such as whether it is searchable or sortable.
99+
100+
The following snippets from the **Hotel.cs** file show how a single field, and a reference to another data model class, can be specified.
101+
102+
```csharp
103+
. . .
104+
[IsSearchable, IsSortable]
105+
public string HotelName { get; set; }
106+
. . .
107+
public Address Address { get; set; }
108+
. . .
109+
```
110+
111+
In the **Program.cs** file, the index is defined with a name and a field collection generated by the `FieldBuilder.BuildForType<Hotel>()` method, and then created as follows:
112+
113+
```csharp
114+
private static async Task CreateIndex(string indexName, SearchServiceClient searchService)
115+
{
116+
// Create a new search index structure that matches the properties of the Hotel class.
117+
// The Address class is referenced from the Hotel class. The FieldBuilder
118+
// will enumerate these to create a complex data structure for the index.
119+
var definition = new Index()
120+
{
121+
Name = indexName,
122+
Fields = FieldBuilder.BuildForType<Hotel>()
123+
};
124+
await searchService.Indexes.CreateAsync(definition);
125+
}
126+
```
127+
128+
### Generating data
129+
130+
A simple class is implemented in the **DataGenerator.cs** file to generate data for testing. The sole purpose of this class is to make it easy to generate a large number of documents with a unique id for indexing.
131+
132+
To get a list of 100,000 hotels with unique ids
133+
134+
```csharp
135+
DataGenerator dg = new DataGenerator();
136+
List<Hotel> hotels = dg.GetHotels(100000, "large");
137+
```
138+
139+
There are two sizes of hotels available for testing:
140+
1. `small` - 100,000 small hotels takes up ~ 12MB in the search index
141+
1. `large` - 100,000 large hotels takes up ~ 65MB in the search index
142+
143+
The schema of your index can have a significant impact on indexing speeds to after you run through this tutorial, it makes sense to convert this class to generate data matching your intended index schema.
144+
145+
## 4 - Test batch sizes
146+
147+
Indexing documents in batches will significantly improve indexing performance. Batches can be up to 1000 documents, or up to about 16 MB per batch.
148+
149+
Determining the optimal batch size for your data is a key component of optimizing indexing speeds. The following function demonstrates how to test different batch sizes.
150+
151+
```csharp
152+
public static async Task TestBatchSizes(ISearchIndexClient indexClient, int min = 100, int max = 1000, int step = 100, int numTries = 3)
153+
{
154+
DataGenerator dg = new DataGenerator();
155+
156+
Console.WriteLine("Batch Size \t Size in MB \t MB / Doc \t Time (ms) \t MB / Second");
157+
for (int numDocs = min; numDocs <= max; numDocs += step)
158+
{
159+
List<TimeSpan> durations = new List<TimeSpan>();
160+
double sizeInMb = 0.0;
161+
for (int x = 0; x < numTries; x++)
162+
{
163+
List<Hotel> hotels = dg.GetHotels(numDocs, "large");
164+
165+
DateTime startTime = DateTime.Now;
166+
await UploadDocuments(indexClient, hotels);
167+
DateTime endTime = DateTime.Now;
168+
durations.Add(endTime - startTime);
169+
170+
sizeInMb = EstimateObjectSize(hotels);
171+
}
172+
173+
var avgDuration = durations.Average(timeSpan => timeSpan.TotalMilliseconds);
174+
var avgDurationInSeconds = avgDuration / 1000;
175+
var mbPerSecond = sizeInMb / avgDurationInSeconds;
176+
177+
Console.WriteLine("{0} \t\t {1} \t\t {2} \t\t {3} \t {4}", numDocs, Math.Round(sizeInMb, 3), Math.Round(sizeInMb / numDocs, 3), Math.Round(avgDuration, 3), Math.Round(mbPerSecond, 3));
178+
179+
// Pausing 2 seconds to let the search service catch its breath
180+
Thread.Sleep(2000);
181+
}
182+
```
183+
184+
Because not all documents are the same size (although they are in this sample), we estimate the size of the data we're sending to the search service using the function below. The function converts the object to json and then converts the json to an array of bytes to determine its size:
185+
186+
```csharp
187+
public static double EstimateObjectSize(object data)
188+
{
189+
// converting data to json for more accurate sizing
190+
var json = JsonConvert.SerializeObject(data);
191+
192+
// converting object to byte[] to determine the size of the data
193+
BinaryFormatter bf = new BinaryFormatter();
194+
MemoryStream ms = new MemoryStream();
195+
byte[] Array;
196+
197+
bf.Serialize(ms, json);
198+
Array = ms.ToArray();
199+
200+
// converting from bytes to megabytes
201+
double sizeInMb = (double)Array.Length / 1000000;
202+
203+
return sizeInMb;
204+
}
205+
```
206+
207+
This allows us to determine which batch size is most efficient by MB/s.
208+
209+
## 5 - Index data
210+
211+
### Use multiple threads/workers
212+
213+
To take full advantage of Azure Cognitive Search's indexing speeds, you'll likely need to use multiple threads to send batch indexing requests concurrently to the service.
214+
215+
The optimal number of threads is determined by the tier of your search service, the size of your batches, and the schema of your index. You can modify this sample and test it with different thread counts to determine the optimal thread count for your scenario. In general, as long as you have at least a few threads running,
216+
217+
As you ramp up the requests hitting the search service, you may encounter [HTTP status codes](https://docs.microsoft.com/en-us/rest/api/searchservice/http-status-codes) indicating the request did not fully succeed. During indexing, two common HTTP status codes during indexing are:
218+
219+
* **503 Service Unavailable** - This error means that the system is under heavy load and your request can't be processed at this time.
220+
* **207 Multi-Status** - This error means that some documents succeeded, but at least one failed.
221+
222+
### Implement an exponential backoff retry strategy
223+
224+
In the event of a failure, requests should be retried using an [exponential backoff retry strategy](https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/implement-retries-exponential-backoff).
225+
226+
Azure Cognitive Search's .NET SDK automatically retries 503s and other failed requests but you'll need to implement your own logic to retry 207s. Open source tools such as [Polly](https://github.com/App-vNext/Polly) can also be used to implement a retry strategy. In this sample,
227+
228+
## 6 - Explore index
229+
230+
You can explore the populated search index after the program has run, using the [**Search explorer**](search-explorer.md) in the portal.
231+
232+
In Azure portal, open the search service **Overview** page, and find the **optimize-indexing** index in the **Indexes** list.
233+
234+
![List of Azure Cognitive Search indexes](media/tutorial-multiple-data-sources/index-list.png "List of Azure Cognitive Search indexes")
235+
236+
## Reset and rerun
237+
238+
In the early experimental stages of development, the most practical approach for design iteration is to delete the objects from Azure Cognitive Search and allow your code to rebuild them. Resource names are unique. Deleting an object lets you recreate it using the same name.
239+
240+
The sample code for this tutorial checks for existing indexes and deletes them so that you can rerun your code.
241+
242+
You can also use the portal to delete indexes.
243+
244+
## Clean up resources
245+
246+
When you're working in your own subscription, at the end of a project, it's a good idea to remove the resources that you no longer need. Resources left running can cost you money. You can delete resources individually or delete the resource group to delete the entire set of resources.
247+
248+
You can find and manage resources in the portal, using the **All resources** or **Resource groups** link in the left-navigation pane.
249+
250+
## Next steps
251+
252+
Now that you're familiar with the concept of ingesting data from multiple sources, let's take a closer look at indexer configuration, starting with Cosmos DB.
253+
254+
> [!div class="nextstepaction"]
255+
> [Configure an Azure Cosmos DB indexer](search-howto-index-cosmosdb.md)

0 commit comments

Comments
 (0)