You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/cosmos-db/change-feed-pull-model.md
+53-50Lines changed: 53 additions & 50 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,24 +12,16 @@ ms.reviewer: sngun
12
12
13
13
# Change feed pull model in Azure Cosmos DB
14
14
15
-
The change feed pull model is part of the [Azure Cosmos DB SDK V3](https://github.com/Azure/azure-cosmos-dotnet-v3). You can use the change feed pull model to parallelize processing of changes across multiple change feed consumers.
15
+
With the change feed pull model, you can consume the Azure Cosmos DB change feed at your own pace. As you can already do with the [change feed processor](change-feed-processor.md), you can use the change feed pull model to parallelize the processing of changes across multiple change feed consumers.
16
16
17
17
> [!NOTE]
18
18
> The change feed pull model is currently in [preview in the Azure Cosmos DB .NET SDK](https://www.nuget.org/packages/Microsoft.Azure.Cosmos/3.9.0-preview) only. The preview is not yet available for other SDK versions.
19
19
20
-
## Using FeedRange for parallelization
21
-
22
-
In the change feed pull model, you can use the `FeedRange` to parallelize the processing of the change feed. A `FeedRange` represents a single [physical partition](partition-data.md#physical-partitions).
23
-
24
-
Here's an example showing how to obtain a list of ranges for your container.
Using a `FeedRange`, you can then create a `FeedIterator` to parallelize the processing of change feed across multiple machines or threads. When you initially obtain a `FeedIterator`, you can specify an optional `StartTime` within the `ChangeFeedRequestOptions`. When left unspecified, the `StartTime` will be the current time.
22
+
You can crete a `FeedIterator` to process the change feed using the pull model. When you initially create a `FeedIterator`, you can specify an optional `StartTime` within the `ChangeFeedRequestOptions`. When left unspecified, the `StartTime` will be the current time.
31
23
32
-
The `FeedIterator` comes in two flavors. In addition to the examples below that return entity objects, you can also obtain the response with `Stream` support.
24
+
The `FeedIterator` comes in two flavors. In addition to the examples below that return entity objects, you can also obtain the response with `Stream` support. Streams allow you to read data without having it first deserialized, saving on client resources.
33
25
34
26
Here's an example for obtaining a `FeedIterator` that returns entity objects, in this case a `User` object:
35
27
@@ -43,15 +35,14 @@ Here's an example for obtaining a `FeedIterator` that returns a `Stream`:
Here's a sample that shows reading an example `User` object from the beginning of the container's change feed using two hypothetical separate machines that are reading in parallel:
47
-
48
-
Machine 1:
38
+
Using a `FeedIterator`, you can easily process an entire container's change feed at your own pace. Here's an example:
@@ -60,13 +51,16 @@ while (iteratorA.HasMoreResults)
60
51
}
61
52
```
62
53
63
-
Machine 2:
54
+
## Consuming a partition key's changes
55
+
56
+
In some cases, you may only want to process a specific partition key's changes. You can obtain a `FeedIterator` for a specific partition key and process the changes the same way that you can for an entire container:
@@ -75,40 +69,47 @@ while (iteratorB.HasMoreResults)
75
69
}
76
70
```
77
71
78
-
## Saving continuation tokens
72
+
## Using FeedRange for parallelization
79
73
80
-
You can save the position of your `FeedIterator` by creating a continuation token. A continuation token is a string value that keeps of track of your FeedIterator's last processed changes. This allows the `FeedIterator` to resume at this point later. The following code will read through the change feed since container creation. After no more changes are available, it will persist a continuation token so that change feed consumption can be later resumed.
74
+
In [change feed processor](change-feed-processor.md), change feed processing is automatically spread across multiple consumers. In the change feed pull model, you can use the `FeedRange` to parallelize the processing of the change feed. A `FeedRange` represents a single [physical partition](partition-data.md#physical-partitions).
75
+
76
+
Here's an example showing how to obtain a list of ranges for your container.
Using a `FeedRange`, you can then create a `FeedIterator` to parallelize the processing of change feed across multiple machines or threads. Unlike the previous example that showed how to obtain a single `FeedIterator` for the entire container, you can use the `FeedRange` to obtain multiple FeedIterators which can process the change feed in parallel.
86
83
87
-
while (iterator.HasMoreResults)
84
+
In the case where you want to use FeedRanges, you need to have an orchestrator process that obtains FeedRanges and distributes them to those machines. This distribution could be:
85
+
86
+
* Using `FeedRange.ToJsonString` and storing/distributing this string value. The consumers can use this value with `FeedRange.FromJsonString`
87
+
* If the distribution is in-process, passing the `FeedRange` object reference.
88
+
89
+
Here's a sample that shows how to read from the beginning of the container's change feed using two hypothetical separate machines that are reading in parallel:
Sometimes you might not need any parallelization when reading the change feed. By creating a `FeedIterator` without any `FeedRange` input, you can read an entire container's change feed on one machine:
@@ -117,27 +118,29 @@ while (iteratorForTheEntireContainer.HasMoreResults)
117
118
}
118
119
```
119
120
120
-
If you need to stop and resume reading from the entire container's change feed, you can obtain a continuation token from the `FeedIterator`, just as you can for a `FeedRange`.
121
-
122
-
## Consuming a partition key's changes
121
+
## Saving continuation tokens
123
122
124
-
In some cases, you may only want to process a specific partition key's changes. You can obtain a `FeedIterator`for a specific partition key.
123
+
You can save the position of your `FeedIterator` by creating a continuation token. A continuation token is a string value that keeps of track of your FeedIterator's last processed changes. This allows the `FeedIterator`to resume at this point later. The following code will read through the change feed since container creation. After no more changes are available, it will persist a continuation token so that change feed consumption can be later resumed.
Console.WriteLine($"Detected change for user with id {user.id}");
136
138
}
137
139
}
138
-
```
139
140
140
-
If you need to stop and resume reading from the change feed for a specific partition key, you can obtain a continuation token from the `FeedIterator`, just as you can for a `FeedRange`.
0 commit comments