Data Subscription with max results

[iancrosssp]

Hi there,

I would like to create a Data Subscription that I can call from an API with a max number of results.

The Data Subscription would close and then be opened again the next time it's called. I need to be able to limit the number of results returned i.e. GetLatestResults(maxResults = 100)

I can see the batchSize on a subscription, but my understanding is that this controls how the data is returned from the database. I believe the database will keep sending batches until there are no more left to send.

Is there a way to tell the Data Subscription to only send a single batch?

I was thinking about throwing a SubscriptionClosed exception from inside the AfterAcknowledgement event, but I am not sure if there is a better way?

Many thanks,

Ian

[lastav5]

Hi Ian,

Can I have some more information about what you are trying to achieve? Is there a specific reason you need a subscription for this? maybe a query that takes 100 and skips each time could work for you.

You can use MaxDocsPerBatch to limit the number of items you get each time but note this means you can get anywhere between 1-100 items in each batch.
If you need exactly a 100 you could set MaxDocsPerBatch = 1, have an external counter and increase it inside AfterAcknowledgment.

worker.AfterAcknowledgment += batch =>
                  {
                       c += batch.Items.Count;
                      if (c == mydesiredvalue)
                      {
                          store.Subscriptions.DropConnection(worker.WorkerId);
                          worker.Dispose(false);
                          return Task.CompletedTask;
                      }
                     
                      return Task.CompletedTask;
                  };

Currently there is no way to ask the server to only send a single batch.

Another option is to get the current ChangeVector of the 100th document you are processing and set the subscription to it before connecting your worker in the next request.
This way, even if you are setting your batch size to a 100 and running the risk of the subscription advancing further, you will be able to force it to begin at a certain point.
See here: https://ravendb.net/docs/article-page/5.3/csharp/client-api/data-subscriptions/advanced-topics/maintenance-operations#updating-subscription

[iancrosssp]

Thanks - it needs to be a subscription rather than a query because I want to process the documents only once. I am subscribing to a stream of documents that are created at some point and do not need to be read again.

Anything up to 100 is fine. I think you have given me the answer I needed:

worker.AfterAcknowledgment += batch =>
{
store.Subscriptions.DropConnection(worker.WorkerId);
worker.Dispose(false);
return Task.CompletedTask;
};

I assume this will stop processing after one batch?

[ayende]

Can you explain your actual scenario a bit more?
I don't understand this.

You want to process a given number of results, and be sure that only happens once.
But you use a subscription for that?

What is the actual business need here?

I would say you are better off doing this with cluster-wide transactions, but can't really tell without knowing the context.

[iancrosssp]

Hi Oren,

• We have a system containing complex Quotes, Orders and Service Contracts (the document modelling here splits up these into pages of line items for scalability / performance and doesn't contain pricing). Think quotes with up to 100,000 items with separate consolidated pricing tables. A Quote can therefore be made up of many inter-related RavenDB documents.
• As changes are made by users to these business entities we need to allow an API caller to get a Financial Snapshot of the updated Quote. This is the current state of the quote combined with pricing and various other bits.
• A scheduled task (using a RavenDB subscription) identifies Quotes, Orders and Service Contracts changed in the last x minutes and creates a queue of Ids to be processed
• Another Data Subscription is listening for these and preparing the Financial Snapshots of these Quotes in the background
• An external process needs to call our API and say: 'give me updated Financial Snapshots'. The caller can set the frequency of how often they call us to get these snapshots. If they call often, there won't be many. If the call rarely, there would be more snapshots
• Once consumed, I guess the Financial Snapshots could be deleted in the same subscription batch because they are not needed again
• The caller then uses this information to update a Data Warehouse with the latest details of what's in the Quote, Order or Service Contracts

From an API perspective, I want the API caller to be able to specify the maximum number of snapshots to return (like a page size). We would impose a ceiling on this.

So when the API is called, we open the Data Subscription and ask if there are any Financial Snapshots to return
If there are any Financial Snapshots, we return them
We close the Data Subscription and wait until we get called again

Hope this helps clarify the scenario. Open to all ideas.

Cheers,

Ian

[ayende]

Is there a reason that you aren't just handling that with a query?
Basically, you already emits the documents you want. When the API is called, you read the N older docs that match and return them to the caller. Marking them as processed / deleted along the way.

[iancrosssp]

I guess an alternative and more traditional approach would be to force the caller to keep track of the Date / Time of the last document they received and query greater than that Date / Time and use streaming on the query.

[ayende]

If you already have queuing infrastructure, you can also use the new queuing integration, to simplify this process.