-
Notifications
You must be signed in to change notification settings - Fork 1.3k
[Cosmos] cross partition continuation token #35511
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
topshot99
wants to merge
47
commits into
Azure:main
Choose a base branch
from
topshot99:feature/QueryContinuationToken
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 36 commits
Commits
Show all changes
47 commits
Select commit
Hold shift + click to select a range
a4b69f7
add continuation token
mkhandelwal-123 4722fcf
Refactor query execution context to handle response buffers consisten…
mkhandelwal-123 ee19dc5
Implement query control feature with composite continuation token and…
mkhandelwal-123 8d90660
Add ContinuationTokenManager for improved multi-partition query handl…
mkhandelwal-123 4402ac0
Merge branch 'main' of https://github.com/topshot99/azure-sdk-for-js …
mkhandelwal-123 b826254
Enhance ORDER BY query handling and continuation token management
mkhandelwal-123 865e0a2
add new tests for fetchBufferEndIndexForCurrentPage and ORDER BY quer…
mkhandelwal-123 a86b8a9
Add unit tests for _enableQueryControlFetchMoreImplementation
mkhandelwal-123 70f7482
Implement code changes to enhance functionality and improve performance
mkhandelwal-123 adb598e
Enhance continuation token management by adding offset and limit hand…
mkhandelwal-123 5586a14
Enhance continuation token management by adding offset and limit hand…
mkhandelwal-123 b1998f9
Add offset and limit support to OrderByQueryContinuationToken and upd…
mkhandelwal-123 c6d38ce
enhance OffsetLimitEndpointComponent to accept options for offset and…
mkhandelwal-123 0109bb6
add hashed last result support for distinct order queries
mkhandelwal-123 92eebc5
Enhance query execution context by adding continuation token manageme…
mkhandelwal-123 e1da4f8
Refactor filterPartitionRanges method by removing unused queryInfo pa…
mkhandelwal-123 3f6fac8
Add unit tests for ParallelQueryExecutionContextBase continuation tok…
mkhandelwal-123 b30f6a3
Refactor query execution components to remove buffer handling and imp…
mkhandelwal-123 d10d470
Enhance continuation token validation and handling in ParallelQueryRa…
mkhandelwal-123 c671f54
Refactor OrderByQueryRangeStrategy tests for improved clarity and cov…
mkhandelwal-123 b7af5a7
Refactor split merge usecase
mkhandelwal-123 1e8a44c
Refactor partition data patch mapping
mkhandelwal-123 96bc23f
Enhance partition key range comparison by adding secondary EPK sortin…
mkhandelwal-123 304b13a
Refactor document producer comparison and metadata clearing in query …
mkhandelwal-123 5cd0d9c
Refactor ContinuationTokenManager and related components for improved…
mkhandelwal-123 e80da8c
Refactor fetchMore method to improve readability by consolidating con…
mkhandelwal-123 da612a1
Refactor initialization of offset and limit values to streamline hand…
mkhandelwal-123 fea12e0
Refactor response handling in various components to streamline result…
mkhandelwal-123 a43970a
Refactor ContinuationTokenManager to encapsulate updateOffsetLimit me…
mkhandelwal-123 a295fbd
Add error handling for unsupported continuation tokens in cross-parti…
mkhandelwal-123 8111cd0
Add CompositeQueryContinuationToken class and update related imports
mkhandelwal-123 d5e80cd
Refactor OrderByQueryContinuationToken to an interface and update Con…
mkhandelwal-123 527d975
Refactor QueryRangeMapping interface documentation and remove unused …
mkhandelwal-123 d5a24b8
Refactor OffsetLimitEndpointComponent to improve continuation token h…
mkhandelwal-123 23d1ada
Refactor continuation token handling in Cosmos DB SDK
mkhandelwal-123 db7cae5
Refactor offset/limit calculation in PartitionRangeManager to improve…
mkhandelwal-123 faed188
Enhance continuation token management in pipelined query execution co…
mkhandelwal-123 5d21731
Refactor continuation token handling in query execution context to im…
mkhandelwal-123 5d4664b
Refactor query execution context to support range-token pair management
mkhandelwal-123 6c75575
Refactor query execution context to support parallel query results
mkhandelwal-123 8ea6701
Refactor endpoint components to support parallel query results and im…
mkhandelwal-123 9c5feea
Add orderByItems support to parallel query results and related compon…
mkhandelwal-123 371f3d2
Refactor continuation token management for improved query execution
mkhandelwal-123 5820d6e
Remove error throwing for insufficient orderByItemsArray length durin…
mkhandelwal-123 945bfe9
Refactor query execution components to support new ParallelQueryResul…
mkhandelwal-123 c2c45c1
Refactor targetPartitionKeyRangeDocProdComparator for improved compar…
mkhandelwal-123 6a79c4e
Add PartitionRangeUpdate interface and update ContinuationTokenManage…
mkhandelwal-123 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
84 changes: 84 additions & 0 deletions
84
sdk/cosmosdb/cosmos/src/documents/ContinuationToken/OrderByQueryContinuationToken.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,84 @@ | ||
// Copyright (c) Microsoft Corporation. | ||
// Licensed under the MIT License. | ||
|
||
/** | ||
* Continuation token for order by queries. | ||
* @internal | ||
*/ | ||
export interface OrderByQueryContinuationToken { | ||
/** | ||
* Composite token for the query continuation | ||
*/ | ||
compositeToken: string; | ||
topshot99 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
/** | ||
* Order by items for the query | ||
*/ | ||
orderByItems: any[]; | ||
|
||
/** | ||
* Resource ID of the container for which the continuation token is issued | ||
*/ | ||
rid: string; | ||
|
||
/** | ||
* Number of items to skip in the query | ||
*/ | ||
skipCount: number; | ||
|
||
/** | ||
* Current offset value for OFFSET/LIMIT queries | ||
*/ | ||
offset?: number; | ||
|
||
/** | ||
* Current limit value for OFFSET/LIMIT queries | ||
*/ | ||
limit?: number; | ||
|
||
/** | ||
* Hash of the last document result for distinct order queries | ||
* Used to ensure duplicates are not returned across continuation boundaries | ||
*/ | ||
hashedLastResult?: string; | ||
} | ||
|
||
/** | ||
* Creates an OrderByQueryContinuationToken | ||
* @internal | ||
*/ | ||
export function createOrderByQueryContinuationToken( | ||
compositeToken: string, | ||
orderByItems: any[], | ||
rid: string, | ||
skipCount: number, | ||
offset?: number, | ||
limit?: number, | ||
hashedLastResult?: string | ||
): OrderByQueryContinuationToken { | ||
return { | ||
compositeToken, | ||
orderByItems, | ||
rid, | ||
skipCount, | ||
offset, | ||
limit, | ||
hashedLastResult, | ||
}; | ||
} | ||
|
||
/** | ||
* Serializes an OrderByQueryContinuationToken to a JSON string | ||
* @internal | ||
*/ | ||
export function serializeOrderByQueryContinuationToken(token: OrderByQueryContinuationToken): string { | ||
return JSON.stringify(token); | ||
} | ||
|
||
/** | ||
* Deserializes a JSON string to an OrderByQueryContinuationToken | ||
* @internal | ||
*/ | ||
export function parseOrderByQueryContinuationToken(tokenString: string): OrderByQueryContinuationToken { | ||
return JSON.parse(tokenString); | ||
} |
85 changes: 85 additions & 0 deletions
85
sdk/cosmosdb/cosmos/src/queryExecutionContext/CompositeQueryContinuationToken.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
// Copyright (c) Microsoft Corporation. | ||
// Licensed under the MIT License. | ||
|
||
import type { QueryRangeMapping } from "./QueryRangeMapping.js"; | ||
topshot99 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
/** | ||
* @hidden | ||
* Composite continuation token for parallel query execution across multiple partition ranges | ||
*/ | ||
export interface CompositeQueryContinuationToken { | ||
/** | ||
* Resource ID of the container for which the continuation token is issued | ||
*/ | ||
readonly rid: string; | ||
|
||
/** | ||
* List of query range mappings part of the continuation token | ||
*/ | ||
rangeMappings: QueryRangeMapping[]; | ||
|
||
/** | ||
* Current offset value for OFFSET/LIMIT queries | ||
*/ | ||
offset?: number; | ||
|
||
/** | ||
* Current limit value for OFFSET/LIMIT queries | ||
*/ | ||
limit?: number; | ||
|
||
} | ||
|
||
/** | ||
* Creates a new CompositeQueryContinuationToken | ||
* @hidden | ||
*/ | ||
export function createCompositeQueryContinuationToken( | ||
rid: string, | ||
rangeMappings: QueryRangeMapping[], | ||
offset?: number, | ||
limit?: number | ||
): CompositeQueryContinuationToken { | ||
return { | ||
rid, | ||
rangeMappings, | ||
offset, | ||
limit, | ||
}; | ||
} | ||
|
||
/** | ||
* Adds a range mapping to the continuation token | ||
* @hidden | ||
*/ | ||
export function addRangeMappingToCompositeToken(token: CompositeQueryContinuationToken, rangeMapping: QueryRangeMapping): void { | ||
token.rangeMappings.push(rangeMapping); | ||
} | ||
|
||
/** | ||
* Serializes the composite continuation token to a JSON string | ||
* @hidden | ||
*/ | ||
export function compositeTokenToString(token: CompositeQueryContinuationToken): string { | ||
topshot99 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
return JSON.stringify({ | ||
topshot99 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
rid: token.rid, | ||
rangeMappings: token.rangeMappings, | ||
offset: token.offset, | ||
limit: token.limit, | ||
}); | ||
} | ||
|
||
/** | ||
* Deserializes a JSON string to a CompositeQueryContinuationToken | ||
* @hidden | ||
*/ | ||
export function compositeTokenFromString(tokenString: string): CompositeQueryContinuationToken { | ||
const parsed = JSON.parse(tokenString); | ||
return createCompositeQueryContinuationToken( | ||
parsed.rid, | ||
parsed.rangeMappings, | ||
parsed.globalContinuationToken, | ||
parsed.offset, | ||
parsed.limit, | ||
); | ||
} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add logger wherever needed.