Update docs for incremental update / change capturing to clarify things. #268
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
badmonster0
commented
Apr 5, 2025
| * Data sources provides a **change stream**. | ||
| * Configured with a [refresh interval](flow_def#refresh-interval), which is generally applicable to all data sources. | ||
| * Specific data sources also provide their specific change capture mechanisms. | ||
| See documentations for specific data sources for details. |
Member
Author
There was a problem hiding this comment.
can there be an example
badmonster0
commented
Apr 5, 2025
| * Data sources configured with a [refresh interval](flow_def#refresh-interval). | ||
| * Data sources provides a **change stream**. | ||
| * Configured with a [refresh interval](flow_def#refresh-interval), which is generally applicable to all data sources. | ||
| * Specific data sources also provide their specific change capture mechanisms. |
Member
Author
There was a problem hiding this comment.
does push events belong here as well
Member
Author
There was a problem hiding this comment.
Different sources have different channels to push events, so it belongs to specific data sources.
We can add more words to clarify here after we start to support a real push events based change capture mechanism.
badmonster0
commented
Apr 5, 2025
| For data sources ineligible for live updates, or when the `live_mode` is `False`, | ||
| the `FlowLiveUpdater` only performs a one-time update, i.e. similar to the one-time update (`update()` method) above, | ||
| under a unified interface. | ||
| Note that `cocoindex.FlowLiveUpdater` provides a unified interface for both one-time update and live update. |
Member
Author
There was a problem hiding this comment.
does it make sense to link?
Member
Author
There was a problem hiding this comment.
Which part do you refer to? FlowLinkUpdater, or one-time update / live update?
badmonster0
commented
Apr 5, 2025
| So when a source is configured with a change stream, it's still recommended to set a `refresh_interval`, with a larger value. | ||
| So for most changes can be covered by the change stream (with low latency), and remaining changes (files no longer exist or accessible) will still be covered (with a higher latency). | ||
| So when a `GoogleDrive` source enabled `recent_changes_poll_interval`, it's still recommended to set a `refresh_interval`, with a larger value. | ||
| So that most changes can be covered by polling recent changes (with low latency), and remaining changes (files no longer exist or accessible) will still be covered (with a higher latency). |
Member
Author
There was a problem hiding this comment.
wonder if make sense to give an example, e.g., every 2 hours, based on your requirement.
badmonster0
added a commit
that referenced
this pull request
Apr 9, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.