pr_commits table is not ingested properly because of bookmarks #185
Description
Hello,
I just noticed missing data (commits) on the pr_commits table using this integration on Stitch.
The root cause I suspect for this is the incremental mode for this table and the use of bookmarks. Indeed, we fixed the lack of data by triggering a full re-sync of the integration data, but missing commits on this table continued to appear in the future.
Let's take this sample use case :
- Integration bookmark for this table and a given repo is set to :
"pr_commits": {"since": "T"} - Commit A is created at T-2
- Commit B is created a T-1
- Pull Request 1 is opened at T, and includes commits A and B
- Commit C is created at T+1 and associated to PR 1
When ingestion will be triggered again, it will fetch PR 1 and associate it Commit C, as it has been done after T, the current bookmark for the table, but Commits A & B won't be ingested by the integration.
A quick workaround, that won't be cost efficient but will allow to fix the issue is to set this table ingestion mode to full (or allow users to customize this behavior).
It would also be possible to set an offset in days, to allow the retrieval of olders commits in the pr_commits table.
This is a a blocker as it will happen a lot when you have a git workflow using feature branches merged into a develop branch and then picking commits to be merged into main.