[tiktok] do not exit account extraction early when we need to manually roll back the cursor #8968
+7
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…y roll back the cursor
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In very rare edge cases, it is possible for TikTok to return no items from a
creator/item_listrequest, whilst indicating that there are more items with thehasMorePrevious: trueflag.The
yt-dlpcode our extractor took inspiration from addressed this by manually rolling back the cursor. This is something we also carry out (seeTiktokLegacyTimeCursor.next_page()), but there was a fault in the logic withinTiktokItemListRequest.extract_items()that caused account extraction to exit early if it encountered no items, even ifhasMorePreviouswastrue.This PR addresses the fault by only exiting account extraction early if we encounter no items and TikTok indicates that no more posts are available. An item counter was added alongside the page logs to give users a better understanding of the extraction's progress.
It is worth noting that manually rolling back the cursor is something the TikTok website does not do, which causes affected accounts to have an empty or partial list of posts on their page, despite the user having posts uploaded to their account.