Application stops downloading images in tagged blog scrape after reaching post dates from up to 5 years prior to current date

[Abukes01]

Describe the bug

Hello, I'm having a bit of an unexpected problem with the app. I'm trying to archive a tag (https://www.tumblr.com/tagged/asriel), but the app doesn't download anything from back before and up to ~5.5 years prior to current date.

I have done it once, but lost the data due to hard drive failure. I know for a fact that the images and videos from further in the past are still available on the site, but the app stops downloading anything past 20180224 (YYYYMMDD) [At time of posting].

I have tried to force From-To download in the settings but it doesn't do anything other than re-scan and stop. I'm using default factory settings with only 'Group photo sets' enabled and %d_%o_%i_%s_%x filename template.

Steps to reproduce the behavior:

1. Add https://www.tumblr.com/tagged/asriel to blog list and queue, no need for the link to contain the '?sort=recent' query as it's default.
2. Start the download and wait for it to finish (the tag goes all the way back to late 2015, lots of images)
3. Go to the folder the scrape results are saved to, sort by 'Date Modified' ascending
4. The furthest in the past image does not go past 201802XX datecode while it could go up until 2015/2016 (known from previous scrape)

Expected behavior

I expected the application to download everything tagged with 'asriel' as far back as late 2015 or early 2016, but keep getting stuck on early 2018.

Desktop:

• TumblThree version: 2.11.0.413
• OS: Windows 11
• Browser (pasted from settings): Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36

Any possible reasons to this behavior or fixes are welcome if available, I can test various settings and test-download, please and thank you!

[thomas694]

Thanks for reporting a possible bug. We'll take a look at it.

[Abukes01]

Thanks for reporting a possible bug. We'll take a look at it.

Any progress or updates?

[thomas694]

I had a look on it. The app behaves fine until the API stops returning entries. It's unclear why it stops doing it.
As their search function is simple, I cannot find a tag that existed for sure before these years.

Does someone know another tag that certainly existed in the years before 2018? Maybe you either downloaded the tag with the tag search or you downloaded a blog with old posts that have tags assigned.

[Abukes01]

I had a look on it. The app behaves fine until the API stops returning entries. It's unclear why it stops doing it.
As their search function is simple, I cannot find a tag that existed for sure before these years.

Does someone know another tag that certainly existed in the years before 2018? Maybe you either downloaded the tag with the tag search or you downloaded a blog with old posts that have tags assigned.

I know FNAF (#fnaf or #five nights at freddy[']s) must have existed way back before 2018 and is still active. Will give it a go when I'm back at my computer in a day or two since now.

[Abukes01]

Also, figured I'd mention that I have tried both the API and SVC presets with the Override Tumblr crawler detection setting, both resulted in the same behavior as reported.

I'm also gonna try to get permission to sample-download one of the artists I know created before 2018 (I'm in a discord server for several games and we have some artists there that have Tumblr accounts and active blogs and are pretty chill)

If for some reason the API fails to recognize their entries from before 2018 I'll let you know and maybe also try to get a hold of the Tumblr devs in regards to this happening, maybe they may have a couple of ideas as to what may be the reason for this behavior.

[Abukes01]

I have checked the #fnaf https://www.tumblr.com/tagged/fnaf tag. The same thing happens, AND it missed a lot of posts, jumps from 2018 straight to 2021 and after 8 posts jumps to 2022 and continues forth from there. This tag is massively bigger (The app downloaded 41GB of files with 78 269 downloaded items) and as I have said, I know for a fact this tag existed way before 2018.

[Abukes01]

@thomas694

I have tested the standard blog-oriented downloads, it works as it should and downloads the entire blog, no exceptions, but the reported tag problem still remains unchanged.

Do you think the API might be reaching the rate limits and stopping traffic after a certain point? Seems kinda impossible for this to occur, as a couple months ago it worked fine and downloaded the entire tags, and afaik Tumblr didn't change the rates recently.

[thomas694]

It ran several hours without hitting any rate limits collecting all the available links, but with the same result. And there's nothing in common except it stops earlier than expected.
The tag is often used. This time it was 24764 requests resp. 224714 post URLs and stopping on Aug 26th 2021, the last time it was around 2.5k requests resp. 18k post URLs. So I guess rate limit is not the reason.

[Abukes01]

Hey, I don't know if anything got figured out on your side, but I managed to persuade one of my friends to give me temporary access to the API, as well as did some testing on other downloaders. This seems to be a radical sudden change in the API itself, as none of the downloaders I tested (TumblrRipper, Extreme Picture Downloader 3, WFDownloader) as well as a simple python scraper script stopped at the exact same point I mentioned at the beginning of the issue.

[thomas694]

Thanks for testing further and letting us know the results.
So I guess there's nothing we can do about it.