Kemono downloaders failing. (403, possibly DDoS Guard)

[jotarokujosan5-byte]

Since the change to the new TLD, the downloader for Kemono seems to've broken. Looks like the DDoS Guard config is too aggressive. Mentioning this in their chan was brushed off with "File an issue on git and provide debugging info so I can tell you what you are doing wrong", which is unsurprising.

[major-minor-patch]

Yes, I noticed this as well. Nothing but 403 in Hydrus. Additionally, I am seeing the DDOS-Guard blocked message in JDownloader when attempting to download anything from Kemono. Unfortunately, like you said, the issue is may be that it's set too aggressive on their end.

[wendy-carter]

I think they set up some new kind of CDN that the parser doesn't know how to handle, there's older file URLs that download fine but any of these new "n1, n2, n3, n4" URLs 403, even when you send cookies with Hydrus Companion.

Edit: Seems to be an issue with North American VPN servers, as the site flat out refuses to load in US or Canadian VPN connections. Switching to EU ones both cause the site to work again and Hydrus to download files again.

[jotarokujosan5-byte]

API endpoints now need updating to new TLD

[Wyrrrd]

Changing to the new TLD isn't enough.

One also needs to:

• add a new header Accept: text/css
• work around them dropping the /api/v1/{service}/user/{id} endpoint for artist/creator lookups

[bennyz327]

Is there any progress on this issue? Does anyone have a working new version of kemono downloader?

[Wyrrrd]

I am close, just missing one little thing.

The new user gallery API endpoint /api/v1/{service}/user/{id}/posts accepts pagination parameters ?o=50, but it 404s for ?o=0. It expects you to omit if 0, and I don't know how to address this in the hydrus url classes...

[LonelyBard]

Any Updates on the downloader?

[jav-lin]

Any Updates on the downloader?

I'm a different person but I think I got it working? I can try to create an import image but I'm not sure if the way I did it was optimal.

First thing, is I was getting a 403 error when trying to access the Kemono API for any reason. It gives the message:

fetch failed: 403: If you want to scrape, use "Accept: text/css" header in your requests for now. For whatever reason DDG does not like SPA and JSON, so we have to be funny. And you are no exception to caching.

If you want to scrape, use "Accept: text/css" header in your requests for now. For whatever reason DDG does not like SPA and JSON, so we have to be funny. And you are no exception to caching.

This was fixed by adding this item under network > data > manage http headers...

After that I created a duplicate url class from kemono.cr user gallery page api. The first of them has no parameters and also checks options > disallow match on any extra parameters. It also does not have anything under next gallery page.

The second has a parameter for o and has no default value. This makes it require a page value. This one does have next gallery page with a value of 50.

Then I created a duplicate parser for each of them respectively. You have to make sure to set the url class equal to the respective parser under network > downloader components > manage url class links.

Then I had the first parser create a subsidiary page parser that just takes random text and forcefully converts it into the proper link with ?o=50 and adds it as a gallery to pursue.

I can try to make the import image and clean up some of the text as long as no one notices anything wrong or in bad practice with method.

[Wyrrrd]

Thank you for figuring this out. I think it is the best version we can currently manage, but to have it clean we'd need a feature request to hydrusnetwork/hydrus to support omitting a parameter if it is zero (or ideally any set value, since maybe someone decides to start on o=1 and 404 on it on some other site in the future).

[jav-lin]

Thank you for figuring this out. I think it is the best version we can currently manage, but to have it clean we'd need a feature request to hydrusnetwork/hydrus to support omitting a parameter if it is zero (or ideally any set value, since maybe someone decides to start on o=1 and 404 on it on some other site in the future).

Thanks! It's my first contribution. I've had a bunch of janky solutions before, like trying to paginate Kemono by using like 100 instances of "replace the first X with 1... replace the next X with 2" lol. I can omit that from any uploads lol.

I haven't done anything before here; if anyone wants it what's the best way to upload the import image? Would I just upload it locally to my own files, paste it here, or would I want to do a pull request? Been a long time since I studied git.

I also don't remember, but the current Kemono downloader on this repository might not have been updated to the new API format? It used to be

User Page
api/v1/{service}/user/{user_id}
Posts
api/v1/{service}/user/{user_id}/post/{post_id}

But now it's

User Page
api/v1/{service}/user/{user_id}/posts
Page (unchanged)
api/v1/{service}/user/{user_id}/post/{post_id}

I obviously changed it on my side but I don't remember if that is current in the repository.

[mutag3ne]

Then I created a duplicate parser for each of them respectively. You have to make sure to set the url class equal to the respective parser under network > downloader components > manage url class links.

Then I had the first parser create a subsidiary page parser that just takes random text and forcefully converts it into the proper link with ?o=50 and adds it as a gallery to pursue.

I got lost here. I'm not intimately familiar with how all of the downloader components work and interact. I spent half a day trying to fix the Fanbox downloader a couple of months ago, only to get roadblocked, with nothing to help online, and no response to my post on the support thread. Everything looked like it should have worked, but it didn't.

I guess I should probably just try to figure it out myself anyway.

[jav-lin]

I got lost here. I'm not intimately familiar with how all of the downloader components work and interact. I spent half a day trying to fix the Fanbox downloader a couple of months ago, only to get roadblocked, with nothing to help online, and no response to my post on the support thread. Everything looked like it should have worked, but it didn't.

It's pretty confusing, and the different parts aren't organized on the GUI in order of how they logically function.

The simple version, is each url class has to be manually assigned a specific parser to use for extracting information. That is done under network > downloader components > manage url class links.

Because you're creating 2 new separate url classes, you need to make sure that each of them points to the 2 new parsers respectively.

The left-side url class objects are what you create under network > downloader components > manage url classes.
The left-side parser objects are what you create under network > downloader components > manage parsers.

That part is pretty easy if you've made the components correctly, so I'm guessing the issue probably lies within the url classes and parsers. The 2 parsers are functionally 99% similar. The main difference is how they each try to get the next page.

• The 1st url class (o=0) does not contain the "o" parameter. It does not try to find a next page. The class treats this url as a one-and-done parse.

• The 1st parser (o=0) has an extra item under subsidiary page parsers

This is the main funky part of the fix. You are trying to manually construct the new url by pretending to scrape new information. Within the subsidiary page parsers under content parsers you create a new object for a url. Normally you scrape the page looking for a url that's already within the html/json data. But this time we know exactly what url we want; just adding the literal string "?o=50" to the existing url.

So what we do is, we create that content parser and set content type to gallery parsers only: next gallery page. This takes the parsed data and appends it to the url. For the parsed data, we aren't actually parsing for anything, so you can put anything you want under the main parsing formula.

All the work is done under regex substitution. We add a String Converter item under processing steps, and then within that item we add a regex substitution. The "^.*$" expression captures everything and replaces it with "?o=50". This is then added to the end of the original url and sends this back to be evaluated as a new url class that should trigger the 2nd url class and parser.

• The 2nd url class/parser (o=50) functions exactly like most normal url classes/parsers do. The only difference is for the url class, the default value for the "o" parameter part is set to 50 instead of the typical 0.

I guess I should probably just try to figure it out myself anyway.

I'm trying to clean up a package and I can upload a downloader image. I have a lot of custom modifications to mine so I have to try and revert the parsers to a more generic state lol.

Behold my pagination in all it's glory.

[Wyrrrd]

Opened a feature request on hydrus for this.

hydrusnetwork/hydrus#1906

[mutag3ne]

I'm trying to clean up a package and I can upload a downloader image. I have a lot of custom modifications to mine so I have to try and revert the parsers to a more generic state lol.

I actually screwed up even more and earlier than I had thought, and probably butchered at least one of the classes and/or parsers. I'll just wait for your upload, if possible, not like I desperately need this or anything. And, thank you for your work regardless, and for doing your best to try to help my peanut brain.

[oppenHIMARS]

Was this ever opened as a PR?