Can not run action update filter list

[nightmare199x]

Hi, i have error and can not update filter after sync folk. Please help!

Run npm run cloudflare-create

cloudflare-create
npm run cloudflare-create:list && npm run cloudflare-create:rule

cloudflare-create:list
node cf_list_create.js

Processing allowlist.txt
Processing blocklist.txt

Number of processed domains: 687961
Number of duplicate domains: 347126
Number of unnecessary domains: 113312
Number of allowed domains: 768
Number of blocked domains: 226755
Number of lists to be created: 227

Creating 227 lists for 226755 domains...
Error occurred while creating lists - TypeError: Cannot use 'in' operator to search for 'errors' in undefined
file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/helpers.js:129
throw new Error(HTTP error! Status: ${response.status} - ${ 'errors' in data ? data.errors[0].message : JSON.stringify(data) });
^

TypeError: Cannot use 'in' operator to search for 'errors' in undefined
at request (file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/helpers.js:129:79)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Promise.all (index 172)
at async createZeroTrustListsAtOnce (file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/api.js:77:5)
at async file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/cf_list_create.js:139:5

Node.js v20.14.0
Error: Process completed with exit code 1.

[arunsathiya]

I saw the same issue after a5d3fdb, but the overall underlying issue remains this: #111.

[galpt]

@arunsathiya
What about using the previous 3-second sleep (the example I wrote there #111)? I guess it'll work for you.

If you're getting rate limited, the best practice is to wait, not to keep sending more requests until they accept it.
If they're detecting more requests coming from you after they placed the rate limit on you, and you're not slowing down, you're just giving them more reasons to keep the rate limit even longer.

They offer DDoS protection as their main product, and one of the core features is to prevent non-human requests from reaching their servers faster than what their servers can actually handle. This usually includes blocking requests until the sender of the requests is slowing everything down.

What I've noticed from the latest PR is that it's not slowing down until you're getting rate limited, which is already too late.
You don't want to trigger the rate limit in the first place, so adding sleep() between requests should be the correct logic.

[galpt]

Someone might want to experiment with a logic where, let's say we're sending 3 requests to their API, and then sleep for 3 seconds, and sending 3 more requests again, and sleep again for 3 seconds.

So that way you're not waiting 3 seconds for every request, but for every 3 requests, which is probably faster and won't trigger the rate limit at all.

The above example is what I could think of right now.
3-sec sleep for every request feels slow but it's a guaranteed success, but there should be a better logic.

[erik-de-bont]

The 3 seconds solution (#111) works fine for me, and I think it's a better approach than the current implemented workaround (keep sending requests until it continuous). I prefer stability over speed in this case. It's updating at 03:00 AM automatically so I don't mind if it takes 30 minutes or longer, since I am not waiting for it.

[arunsathiya]

Confirm, the delayed retry approach (#111 (comment), a3ade64) works well with fast mode disabled.