-
Notifications
You must be signed in to change notification settings - Fork 37
Open
Description
Hey,
I find myself writing often something like this:
FROM read_json([
'http://example.com/data.json',
'http://non-existent.test/data.json'
]);or
FROM read_json(getvariable('resource_urls'));
or
FROM read_text(getvariable('resource_urls'));
If there are hundreds of URLs every now and then some of the resources will returns errors like these:
Could not establish connection error for HTTP HEAD to 'http://non-existent.test/data.json'
Unable to connect to URL "http://example.com/data.json": 404 (Not Found).
In this situation all other results will go to waste if 1 url from 300 urls will fail.
There are already excellent configs I can use to increase the retry count and wait times:
SET http_retries = 10;
SET http_retry_wait_ms = 2000;
Sometimes it's not enough to retry and the resource is not there for a hour or a day or for a week or for a month but will eventually appear again.
Would there be a way to allow missing or failing resource in the larger arrays and still be able to load all other files?
Metadata
Metadata
Assignees
Labels
No labels