Skip to content

Make crawler retry mechanism configurable #355

New issue
New issue
Open
Feature
#361
Open
Make crawler retry mechanism configurable#355
Feature
#361
Assignees
ndr-brt
Labels
enhancementNew feature or request
@ndr-brt

Description

@ndr-brt
ndr-brt
opened on Sep 4, 2025

Feature Request

At the moment the number of retries (5) and the policy (5 seconds + a random number between 0 and 20) is used

FederatedCatalog/core/crawler-core/src/main/java/org/eclipse/edc/catalog/cache/ExecutionManager.java

Lines 114 to 122 in 91f00fa

monitor.severe("Unexpected exception occurred while crawling: " + item.getId(), throwable);
if (item.getErrors().size() > 5) {
monitor.severe(format("The following WorkItem has errored out more than 5 times. We'll discard it now: [%s]", item));
} else {
var random = new Random();
var delaySeconds = 5 + random.nextInt(20);
monitor.debug(format("The following work item has errored out. Will re-queue after a delay of %s seconds: [%s]", delaySeconds, item));
crawlers.schedule(createCrawler(item), delaySeconds, TimeUnit.SECONDS);
}

It should be configurable, and the random generation should be avoided

Which Areas Would Be Affected?

execution manager

Why Is the Feature Desired?

configurability

Who will sponsor this feature?

Please @-mention the committer that will sponsor your feature.

Solution Proposal

If possible, provide a (brief!) solution proposal.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

Feature

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions