Introduce custom semantics for message delivery retries

[ayeshLK]

Current Limitation

Currently, WebSubHub message delivery retries are tightly coupled with HTTP client retries. Once the HTTP retries are exhausted, the subscription is marked as stale and message delivery is paused until the subscriber resubscribes.

In an enterprise environment, however, resubscription may not be feasible when the number of subscribed applications grows, especially since we do not have a UI-based mechanism to easily identify stale subscribers.

[ayeshLK]

[Update on 4th March 2026]

As of now, WebSubHub message delivery retries are handled only at the transport level via the underlying HTTP client retry configuration. Once those retries are exhausted, message delivery to the subscriber stops and the subscriber is marked as stale. Delivery resumes only when the subscriber explicitly re-subscribes, continuing from the last processed point.

In an enterprise scenario, where the number of subscribers can be significantly high, managing re-subscriptions for all stale subscribers becomes operationally challenging.

To address this, it would be beneficial to introduce an application-level retry mechanism with configurable on-failure actions, in addition to the existing transport-level retries. This would provide better resilience and operational flexibility beyond what is currently supported at the HTTP layer.

The two retry layers operate independently and in sequence for each delivery attempt:

Message received from store
         │
         ▼
┌─────────────────────────────────────────────────────────┐
│  Application-level retry loop (DeliveryRetryConfig)     │
│  attempt 1 of (count + 1)                               │
│    │                                                     │
│    ▼                                                     │
│  ┌────────────────────────────────────────────────────┐  │
│  │  HTTP call: notifyContentDistribution              │  │
│  │  ┌── HTTP-level retries (http:RetryConfig) ──────┐ │  │
│  │  │  retry on: predefined-status-codes     │ │  │
│  │  └───────────────────────────────────────────────┘ │  │
│  └────────────────────────────────────────────────────┘  │
│    │ success → ack, exit loop                            │
│    │ failure → wait(interval * backoffFactor^n)          │
│    ▼                                                     │
│  attempt 2 of (count + 1) ...                           │
└─────────────────────────────────────────────────────────┘
         │
         ▼ all attempts exhausted
  onExhausted == "dlq"  →  deadLetter(message), continue loop
  onExhausted == "stale" →  nack(message), propagate error → subscription goes stale

---

With the proposed changes following changes are required for the message delivery configurations:

• Update http:RetryConfig to separate transport-level retries (handled by the HTTP client) from application-level retries (applied after transport retries are exhausted).
• Add AdvanceRetryConfig to define retry behavior at the outer layer, including count, interval, backoff, max wait, and terminal action (onExhausted).

Type definitions:

public type HttpClientConfig record {|
    decimal timeout = 60;
    record {|
        http:RetryConfig transport;
        AdvanceRetryConfig application?;
    |} 'retry?;
    http:ClientSecureSocket secureSocket?;
|};

public type AdvanceRetryConfig record {|
    int count = 0;
    decimal interval = 0;
    float backOffFactor = 0.0;
    decimal maxWaitInterval = 0;
    "dlq"|"stale" onExhausted = "stale";
|};

Example configuration via Config.toml:

[websubhub.config.delivery]
timeout = 60.0

[websubhub.config.delivery.retry.transport]
count = 3
interval = 3.0
backOffFactor = 2.0
maxWaitInterval = 20.0
statusCodes = [500, 502, 503]

[websubhub.config.delivery.retry.application]
count = 10
interval = 15.0
backOffFactor = 2.0
maxWaitInterval = 300.0
onExhausted = "dlq"

[ayeshLK]

[Update on 9th March 2026]

dlq can be used to handle poison messages (e.g., serialization/deserialization failures). However, it may not be suitable for recovery scenarios where the subscriber is temporarily unavailable. In such cases, sending messages to a DLQ does not help resume delivery once the subscriber becomes available again.

To address this, we can improve the retry policy by introducing a time-bound application-level retry mechanism. Instead of retrying a fixed number of times, the hub will continue retrying message delivery for a configured duration (e.g., 1–2 days). If the subscriber becomes available within this period, delivery can recover automatically without requiring manual intervention.

With this approach, the configuration can be updated as follows:

public type HttpClientConfig record {|
    decimal timeout = 60;
    record {|
        http:RetryConfig transport;
        AdvanceRetryConfig application?;
    |} 'retry?;
    http:ClientSecureSocket secureSocket?;
|};

public type AdvanceRetryConfig record {|
    # Maximum duration in seconds for delivery retry attempts per message.
    # A value of 0.0 disables application-level retries. Default is 1 day.
    decimal maxDuration = 86400.0;
    decimal interval = 30.0;
    float backOffFactor = 2.0;
    decimal maxWaitInterval = 3600.0;
|};

Based on this configuration, the hub will attempt to deliver a message to the subscriber for a maximum duration of 1 day. If delivery still fails after this period, the subscriber will be marked as stale.

[wggihan]

Hello @ayeshLK

Shall we keep retrying without moving to a stale state if the admin decides to configure it that way?

So, the proposal is to allow defining an exponential retry with the following options additionally.

Exponential retry is enabled with a maximum interval cap. Once this interval is reached, there will be options to retry with the maximum interval or reset it to 0 and continue.

[ayeshLK]

@wggihan understood the requirement here.

My suggestion is as follows:

We can introduce an onExhaust strategy in the configuration. This strategy can take either "stale" or "reset" as values. If "reset" is selected, once the configured maxDuration is reached, the retry cycle will restart with a new time limit (i.e., recalculated based on the current time and the configured maxDuration). This allows the system to continue retrying delivery instead of marking the subscriber as stale immediately.

Then the configurations would look like this:

public type HttpClientConfig record {|
    decimal timeout = 60;
    record {|
        http:RetryConfig transport;
        AdvanceRetryConfig application?;
    |} 'retry?;
    http:ClientSecureSocket secureSocket?;
|};

public type AdvanceRetryConfig record {|
    # Maximum duration in seconds for delivery retry attempts per message.
    # A value of 0.0 disables application-level retries. Default is 1 day.
    decimal maxDuration = 86400.0;
    decimal interval = 30.0;
    float backOffFactor = 2.0;
    decimal maxWaitInterval = 3600.0;
    "reset"|"stale" onExhausted = "stale";
|};

[wggihan]

+1 for the last suggestion.

[ayeshLK]

Had a discussion on this feature with @shafreenAnfar, @daneshk, and @gayaldassanayake, and we decided to enhance the Ballerina HTTP client to support this capability instead of introducing a separate semantic specifically within WebSubHub.

Accordingly, I will proceed with implementing an improvement to the ballerina/http library to support a retry attempt "reset" mechanism.

As of now I have created a ballerina/http library issue for this feature [1] and wrote a proposal for this [2][3]

[1] - ballerina-platform/ballerina-library#8701
[2] - ballerina-platform/ballerina-spec#1439
[3] - https://github.com/ayeshLK/ballerina-spec/blob/http-resettable-retry/beps/lib-http/1439_client_retry_reset.md

[ayeshLK]

[Update on 10th March 2026]

I think it would be useful to revisit the original problem we are trying to solve here. The main objective is to provide a mechanism to refresh a stale subscription once the subscriber endpoint recovers from a non-transient failure.

If we focus purely on improving the retry semantics, the solution may become unnecessarily complex and might address the problem in a somewhat brute-force manner, rather than solving it in a more systematic way.

Instead, if we take a step back and look at this from a different and more wider perspective, a cleaner solution might be to introduce an asynchronous stale-subscription refresh mechanism.

At any given time, a hub implementation may have zero or more stale subscribers. We could introduce an async worker that periodically scans these stale subscriptions (for example, at a configurable interval such as every 15 minutes). During this scan, the worker can check the liveness of the subscriber endpoint using a lightweight ping-style semantic implemented over HTTP.

If the subscriber endpoint responds successfully, the hub can immediately move the subscriber back to the active state.

This approach helps avoid prolonged stale subscriptions. Even if a subscriber endpoint goes down and comes back up after we have already exhausted the retry attempts, the maximum delay before it starts receiving messages again would be bounded by the stale-subscription refresh interval. For example, with a 15-minute interval, the subscriber would resume receiving messages within a maximum delay of approximately 15 minutes.

Possible Ping Semantic

One possible implementation would be for the hub to send an HTTP GET request to the subscriber callback endpoint with the following parameter:

• hub.mode = "ping"

The subscriber can acknowledge the ping by responding with HTTP 200 OK. Any other response (or failure to respond) can be treated as a failed ping attempt.

[gayaldassanayake]

+1 from me. This addresses the problem more cleanly

[ayeshLK]

[Update on 11th March 2026]

Had a discussion with @shafreenAnfar regarding the suggestion above. Key points from the discussion are as follows:

• The suggestion mentioned above [1] is a valid alternative approach to address the problem.
• However, when considering unbounded retries, in Ballerina this would be executed on a separate strand, which is lightweight. This raises the question of the memory overhead required to maintain the objects involved in this functionality. Given the lightweight nature of strands, it should be acceptable to block a strand to implement unbounded retries.
• In successful delivery scenarios, the strand would already be blocked while waiting for the response. Therefore, allowing the strand to remain blocked during retry attempts in failure scenarios would be conceptually similar.

Based on this discussion, we will proceed with implementing unbounded retries at the application level. However, this will not be introduced as a feature in the Ballerina HTTP client, as unbounded retries are generally considered an anti-pattern and should not be exposed as an official library feature.

[1] - #49 (comment)

[ayeshLK]

[Update on 12th March 2026]

With the above change we have update the retry configurations in the websubhub configs as below;

# Provides configurations for controlling the retrying behavior in failure scenarios.
public type RetryConfig record {|
    *http:RetryConfig;
    # When set to `true`, the internal retry counter is reset to zero after the count attempts are exhausted
    boolean resetOnExhaust = false;
|};