bug: Discovery synchronization fails for large services due to shared memory payload constraints

[jizhuozhi]

Current Behavior

While integrating APISIX as an ingress / access gateway for large downstream services, I encountered a discovery synchronization issue that appears to be structural rather than configuration-related.

During routine testing last night (this surfaced as part of ongoing work), APISIX failed to update service discovery state with the following error:

2025/12/26 12:32:39 [error] 49#49: *734 [lua] init.lua:583: post_event failure with discovery_consul_update_all_services, update all services error: failed to publish event: payload exceeds the limitation (65536), context: ngx.timer

This happened during a full discovery update from the registry (Consul in this case), triggered by a timer.

---

Expected Behavior

APISIX should be able to reliably synchronize service discovery state from the control plane to all workers, regardless of downstream service scale or autoscaling behavior.

Specifically:

• Discovery updates should not fail when the number of endpoints grows due to elastic scaling (e.g. Kubernetes HPA).
• Newly added instances should be correctly propagated to all workers and become eligible to receive traffic.
• Restarting an APISIX instance should always result in a complete and consistent discovery state being loaded from the registry.

In other words, service discovery should remain correct and complete as endpoint count grows.

---

Actual Behavior

When the number of instances for a single service grows beyond a certain point, APISIX exhibits the following failure modes:

1. After elastic scaling, newly added instances cannot receive traffic

   As endpoint count increases, the serialized discovery snapshot exceeds the payload size that can be delivered via worker events.
   When this happens, the discovery update event fails to be published, and newly registered instances are not propagated to workers.

   As a result, these instances exist in the registry (e.g. Consul) but are invisible to APISIX workers and never receive traffic.

2. After an APISIX restart, the node cannot load the full discovery state

   If APISIX is restarted while the total discovery payload already exceeds the event delivery limit, the initial full discovery synchronization also fails.

   In this case, the restarted APISIX node is unable to load all instances from the registry, leading to an incomplete local discovery state from startup.

Both behaviors result in stale or incomplete endpoint information, and the impact becomes more severe as service scale increases.

---

Why this is a structural issue

The current design assumes that discovery state can be delivered as a single event payload.

In large-scale environments:

• Endpoint count grows naturally with traffic and autoscaling
• Event payload size grows linearly with endpoint count
• Delivery via a single shared-memory-backed event becomes a hard bottleneck

As a result, this failure mode is inevitable in sufficiently large deployments and cannot be fully mitigated through configuration tuning alone.

Error Logs

2025/12/26 12:32:39 [error] 49#49: *734 [lua] init.lua:583: post_event failure with discovery_consul_update_all_services, update all services error: failed to publish event: payload exceeds the limitation (65536), context: ngx.timer

Steps to Reproduce

1. Prepare a Consul registry (local or remote) and configure APISIX to use Consul as the service discovery backend.

2. Register a large number of instances for a single service in Consul.

   This can be done by repeatedly calling Consul’s /v1/agent/service/register API to register hundreds or thousands of instances under the same service name.
   Each instance uses a unique ID, but shares the same Name, Address, and Port, and includes instance-level metadata via the Meta field (used for endpoint matching in APISIX).

3. Ensure that the total number of registered instances is large enough (e.g. 1000+) so that the serialized discovery snapshot (service + endpoints + metadata) exceeds the size that can be carried by a single worker event.

4. Wait for APISIX to trigger a full discovery synchronization (e.g. via the periodic discovery timer), or restart APISIX to force an initial full sync.

5. Observe the APISIX error log.
   When the full discovery snapshot is broadcast via worker events, APISIX fails to publish the event and logs an error similar to:

[lua] init.lua:583: post_event failure with discovery_consul_update_all_services, update all services error: failed to publish event: payload exceeds the limitation (65536), context: ngx.timer

6. At this point, workers stop receiving updated discovery state for the affected service, resulting in stale or inconsistent endpoint information.

#!/usr/bin/env python3
"""
Bulk register multiple services in Consul to reproduce APISIX discovery payload overflow,
using Meta fields instead of tags.
"""

import requests

CONSUL_HOST = "http://127.0.0.1:8500"  # Consul address
SERVICE_NAME = "billing.ad.oversea.cost-entry"
INSTANCE_COUNT = 1000  # adjust to trigger payload limit
SERVICE_PORT = 8080

for i in range(1, INSTANCE_COUNT + 1):
    service_id = f"{SERVICE_NAME}-{i}"
    payload = {
        "ID": service_id,
        "Name": SERVICE_NAME,
        "Address": "127.0.0.1",
        "Port": SERVICE_PORT,
        "Meta": {
            "version": "v1",
            "region": "us-east-1"
        }
    }
    resp = requests.put(f"{CONSUL_HOST}/v1/agent/service/register", json=payload)
    if resp.status_code != 200:
        print(f"Failed to register {service_id}: {resp.text}")
    else:
        if i % 100 == 0:
            print(f"Registered {i} instances...")

print("Done. Now trigger APISIX discovery sync to reproduce the issue.")

Environment

• APISIX version (run apisix version):
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[jizhuozhi]

Approaches

1. Emit discovery events per service

Split update_all_services into per-service events.

• ✅ Mitigates the issue for multi-service setups
• ❌ Still fails for a single very large service
• ❌ Does not fundamentally remove the payload size ceiling

---

2. Adjust the shared memory size for events

The worker events mechanism already stores event payloads in shared memory.

• ✅ Keeping event payloads within the current shm limits can reduce immediate failures
• ❌ Discovery updates are still fundamentally constrained by shm capacity
• ❌ Requires continuous tuning of shm size to accommodate large payloads
• ❌ Does not eliminate the structural scalability limitation of full snapshot broadcasts

In practice, this turns the problem into ongoing shm tuning, rather than a true fix for large-scale deployments.

---

3. Let workers pull directly from the registry

Master only watches; workers fetch endpoints themselves.

• ✅ Simple to implement
• ❌ Multiplies registry load by worker count
• ❌ Breaks the current “single watcher” design
• ❌ Not registry-friendly at scale

---

4. Chunked / streaming discovery updates (preferred direction)

Split large discovery payloads into multiple smaller chunks and deliver them incrementally via events, allowing workers to assemble the full state.

• ✅ Avoids exceeding event payload limits
• ✅ Preserves existing master/worker responsibilities
• ✅ Does not increase registry pressure
• ✅ Can evolve toward incremental or streaming discovery in the future
• ❌ Requires chunk ordering, versioning, and completion semantics

---

Additional Context: Kubernetes HPA makes this issue inevitable

This issue surfaced after enabling Kubernetes HPA for downstream services. Initially, the number of endpoints was small and the problem was not observable. As HPA scaled the number of pods, discovery payload size grew until it exceeded the 64KB event limit.

• The problem does not appear at deployment time but emerges naturally as traffic grows
• In HPA-enabled environments, this failure mode is inevitable, not accidental

[ChuanFF]

use shared_dict rather than event broadcast seems to solve this problem?
#12773

[jizhuozhi]

It seems to be the same issue, so I will move the discussion to that pull request.

[Baoyuantop]

1. The lua-resty-events library has a default max_payload_len limit: according to the lua-resty-events source code, the default value is DEFAULT_MAX_PAYLOAD_LEN = 1024 * 64 (i.e., 65536 bytes = 64KB). The APISIX events module does not have max_payload_len configured during initialization.
2. To temporarily avoid this problem, you can switch to lua-resty-worker-events in the configuration.
3. The changes described in fix: change consul event module to shared dict #12773 can also alleviate this problem. After the changes, the size of the shared dict can be adjusted via configuration file.