clientContextImpl: Cap the number and age of beacons #191

JJL772 · 2023-10-25T01:38:07Z

Each beacon has an associated mutex. If we don't cap the beacon count, IOCs running on resource limited platforms like RTEMS may eventually run out of resources and crash.

Closes #184

The configuration options are probably unnecessary, let me know if I should remove them.

Requires some changes to pvData: epics-base/pvDataCPP#94

mdavidsaver

I am happy to see a PR addressing this issue. I think some further work is required though.

src/remoteClient/clientContextImpl.cpp

mdavidsaver · 2023-10-25T02:42:00Z

Since it is not super obvious. The Beacon TX timing of pvAccessCPP differs from what experience with CA servers might lead you to expect.

pvAccessCPP/src/server/beaconEmitter.cpp

Lines 31 to 33 in 581d100

    
           _fastBeaconPeriod(std::max(context->getBeaconPeriod(), EPICS_PVA_MIN_BEACON_PERIOD)), 
        
           _slowBeaconPeriod(std::max(180.0, _fastBeaconPeriod)), // TODO configurable 
        
           _beaconCountLimit((int16)std::max(10.0f, EPICS_PVA_MIN_BEACON_COUNT_LIMIT)), // TODO configurable

A server will send out the first 10 beacons (not configurable) with a 15 second interval (by default), then switch to a 180 second period (not configurable). While $EPICS_PVAS_BEACON_PERIOD can override the first "fast" period, I don't think this is in practice useful.

So I think the beacon tracking lifetime must be >= 360 seconds.

(fyi. with PVXS I try to follow the same model and timings, with a non-configurable limit of 20k servers. Of course, there I only allocate ~64 bytes per server)

mdavidsaver · 2023-10-25T03:06:23Z

The windows CI failures are due to epics-base/ci-scripts#84. When you update, please rebase to pick up ed7eae5.

AppVeyorBot · 2023-10-25T05:26:03Z

✅ Build pvAccessCPP 1.0.70 completed (commit cdf3720715 by @JJL772)

JJL772 · 2023-12-13T00:30:43Z

@mdavidsaver Thanks for the feedback! I finally got around to applying the requested changes.
I ended up using pvxs as a reference and copied the max beacon lifetime (360s) and beacon limit (20000).

AppVeyorBot · 2023-12-13T01:11:05Z

❌ Build pvAccessCPP 1.0.74 failed (commit 73c3932b45 by @JJL772)

AppVeyorBot · 2023-12-13T01:54:52Z

❌ Build pvAccessCPP 1.0.75 failed (commit 4e054f2e07 by @JJL772)

AppVeyorBot · 2023-12-20T02:51:13Z

✅ Build pvAccessCPP 1.0.79 completed (commit 2ae88b70f1 by @JJL772)

JJL772 · 2024-02-28T20:21:03Z

@mdavidsaver Just wanted to follow up, are there any other changes required for this?

mdavidsaver · 2024-02-28T21:58:31Z

src/remoteClient/clientContextImpl.cpp

+            if (m_beaconHandlers.size() >= maxTrackedBeacons)
+            {
+                char ipa[64];
+                sockAddrToDottedIP(&responseFrom->sa, ipa, sizeof(ipa));
+                LOG(logLevelDebug, "Tracked beacon limit reached (%d), ignoring %s\n", maxTrackedBeacons, ipa);


To minimize log spam it would be friendlier to only log when size()==max. So once each time the limit is reached, but not again until falling below the limit. eg. consider if some PVA server gets stuck in a reset loop.

src/remoteClient/clientContextImpl.cpp

JJL772 · 2024-06-10T19:42:58Z

This PR now depends on some changes made to pvData: epics-base/pvDataCPP#94

I'm going to mark this as a draft for now because I'm not exactly happy with these changes yet.

AppVeyorBot · 2024-06-11T03:03:04Z

❌ Build pvAccessCPP 1.0.108 failed (commit c4e4658381 by @JJL772)

AppVeyorBot · 2024-06-13T01:03:06Z

✅ Build pvAccessCPP 1.0.109 completed (commit 9651462441 by @JJL772)

src/remoteClient/clientContextImpl.cpp

mdavidsaver · 2025-11-20T01:49:57Z

src/remoteClient/clientContextImpl.cpp

+    epicsTimeStamp ts;
+    epicsTimeGetCurrent(&ts);
+    if (epicsTimeDiffInSeconds(&ts, &m_handler.m_lastTime) > maxBeaconLifetime)
+        m_handler.remove();


I don't see any need for precision here. And calling epicsTimeGetCurrent() potentially 20,000 times could be avoided.

Replace epicsTimeStamp m_lastTime with a integer counter. Set the timer to eg maxBeaconLifetime/2. Increment the counter on expiration. If it reaches 3, then remove(). Have touch() zero the counter.

That said, this is an optimization. What you have here will work. So please let me know if you will not have to for this change.

This is a good optimization, though it does introduce a bit of jitter to the beacon timer. I'm setting the timer to maxBeaconLifetime/4 so that beacon lifetime is 360-450s (worse case)

I've also reduced the beacon cap to be 2048, which is still excessive imo. 20k was just unnecessary

I don't have a good feel for what this cap should be. My most recent solid point of reference is NSLS2 circa 2015, when I snooped on beacon traffic to identify ~1000 CA server instances running on ~100 hosts. Well short of 20k, but only a factor of 2 from 2048.

PVXS currently tracks up to 20k addresses. I picked this number from thin air. Although I opted for a larger number because PVXS uses fewer resources per server ( ~= 256 bytes per).

Assuming this cap is the maximum number of servers that could be generating beacons visible to a client, the APS accelerator currently has 1136 IOCs in production (mostly CA currently) spread across 5 VLANs, with 881 IOCs in the largest VLAN. I don't see our total number increasing beyond 2048 IOCs, but we do have 13 VMs that run multiple soft IOCs (the largest VM has 214) so the number of visible servers isn't limited to the subnet size. Presumably configuring PVA servers through a shared multicast address would include the servers from multiple subnets too.

I'm fine with the cap being 2048 for pvAccessCPP. It might be worth having the PVXS cap be configurable by an env-var at some point, how many IOCs is ITER expecting to run?

(* looking into the crystal bowl *)
Maybe 2500 on 500 hosts across 15 networks.
But that's 10 years from now, with an uncertainty of maybe 30%.

Each beacon has an associated mutex. If we allocate too many beacons on resource constrained systems, i.e. RTEMS, we may run out of resources and crash.

mdavidsaver

Looks ok. Thanks for your persistence!

mdavidsaver requested changes Oct 25, 2023

View reviewed changes

src/remoteClient/clientContextImpl.cpp Outdated Show resolved Hide resolved

src/remoteClient/clientContextImpl.cpp Outdated Show resolved Hide resolved

src/remoteClient/clientContextImpl.cpp Outdated Show resolved Hide resolved

JJL772 force-pushed the fix_semaphore branch 2 times, most recently from 2b51b97 to 777d68d Compare December 13, 2023 00:27

JJL772 force-pushed the fix_semaphore branch from 777d68d to afca135 Compare December 13, 2023 01:12

JJL772 force-pushed the fix_semaphore branch from afca135 to d44adc9 Compare December 19, 2023 22:52

mdavidsaver requested changes Feb 28, 2024

View reviewed changes

JJL772 force-pushed the fix_semaphore branch from d44adc9 to 4402841 Compare June 10, 2024 19:37

JJL772 marked this pull request as draft June 10, 2024 19:43

JJL772 force-pushed the fix_semaphore branch from 4402841 to 1bcc944 Compare June 12, 2024 20:43

JJL772 marked this pull request as ready for review June 12, 2024 20:44

mdavidsaver reviewed Nov 19, 2025

View reviewed changes

src/remoteClient/clientContextImpl.cpp Show resolved Hide resolved

mdavidsaver requested changes Nov 20, 2025

View reviewed changes

JJL772 force-pushed the fix_semaphore branch 2 times, most recently from 7a0921d to e21830f Compare November 21, 2025 01:09

clientContextImpl: Cap the number and age of beacons

ca85002

Each beacon has an associated mutex. If we allocate too many beacons on resource constrained systems, i.e. RTEMS, we may run out of resources and crash.

JJL772 force-pushed the fix_semaphore branch from e21830f to ca85002 Compare November 21, 2025 01:29

mdavidsaver approved these changes Nov 21, 2025

View reviewed changes

mdavidsaver merged commit 205dc58 into epics-base:master Nov 21, 2025
16 of 25 checks passed

JJL772 deleted the fix_semaphore branch November 21, 2025 17:11

JJL772 mentioned this pull request Nov 21, 2025

Fix build errors and warnings from #184 #218

Merged

clientContextImpl: Cap the number and age of beacons #191

clientContextImpl: Cap the number and age of beacons #191

Uh oh!

Conversation

JJL772 commented Oct 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mdavidsaver left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mdavidsaver commented Oct 25, 2023

Uh oh!

mdavidsaver commented Oct 25, 2023

Uh oh!

AppVeyorBot commented Oct 25, 2023

Uh oh!

JJL772 commented Dec 13, 2023

Uh oh!

AppVeyorBot commented Dec 13, 2023

Uh oh!

AppVeyorBot commented Dec 13, 2023

Uh oh!

AppVeyorBot commented Dec 20, 2023

Uh oh!

JJL772 commented Feb 28, 2024

Uh oh!

mdavidsaver Feb 28, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JJL772 commented Jun 10, 2024

Uh oh!

AppVeyorBot commented Jun 11, 2024

Uh oh!

AppVeyorBot commented Jun 13, 2024

Uh oh!

Uh oh!

Uh oh!

mdavidsaver Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

JJL772 Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

mdavidsaver Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

anjohnson Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

ralphlange Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

mdavidsaver left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

JJL772 commented Oct 25, 2023 •

edited

Loading