traffic components #271

Ian-Nara · 2025-12-09T00:59:13Z

No description provided.

…c-components

asloobq

submitting partial review

asloobq · 2025-12-10T14:35:48Z

src/main/java/com/uid2/optout/traffic/TrafficCalculator.java

+            // process delta files and count records in [deltaWindowStart, newestDeltaTs]
+            // files are sorted newest to oldest, records within files are sorted newest to oldest
+            // stop when the newest record in a file is older than the window
+            int sum = 0;


suggestion: rename sum to totalRecords or totalRequests

asloobq · 2025-12-10T14:43:31Z

src/main/java/com/uid2/optout/traffic/TrafficCalculator.java

+    /**
+     * Extract timestamp from SQS message (from SentTimestamp attribute)
+     */
+    private Long extractTimestampFromMessage(Message msg) {


can you reuse this method from SQSMessageParser.extractTimestamp()

asloobq · 2025-12-10T14:58:46Z

src/main/java/com/uid2/optout/traffic/TrafficCalculator.java

+    }
+
+    /**
+     * Count SQS messages from oldestQueueTs to oldestQueueTs + 5 minutes


can you remind me why the unprocessed messages in queue are capped upto 5 mins ?

We do not wish to read the entire queue (to apply the allowlist based on timestamp), just 5 minutes at a time.

asloobq · 2025-12-10T14:59:46Z

src/main/java/com/uid2/optout/traffic/TrafficCalculator.java

+        for (Message msg : sqsMessages) {
+            Long ts = extractTimestampFromMessage(msg);
+
+            if (ts < oldestQueueTs || ts > windowEnd) {


ts < oldestQueueTs I think this is not possible because oldestQueueTs was determined from the same sqsMessages

asloobq · 2025-12-10T15:02:27Z

src/main/java/com/uid2/optout/traffic/TrafficCalculator.java

+     * Determine traffic status based on current vs baseline traffic.
+     * Logs warnings at 50%, 75%, and 90% of the circuit breaker threshold.
+     */
+    TrafficStatus determineStatus(int sumCurrent, int baselineTraffic) {


suggestion: rename sumCurrent to last24HrTraffic or similar

asloobq · 2025-12-10T15:05:08Z

src/main/java/com/uid2/optout/traffic/TrafficCalculator.java

+            LOGGER.warn("high_message_volume: 90% of threshold reached, sumCurrent={}, threshold={} ({}x{}), thresholdPercent={}%", 
+                       sumCurrent, threshold, thresholdMultiplier, baselineTraffic, String.format("%.1f", thresholdPercent));
+        } else if (thresholdPercent >= 75.0) {
+            LOGGER.warn("high_message_volume: 75% of threshold reached, sumCurrent={}, threshold={} ({}x{}), thresholdPercent={}%", 


the log messages are same except for 90/75/50. I think you can do it with one log message

asloobq · 2025-12-10T15:07:17Z

src/main/java/com/uid2/optout/traffic/TrafficCalculator.java

+        }
+
+        if (sumCurrent >= threshold) {
+            LOGGER.error("circuit_breaker_triggered: traffic threshold breached, sumCurrent={}, threshold={} ({}x{})", 


Is the log type ERROR on purpose ? I think the system is working fine so this shouldn't be an error. May be WARN ?

asloobq · 2025-12-10T15:16:36Z

src/main/java/com/uid2/optout/traffic/TrafficCalculator.java

+     * @param sqsMessages List of SQS messages this consumer has read (non-denylisted)
+     * @param queueAttributes Queue attributes including invisible message count (can be null)
+     * @param denylistedCount Number of denylisted messages read by this consumer
+     * @param filteredAsTooRecentCount Number of messages filtered as "too recent" by window reader


wouldn't these be part of sqsMessages ?

no, these are not returned by SqsBatchProcessor

// Filter for eligible messages (>= deltaWindowSeconds old) List<SqsParsedMessage> eligibleMessages = filterEligibleMessages(parsedBatch, currentTime); return BatchProcessingResult.withMessages(eligibleMessages);

But they will appear in the invisible message count returned by getQueueAttributes. We remove / "deduplicate" this from the invisible count, since we are using that to represent messages from other consumers

Okay, this is a bit confusing TBH.

I agree, I will add to the list, to revisit this when the usage of trafficcalculator is added.

asloobq · 2025-12-10T16:08:53Z

src/main/java/com/uid2/optout/traffic/TrafficFilter.java

+        List<TrafficFilterRule> rules = new ArrayList<>();
+        try {
+            JsonArray denylistRequests = config.getJsonArray("denylist_requests");
+            if (denylistRequests == null) {


so config should at least have an empty list ?

yes, just to avoid key typos when configuring

asloobq · 2025-12-10T16:16:48Z

src/main/java/com/uid2/optout/traffic/TrafficFilter.java

+
+                TrafficFilterRule rule = new TrafficFilterRule(range, ipAddresses);
+
+                LOGGER.info("loaded traffic filter rule: range=[{}, {}], IPs={}", rule.getRangeStart(), rule.getRangeEnd(), rule.getIpAddresses());


will this log all the IP addresses or just the size of the list ?

This would log the IP addresses we configured. Do you think we should avoid this?

On one hand, this could flood the logs if the list is too big, on the other hand it would be useful for debugging. Let's keep it for now.

asloobq · 2025-12-10T16:18:43Z

src/main/java/com/uid2/optout/traffic/TrafficFilter.java

+     */
+    private static class TrafficFilterRule {
+        private final List<Long> range;
+        private final List<String> ipAddresses;


probably a Set is better to do contains() efficiently

…timestamp parsing effort

…oldestqueuetimestamp back

…om list to set

…, use two longs for time ranges instead of List<Long>

traffic components

e1da3fc

Ian-Nara changed the base branch from main to ian-UID2-6345-sqs-components December 9, 2025 00:59

Ian-Nara marked this pull request as ready for review December 9, 2025 01:00

Ian-Nara added 10 commits December 9, 2025 13:49

addressing PR comments

d6f6d16

update tests

193ab19

clean up files

8acd1a8

test fix

d9e2641

Merge branch 'ian-UID2-6345-sqs-components' into ian-UID2-6345-traffi…

3aa5c0d

…c-components

clean up files

609c347

try make git render file recreated

950af13

try make git render file recreated

7c49222

create empty

6922b0f

create empty

52e7818

Ian-Nara force-pushed the ian-UID2-6345-traffic-components branch from 2023858 to 52e7818 Compare December 9, 2025 23:44

Ian-Nara added 5 commits December 9, 2025 17:18

file rename

0eb06a2

file update

952dcf3

file update

caf3d0e

file updates

813cd06

update comments

e02e4f2

clarkxuyang approved these changes Dec 10, 2025

View reviewed changes

Ian-Nara force-pushed the ian-UID2-6345-sqs-components branch 2 times, most recently from 6e9fa46 to bbda3f2 Compare December 10, 2025 03:55

asloobq reviewed Dec 10, 2025

View reviewed changes

Base automatically changed from ian-UID2-6345-sqs-components to main December 10, 2025 21:19

Ian-Nara added 5 commits December 10, 2025 15:41

Merge branch 'main' into ian-UID2-6345-traffic-components

0979fc1

rename sum to totalRecords

49660be

refactor traffic calculator to use Parsed message to avoid duplicate …

cecc3e2

…timestamp parsing effort

use stream to find oldest queue timestamp

844332d

rename sumCurrent to recentTrafficTotal, remove dead if condition

f8a3d03

Ian-Nara added 6 commits December 10, 2025 17:16

consolidate warning logs for high message volume, add null check for …

4e22c34

…oldestqueuetimestamp back

update log level for circuit_breaker_triggered. Change ipAddresses fr…

11af1ed

…om list to set

update trafficfiltertest to use Junit5 not Junit4

8d6cd71

remove unused imports

836cd99

TrafficFilter minor efficiency improvements, usee stream to parse IPs…

ac022fb

…, use two longs for time ranges instead of List<Long>

TrafficCalculator: remove redundant checks

7725d44

asloobq approved these changes Dec 11, 2025

View reviewed changes

Ian-Nara merged commit b7161ce into main Dec 11, 2025
4 checks passed

Ian-Nara deleted the ian-UID2-6345-traffic-components branch December 11, 2025 21:24


		TrafficFilterRule rule = new TrafficFilterRule(range, ipAddresses);

		LOGGER.info("loaded traffic filter rule: range=[{}, {}], IPs={}", rule.getRangeStart(), rule.getRangeEnd(), rule.getIpAddresses());

traffic components #271

traffic components #271

Uh oh!

Conversation

Ian-Nara commented Dec 9, 2025

Uh oh!

asloobq left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ian-Nara Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Ian-Nara Dec 10, 2025 •

edited

Loading