Replies: 1 comment
-
|
I should be clear about the role the LLM played here. The pacing logic was not designed by the model. What it provided was acceleration: a way to iterate through reasoning faster, surface edge cases earlier, and pressure-test assumptions, particularly around control-theoretic failure modes that would otherwise have taken much longer to uncover. The final system is entirely the product of human understanding and mathematical invariants. Every decision, constraint, and trade-off was derived, validated, and owned by me. The LLM shortened the path to insight; it did not replace the reasoning required to get there. Below are notes produced through iterative discussion between the LLM and myself. Budget PacingBudget pacing spreads ad spend evenly throughout the day, preventing the budget from being exhausted early. This prevents campaigns from going dark in the afternoon after burning through their budget in the morning. OverviewPacing is always enforced at the AdServer level using a pluggable
ArchitecturePacing is modular and configurable: Flow DiagramIMPORTANT: Pacing decision ( Key design decisions:
RateAwarePacing (Default Strategy)PI-controlled pacing that directly adjusts throttle probability based on spend error. How It Works
PI Control Parameters
Asymmetric GainsThe controller uses asymmetric gains to recover from overspend faster than it accelerates during underspend:
This is because overspend is costly (budget exhaustion stops delivery), while underspend is recoverable (can catch up later). Volatility-Adjusted GainsWhen created via
Grace Period (Hybrid: Time + Request Count + Staleness)The grace period uses a hybrid condition requiring BOTH time AND request count, plus staleness detection: // In AdaptivePacing.throttleProbability()
val initialGraceComplete =
ctx.elapsedSeconds >= MinGraceSeconds && // 10 seconds
ctx.requestCount >= MinGraceRequests // 50 requests
// Staleness threshold scales with day duration
val scaledStaleThresholdMs = max(
MinStaleRateThresholdMs, // 1000ms minimum
BaseStaleRateThresholdMs * dayDurationSeconds / 86400
)
val rateIsStale = ctx.msSinceLastRequest > scaledStaleThresholdMs
val inGracePeriod = !initialGraceComplete || rateIsStaleKey constants (in val MinGraceSeconds: Double = 10.0 // Time before PI activates
val MinGraceRequests: Long = 50L // Requests before PI activates
val BaseStaleRateThresholdMs: Long = 30000L // Base staleness for real days
val MinStaleRateThresholdMs: Long = 1000L // Minimum staleness thresholdScaled staleness examples:
During grace period:
CRITICAL: If Staleness re-entry: After a 30+ second traffic gap (e.g., quiet period), the rate data is considered stale and grace period re-activates. This prevents using outdated rate estimates. Traffic Shape Tracking
Problem: Linear PacingLinear pacing assumes uniform traffic: But real traffic has shape: This causes over-throttling during peaks and impossible targets during valleys. Solution: Traffic-Shaped TargetingThe cumulative distribution function (CDF) of traffic becomes the expected spend curve: Bucket-to-Time MappingThe
Weekday vs Weekend ShapesTwo separate shapes can be configured:
The system automatically selects the appropriate shape based on the current day (UTC). DayClockHandles real vs simulated day timing with consistent UTC timezone. RealDayClock (dayDurationSeconds = 86400)
Example at 14:00 UTC: SimulatedDayClock (dayDurationSeconds < 86400)
Example with dayDurationSeconds Validation
This ensures the traffic shape (24 buckets) maps correctly to time. TrafficObserverTracks request arrival rate using exponential moving average (EMA). // Update on each request
observer.recordRequest(now)
// Get smoothed rate
val reqPerSec = observer.smoothedRate // e.g., 150.3The smoothed rate is used by Without rate tracking, high-traffic scenarios would cause severe throttle oscillation. PacingControllerCoordinates pacing state across components:
Day Boundary Detectiondef hasNewDayStarted(newDayStart: Instant): Boolean = {
val lastDay = LocalDate.ofInstant(lastDayStart, ZoneOffset.UTC)
val newDay = LocalDate.ofInstant(newDayStart, ZoneOffset.UTC)
lastDay != newDay
}All day boundary logic uses UTC for consistency across server and client. UTC Timezone RequirementThe entire pacing system uses UTC consistently:
This ensures client and server agree on which traffic shape bucket to use. PacingContextImmutable snapshot passed to strategies: final case class PacingContext(
dailyBudget: BigDecimal,
todaySpend: BigDecimal,
dayStart: Instant,
now: Instant,
requestArrivalRate: Double = 0.0, // From TrafficObserver
competingCampaigns: Int = 1,
avgCpm: Double = 5.0,
dayDurationSeconds: Int = 86400, // Must be <= 86400
trafficShape: Option[TrafficShapeTracker] = None
) {
def elapsedHours: Double
def expectedSpendFraction: Double // Traffic-shaped or linear
def expectedSpend: BigDecimal
def spendRatio: Double // actual / expected
def remainingBudget: BigDecimal
def remainingHours: Double
}ConfigurationSite Pacing Config# Set pacing config for a site
curl -X PUT http://localhost:8080/v1/publishers/pub-1/sites/site-123/pacing \
-H "Content-Type: application/json" \
-d '{
"dayDurationSeconds": 600,
"weekdayShapeVolumes": [0.3,0.2,1.0,0.2,0.0,0.3,0.5,2.5,0.1,2.0,1.5,2.0,2.5,3.0,2.5,2.0,1.5,1.2,1.2,1.0,0.4,2.0,5.0,0.4],
"weekendShapeVolumes": [0.3,0.2,0.1,0.1,0.1,0.2,0.3,0.5,0.8,1.2,1.5,1.8,2.0,2.2,2.3,2.2,2.0,1.8,2.0,2.5,2.8,2.0,1.2,0.5]
}'Test Throttle OverrideFor testing, you can force a fixed throttle probability: {
"testThrottleOverride": 0.5
}This bypasses PI control and uses Changing the Default StrategyTo use a different strategy, modify AdServer(
publisherId,
// ... other params
pacingStrategy = AdaptivePacing.forShapeVolumes(myShapeArray),
// or: pacingStrategy = FixedThrottlePacing(0.3),
)Observing PacingServer-side Statscurl http://localhost:8080/test/site-stats/site-123Response: {
"siteId": "site-123",
"selected": 58,
"pacingSkipped": 42,
"budgetExhausted": 0,
"noCandidates": 0,
"totalSpend": 0.29,
"elapsedHours": 0.5,
"expectedSpendFraction": 0.5
}RunScenario ReportsWhen running Outcome Types
TestingRun a simulation with budget constraints to observe pacing: # 1. Start the server
sbt "api/run"
# 2. Run pacing test with traffic shapes (10-minute simulated day)
scala-cli scripts/RunScenario.scala -- \
--scenario scenarios/continuous.json \
--continuous
# 3. Or run with real-day timing (aligns with UTC wall clock)
# Edit continuous.json: "dayDurationSeconds": 86400The periodic reports will show:
Custom StrategiesImplement the trait PacingStrategy {
/** Called BEFORE Thompson Sampling - return true to serve, false to skip */
def shouldServe(ctx: PacingContext): Boolean
/** Calculate throttle probability [0.0, 1.0] */
def throttleProbability(ctx: PacingContext): Double
/** Reset state at day boundary */
def reset(): Unit
/** Strategy name for logging */
def name: String
}Example custom strategy: class TimeOfDayPacing extends PacingStrategy {
def throttleProbability(ctx: PacingContext): Double = {
val hour = (ctx.elapsedHours % 24).toInt
if (hour >= 9 && hour <= 17) 0.0 // No throttle during business hours
else 0.8 // Heavy throttle off-hours
}
def name = "time-of-day"
}Debugging Pacing IssuesIssue: 0% Pacing Despite OverspendSymptoms: Root cause: Usually caused by grace period never completing. Check 1: Behavior parameter propagation The def behavior(
...
requestCount: Long = 0L, // Default: 0
lastRequestTimeMs: Long = 0L // Default: 0
)CRITICAL BUG PATTERN: If a // BAD - missing requestCount and lastRequestTimeMs
behavior(
cachedDomainBlocklist,
creativeStats,
serveStats,
lastDayStart,
pacingStrategy,
smoothedReqRate,
pendingSpendByCampaign,
dayDurationSeconds,
spendInfoCache,
trafficShapeTracker,
rolloverGraceUntilMs,
warmupMode
// MISSING: requestCount, lastRequestTimeMs → both become 0!
)
// GOOD - all parameters passed
behavior(
cachedDomainBlocklist,
creativeStats,
serveStats,
lastDayStart,
pacingStrategy,
smoothedReqRate,
pendingSpendByCampaign,
dayDurationSeconds,
spendInfoCache,
trafficShapeTracker,
rolloverGraceUntilMs,
warmupMode,
requestCount, // Preserve state
lastRequestTimeMs // Preserve state
)With How to find this bug: Search for all Check 2: SpendInfo cache empty If If this appears on every request, spend updates aren't being cached. Check 3: Stale SpendUpdate filtering For simulated days, SpendUpdates are filtered if their val isStale = dayDurationSeconds != 86400 && lastDayStart.exists(currentDayStart =>
su.dayStart.toEpochMilli < currentDayStart.toEpochMilli - 5000
)Check logs for: Issue: Pacing Too AggressiveSymptoms: Very few impressions served, high pacingSkipped Possible causes:
Key Log MessagesEnable debug logging and look for: |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I’m currently working on a publisher-centric, context-based ad network built on Apache Pekko Cluster.
The project is still in active development, but it is intended to be fully open source. I plan to disclose architectural details and share the repository once it is ready.
Promovolve is meant to show what Akka/Pekko can actually build when it’s used as a runtime rather than just a convenient toolkit.
A lot of systems that moved away after the license change could do so because actors were mainly used as a threading or batch-execution model. That usage was never “wrong”because Akka was explicitly positioned as a toolkit, and partial adoption was always valid. But that part is also the easiest to replace.
What tends to be overlooked is the value of the full cluster model. When you commit to this you can build an entire production system with one coherent set of ideas, without stitching together a large collection of unrelated frameworks.
Promovolve is intentionally built that way. It’s a complete ad-serving runtime that relies primarily on cluster semantics. The point isn’t that other approaches are wrong, but that the cluster model already gives you all of that, if you let it.
Used this way, Akka/Pekko isn’t just a convenience layer, but most simple examples fail to show this, because they stop at actors as a concurrency tool and never reach the cluster semantics where that architectural value actually appears.
I felt that going beyond examples and building something fully production-ready would attract practitioners, and that real usage would naturally bring in stakeholders later, thereby increasing Pekko's visibility.
While ad tech can be seen as niche, it’s where I’ve spent most of my career, and it’s a domain with a clear path to real revenue.
Adaptive Budget Pacing
Demonstrating ad budget pacing that works across all traffic levels. It's a feature I am currently working on.
https://www.youtube.com/watch?v=DLE8g2SUmJU
On the left is the ad network.
On the right is a traffic simulator.
All configuration of the ad network on the left can be done via APIs.
Creatives are registered directly on a CDN, so the only costs involved are server runtime and CDN fees.
If you have a Kubernetes (K8s) environment, anyone can deploy and use it immediately.
The management UI and ad slots can be freely implemented by developers; anyone may be able to run an ad business themselves (potentially).
This ad network is designed so that no creative is delivered unless the publisher explicitly approves it in advance.
Even if an advertiser’s campaign wins the auction, the ad will not be shown unless the publisher has approved that creative, allowing publishers to use the network with confidence.
For those unfamiliar with ad pacing: imagine an advertiser has a daily budget of $100 to spend on ads, and traffic to a site is not constant. If the system simply buys every available impression as fast as possible, the entire budget could be spent in the first hour of the day, leaving the advertiser invisible for the remaining 23 hours. Pacing is the control logic that decides when not to buy; deliberately skipping some opportunities early so that spend is spread smoothly across time. This becomes harder when traffic is bursty, feedback is delayed, and prices vary, which is why pacing is fundamentally a control problem, not a simple rule.
Beta Was this translation helpful? Give feedback.
All reactions