Skip to content

CHANGELOG.md

Latest commit

 

History

History
691 lines (514 loc) · 50.5 KB

CHANGELOG.md

File metadata and controls

691 lines (514 loc) · 50.5 KB
Raw

Changelog

All notable Percona patches to hetzner-cloud-plugin are documented here.

v103.percona.30 (2026-08-07)

Hardens the v103.percona.29 retention fix based on a multi-model adversarial review before the fleet rollout (v29 was released but never deployed; v30 is the version that rolls out).

  • hetzner_agents_idle_overdue excludes hour-wrap agents. Their policy has no idle-shutdown period (a worker may legitimately idle until the end of its billing hour), so the previous 20-minute threshold would have emitted up to ~35 minutes of false reap-failure signal per hour for every idle hour-wrap agent. Idle-period agents keep the twice-idleMinutes threshold, floored at 20 minutes; covered by a new threshold unit test.
  • IdlePeriodPolicy is now final and its readResolve() is private. The protected hook on an extensible class was the classic readResolve-subclassing trap: XStream would invoke the inherited parent hook on any future subclass and silently replace it with a plain IdlePeriodPolicy, losing subtype state.
  • The reap-health gauge docs now state the observable semantics (effective strategy is Always) instead of the unobservable null field, and the refresh failure handler logs the full stack trace instead of a possibly-empty exception message.
  • The metrics refresher no longer skips the retention gauges while the API token is rate-limited. HetznerCloud.refreshLocalMetrics() (pending provisions + reap-health gauges, no API calls) now runs every cycle; only the API-backed hetzner_running_servers refresh is gated. Before this, a 429 window froze the reap-health gauges at their pre-window values, hiding a newly immortal agent or pinning a stale alert.

v103.percona.29 (2026-08-07)

Fixes shutdown-policy retention loss on deserialization, which produced immortal workers, and adds two reap-health gauges so the failure mode is alertable instead of silent.

Root cause: AbstractShutdownPolicy.retentionStrategy is transient and only assigned in the constructor, which XStream bypasses. A HetznerCloud loaded from the controller's persisted config.xml therefore carries IdlePeriodPolicy instances with idleMinutes intact but a null wrapped CloudRetentionStrategy. HetznerServerAgent bakes template.getShutdownPolicy().getRetentionStrategy() into the node at creation, and core Slave.getRetentionStrategy() maps a null field to RetentionStrategy.Always, so every agent provisioned or rehydrated from a deserialized template is never reaped. Observed fleet-wide on 2026-08-07: cloud.cd 16/16 agents immortal (oldest 58 days, disks at 100% from accumulated buildx debris), ps80.cd 13/13, pxc.cd 2/2. This is the deserialization bug class from v103.percona.1/.25/.27 surfacing in the shutdown path.

Fix: IdlePeriodPolicy.readResolve() rebuilds the instance so the transient strategy is restored after deserialization, and the HetznerServerAgent constructor falls back to the default idle policy when a template still hands it a null strategy (belt and braces; BeforeHourWrapsPolicy already returns its strategy singleton from the getter and is immune). Existing immortal agents are not fixed by upgrading: cure them in place via Script Console (setRetentionStrategy(new CloudRetentionStrategy(idleMinutes))) or recycle them.

Metrics: hetzner_agents_retention_missing (agents whose effective retention is Always; should be constant 0) and hetzner_agents_idle_overdue (online idle agents past twice their idle-shutdown period; catches any reap-failure mode by symptom, including a dead ComputerRetentionWork timer). Both are per-cloud gauges refreshed by the existing 1-minute HetznerMetricsRefresher pass.

v103.percona.28 (2026-06-01)

Resolves SSH credentials by the SSHUserPrivateKey interface instead of the concrete BasicSSHUserPrivateKey, so credentials served by alternative providers (notably the AWS Secrets Manager credentials provider, which exposes AwsSshUserPrivateKey) are visible to the Hetzner cloud. This matches the contract the EC2 and EC2-Fleet plugins already use.

Root cause: Helper.assertSshKey looked up CredentialsProvider.lookupCredentialsInItemGroup(BasicSSHUserPrivateKey.class, ...), filtering on the concrete implementation class. AwsSshUserPrivateKey implements the SSHUserPrivateKey interface but does not extend BasicSSHUserPrivateKey, so it was filtered out at the source and provisioning failed with IllegalStateException: No SSH credentials found with ID '<id>', even though the credential resolved correctly for every other consumer. The concrete-class lookup also hid such credentials from the connector's config dropdown.

Fix: look up the SSHUserPrivateKey interface at all four sites (Helper.assertSshKey, the two HetznerServerComputerLauncher connection paths, and the AbstractHetznerSshConnector config dropdown), and read the key material via the interface getPrivateKeys() accessor rather than the deprecated singular getPrivateKey(). The change is strictly widening: every BasicSSHUserPrivateKey is an SSHUserPrivateKey, so existing direct-entry and JCasC keys keep working unchanged; only interface-only implementations gain visibility. The SSHAuthenticator<Connection, ...> generic already binds U extends StandardUsernameCredentials, so it accepts the interface without change, and the alternative provider's lazy value fetch is preserved (the key is read only at connect time). Candidate for upstream contribution to jenkinsci/hetzner-cloud-plugin, which carries the same concrete-class coupling.

v103.percona.27 (2026-05-31)

Scopes ghost-node cleanup to the owning cloud, closing a multi-cloud mass-deletion regression. Ported from upstream commit 5a7a304, a fix that upstream layered on top of our own CRW-timer-death patch (contributed back to jenkinsci and shipped there as v106 / commit 796d19b).

Root cause: the bi-directional OrphanedNodesCleaner (added in v103.percona.1) compares Helper.getHetznerAgents(), which returns agents from ALL Hetzner clouds, against fetchAllServers(cloud.name), which returns VMs for a single cloud. On a controller configured with more than one HetznerCloud, the per-cloud cleanup pass treats every agent owned by every other cloud as a "ghost node" and removes it, killing active builds every hour. The transient cloud field on HetznerServerAgent is null after deserialization, so the cleaner could not previously tell which cloud owns which agent.

Fix: add a persistent cloudName field to HetznerServerAgent (survives restart/deserialization; serialVersionUID 1 to 2) and scope the ghost-node comparison to agents owned by the cloud being cleaned, so each cloud's pass only considers its own agents. Ownership is resolved by OrphanedNodesCleaner.ownerCloudName(): the persistent cloudName field, falling back to the cloud name carried by the persistent provisioningId for agents provisioned before that field existed. This goes one step beyond upstream's port, which skips (and therefore leaks) legacy agents whose cloudName is null; the provisioningId fallback lets the owning cloud still reap them.

The Percona fleet runs one HetznerCloud per master, so the regression is latent today; the fix ships as defense-in-depth before any multi-cloud topology lands. Tests: OrphanedNodesCleanerTest proves ghost removal is scoped per cloud and that legacy (null-cloudName) agents are attributed via provisioningId; HetznerServerAgentTest covers the new field plus the v103.percona.1 _terminate()/isAlive() null-transient guards, which the fork previously had no test for.

v103.percona.26 (2026-05-23)

Closes the OPEN-breaker API storm path uncovered by the 2026-05-22 cax arm64 shortage incident. With all three Hetzner arm64 DCs OPEN on ps80.cd (cax capacity event, same pattern as PS-11149), the plugin sustained ~35 API requests/minute trying to provision against unhealthy DCs. That single master alone drained the Hetzner project-level API budget shared across the whole fleet, dropping hetzner_api_rate_limit_remaining from ~3599 to single-digit headroom on all 10 masters in ~30 minutes (cross-master Pearson r = +0.96 over the 21:00-22:30 UTC window).

Root cause: DcHealthTracker.sortByHealth sorts matching templates by breaker health but does NOT filter unhealthy ones, so HetznerCloud.provision() picked rankedTemplates.get(0) even when all were OPEN; NodeCallable.call() then iterated the full list calling createServer() per template with no per-template isHealthy() gate; and the HALF_OPEN probe was not single-use, so concurrent provision() calls stampeded the API on every reset window.

Added

  • DcHealthTracker.filterHealthy(List<HetznerServerTemplate>) and DcHealthTracker.isHealthy(HetznerServerTemplate) helpers. Used by HetznerCloud.provision() to short-circuit before effectiveNodeCount() (the fetchAllServers API call) when all matching templates' breakers are OPEN. sortByHealth is preserved as-is for diagnostic visibility.
  • DcCircuitBreaker.isProbeable() (non-consuming peek) and DcCircuitBreaker.tryAcquireProbe() (consuming lease acquisition). Replaces the previous single isHealthy() method. isProbeable() is used by filterHealthy and sortByHealth so a filter pass cannot steal the HALF_OPEN probe lease from the actual provisioner. tryAcquireProbe() is called inside NodeCallable.call() at the per-template gate (the actual createServer attempt site); it consumes the lease so exactly one concurrent caller per HALF_OPEN window reaches the Hetzner API. Bounds the per-reset-window API spend on a still-broken DC.
  • DcCircuitBreaker.HALF_OPEN_STALE_TTL_MS (2 * RESET_TIMEOUT_MS = 10 minutes). If a probe-holder consumes the lease and then dies (thread crash, non-DC-attributable exit path that does not call recordSuccess / recordFailure), the lease is re-armed on the next probe call after the TTL elapses. Prevents the breaker from being pinned in HALF_OPEN forever. New counter hetzner_dc_health_stale_half_open_resets_total{location, arch} observes how often this fires.
  • HetznerMetricProvider.REASON_* constant family for PROVISION_SKIPPED reasons (jenkins_quieting, rate_limited, template_suppressed, cap_reached, and new no_healthy_dc), plus ALL_PROVISION_SKIPPED_REASONS enumeration mirroring the existing outcome convention.
  • HetznerMetricProvider.OUTCOME_DC_BREAKER_OPEN = "dc_breaker_open" on PROVISION_ATTEMPTS. Emitted by NodeCallable's per-template gate when a template's breaker is OPEN at iteration time. Added to both ALL_PROVISION_OUTCOMES and PRECHECK_OUTCOMES (no boot work performed) so dashboards do not pollute boot-duration percentiles.

Changed

  • HetznerCloud.provision() now filters matchingTemplates to healthy templates via DcHealthTracker.filterHealthy before calling effectiveNodeCount(). If empty, increments PROVISION_SKIPPED{reason="no_healthy_dc"} and breaks the excessWorkload loop. Filtering is per-loop-iteration so a breaker that trips during this provision() call short-circuits the next iteration. The filtered (healthy) list is passed into rankTemplatesByHealth and then into NodeCallable, so the inner failover loop only ever iterates healthy DCs.
  • HetznerCloud.provision() 4 existing PROVISION_SKIPPED.labels(name, "<inline-literal>") call sites refactored to use the new REASON_* constants. Wire-format identical (same string values).
  • NodeCallable.call() per-template breaker gate: at iteration head, before any createServer call, call DcHealthTracker.tryAcquireProbe(template) (the consuming variant that takes the HALF_OPEN probe lease at the actual API attempt site). If false (CLOSED is always true; OPEN or HALF_OPEN-lease-taken returns false), emit PROVISION_ATTEMPTS{outcome="dc_breaker_open"}, respect the existing isFailoverCompatibleWith invariant before advancing to the next ranked template (same gate as the DC-attributable failure branch), and continue if compatible. Belt-and-suspenders for breakers that open between provision()'s filter and the per-template iteration.
  • DcCircuitBreaker.isHealthy() removed in favour of the non-consuming / consuming split (see Added). Callers in DcHealthTracker and tests updated. Behaviour preserved for the consuming callers (tryAcquireProbe() is exactly the old isHealthy() semantics); filterHealthy / sortByHealth now use the non-consuming isProbeable() so a filter pass cannot steal the HALF_OPEN probe.
  • DcCircuitBreaker.getState() lazy-reset path also arms the probe lease, so a script-console getter call followed by a tryAcquireProbe() in the same HALF_OPEN window correctly gives the probe to the provisioner.

Explicitly not changed

  • OrphanedNodesCleaner.cleanCloud(): hourly fetchAllServers must run during a DC outage to reap orphan VMs and remove ghost Jenkins nodes. Keep the token-level isRateLimited() gate. 10 calls/hour fleet-wide, not a storm contributor.
  • HetznerMetricsRefresher.refreshCloud(): 60 calls/hour per cloud is load-bearing for the dashboard freshness contract; gating on breakers would mask real outages with stale gauges.
  • HungBuildDetector: does not call the Hetzner API.
  • DcHealthTracker.sortByHealth: unchanged contract (sort, do not filter); useful for diagnostic visibility and pinned by existing tests.

Tests

  • New DcCircuitBreakerTest.halfOpenIsSingleProbe, halfOpenLeaseRearmsAfterReopening, getStateLazyResetArmsProbeLease for the HALF_OPEN probe-lease invariants.
  • New DcHealthTrackerTest.filterHealthy{DropsOpenBreakers,AllUnhealthyReturnsEmpty,IsArchScoped,HandlesUnknownArch,HandlesEdgeCases}.
  • New HetznerCloudSimpleTest.provisionSkipsWhenAllMatchingDcsUnhealthy (storm-gate regression: asserts ZERO fetchAllServers / createServer calls when all matching breakers are OPEN, plus PROVISION_SKIPPED{reason="no_healthy_dc"} == 1, plus PROVISIONING_PENDING stays at 0).
  • New HetznerCloudSimpleTest.provisionProceedsWhenOneHealthyDcExists (gate is not over-zealous: one healthy DC keeps provisioning live).
  • New NodeCallableRetryTest.breakerOpenSkipsTemplate, breakerOpenAllSkippedThrowsIllegalState, breakerOpenSkipRespectsFailoverCompat.
  • Updated HetznerMetricProviderTest.provisionOutcomesEnumerationIsExhaustive to include dc_breaker_open (size 11 -> 12).
  • New HetznerMetricProviderTest.provisionSkippedReasonsEnumerationIsExhaustive pinning the 5 REASON_* constants.
  • HetznerCloudSimpleTest + NodeCallableRetryTest now reset HetznerMetricProvider in @BeforeEach/@AfterEach so the new counter assertions cannot flake.

Compatibility

  • No persistence-schema changes ($JENKINS_HOME/hetzner-dc-health.xml format unchanged). Rolling back to v25 is mechanically clean; Mimir may retain v26 label series briefly (no_healthy_dc, dc_breaker_open) but no series is removed.
  • Wire-format PROVISION_SKIPPED reason strings are unchanged (cap_reached, rate_limited, etc.); only the call-site spelling swapped from inline literals to constants.
  • Backward compatible with v25 dashboards. Adds two new label values that will appear naturally during the next forced-open canary test on ps3.

v103.percona.25 (2026-05-21)

Two related fixes that ship together so v25 is the canonical post-incident plugin across the fleet.

Part A: Transient field rehydration in HetznerCloud

XStream's Unsafe.allocateInstance bypasses field initializers on deserialized clouds (plugin dynamic reload, fresh-EBS rebuild). The pendingProvisions AtomicInteger and seenArchExtras set could be null on the first provisioning / refresh tick, NPE'ing the cap accounting and the lazy-arch path. v25 adds ensurePendingProvisions() / ensureSeenArchExtras() lazy guards plus a readResolve() re-init pass (belt-and-suspenders).

Part B: Per-arch DC circuit breakers

DcCircuitBreaker and DcHealthTracker keys widened from a single location string to a composite <location>:<arch> so an ARM-only capacity event does not poison the AMD64 breaker for the same DC. Legacy XML (pre-v25 single-location format) is migrated on first load: each legacy entry is cloned into one breaker per arch in ALWAYS_EMIT_ARCHS. New counter hetzner_dc_health_legacy_keys_migrated_total{location, arch} makes the migration auditable.

Tests

  • New DcHealthPersistenceTest covers the legacy-key migration path end-to-end.
  • DcHealthTrackerTest.archIndependenceWithinSameDc pins the invariant that ARM failures in fsn1 do not trip the AMD64 breaker for fsn1.
  • HetznerCloudRehydrateTest.{readResolveRehydratesBothTransientFields, readResolveIsIdempotent, ensurePendingProvisionsLazyInitialisesOnAccess} cover the Part A guards.

Compatibility

  • One-shot migration on first load: each legacy breaker entry produces N new entries (one per arch in ALWAYS_EMIT_ARCHS). Subsequent loads no-op.
  • Backward compatible with v23/v24 dashboards. Series with arch=arm64 (or arm64 + unknown after v24) are now distinct from the AMD64 versions; aggregations via sum by(...) are unchanged.

v103.percona.24 (2026-05-20)

Trims the arch="unknown" series so it only appears when a non-canonical Hetzner SKU is actually observed, rather than being emitted as a constant 0 on every refresh. v23 deployed to ps3 showed hetzner_running_servers{arch="unknown"} 0 permanently, which is background noise and a false signal for any alert that watches unknown > 0.

Changed

  • HetznerMetricProvider.ALWAYS_EMIT_ARCHS (new) replaces KNOWN_ARCHS as the iteration target in HetznerCloud.runningNodeCount(). Holds only amd64 and arm64 so those two get explicit zero-emission on every pass. KNOWN_ARCHS is kept as the full three-value catalogue for test/reset helpers.
  • HetznerCloud now tracks a per-instance seenArchExtras set (ConcurrentHashMap.newKeySet()). On each refresh, any non-canonical arch observed with a non-zero count is promoted into the set; subsequent passes keep emitting that arch (zero or otherwise) so a recent-incident marker persists for the JVM's lifetime. Cleared at restart, which is the right cadence (a strange-SKU incident weeks ago is not load-bearing if the fleet has been clean since).

Tests

  • HetznerMetricsRefresherTest.refreshMetrics_splitsByArch now asserts that arch="unknown" is absent from the registry when no unknown SKU has been observed (getSampleValue returns null), rather than asserting 0.
  • New refreshMetrics_lazilyEmitsUnknownArchOnceObserved walks the three-pass lifecycle: clean → unknown observed → unknown gone (series persists at 0).
  • New alwaysEmitArchsExcludesUnknown pin in HetznerMetricProviderTest.

Compatibility

  • Backward compatible with v23 dashboards: aggregations across arch are unchanged. Panels that explicitly select arch="unknown" will get no data until an unknown SKU appears, which is the desired behavior.

v103.percona.23 (2026-05-20)

Adds an arch label (amd64 / arm64 / unknown) to the two Hetzner-side metrics that carry per-server context, so the Grafana dashboard can split worker activity by CPU architecture. The Hetzner serverType naming convention is unambiguous: cpx* / cx* / ccx* are AMD64, cax* is ARM64. Everything else collapses to unknown so cardinality stays bounded if Hetzner adds a new SKU prefix.

Added

  • HetznerMetricProvider.archOf(String serverType) public helper plus the ARCH_AMD64 / ARCH_ARM64 / ARCH_UNKNOWN constants and a KNOWN_ARCHS list. Used by call sites that need to clear stale series (set 0 for archs that drop to zero so the gauge does not pin at its last non-zero value in Mimir).

Changed

  • hetzner_running_servers gains an arch label. HetznerCloud.runningNodeCount() groups VMs by arch from ServerDetail.getServerType().getName(), emits one row per arch in KNOWN_ARCHS, and explicitly sets 0 for archs with no running servers in the current pass.
  • hetzner_orphan_servers_reaped_total gains an arch label. OrphanedNodesCleaner.terminateOrphanedServer derives the arch from the doomed server's serverType at reap time.

Out of scope

  • hetzner_provisioning_pending stays cloud-only. Its backing state is a single AtomicInteger per cloud (HetznerCloud.pendingProvisions); splitting it by arch would require refactoring every provision-lifecycle code path (provision(), provisionCompleted(), NodeCallable) and threading an arch through the completion path. Tracked for a follow-up.
  • hetzner_instance_cap stays cloud-only. The cap is a single configured value per cloud with no per-arch decomposition; emitting it as cloud="x",arch="amd64"=50,cloud="x",arch="arm64"=50 would be semantically wrong (a sum would double the real cap).

Compatibility

  • Dashboards using sum by(cloud) or sum without(arch) over the relabelled metrics keep working unchanged; the arch dimension collapses cleanly at query time.
  • Dashboards using raw selectors (hetzner_running_servers{cloud="..."}) will now match multiple series (one per arch). The Percona dashboard at resources/addons/grafana/dashboards/hetzner-plugin.json already aggregates via sum by(master) so no panel breakage is expected.
  • Mimir cardinality impact: 10 masters × 3 archs (effective 2) for the two metrics = ~20-30 extra active series. Negligible against the baseline.

v103.percona.22 (2026-05-19)

Phase 4b of the ps3 canary resilience plan (PS-11173). After a master JVM restart, the plugin now re-adopts the master's own Hetzner VMs as Jenkins agents via Hetzner label selector, instead of letting OrphanedNodesCleaner reap them. This closes the failure class where every in-flight build was lost on a kill -9 / spot interrupt even though the Hetzner VMs themselves had survived. ps3-only canary; default-off on every other master.

Added

  • HetznerWorkerRehydrator.rehydrate() runs as @Initializer(after=JOB_LOADED, before=COMPLETED). For each configured HetznerCloud, it enumerates VMs via fetchAllServers(cloudName) (selector: jenkins.io/managed-by=hetzner-jenkins-plugin,jenkins.io/cloud-name=<cloud>), matches each VM to a HetznerServerTemplate, and Jenkins.addNodes a reconstructed HetznerServerAgent. Per-VM errors are caught and counted; the pass never bubbles to the boot sequence. Single INFO summary per cloud: rehydrated N/M VMs (skipped_existing=..., no_match=..., ambiguous=..., other=...).
  • HetznerConstants.LABEL_TEMPLATE_NAME = "jenkins.io/template-name". Written at provision time so the rehydrator can match a VM to its template with one exact lookup instead of falling back to the heuristic.
  • Heuristic fallback for VMs provisioned before v103.percona.22 (no template-name label): match by Hetzner serverType + datacenter/location + name prefix. Ambiguous matches log WARN, increment rehydrate_failures_total{reason=ambiguous}, and skip the VM.
  • Idempotency guard: skip VMs whose name already has a Jenkins.get().getNode(...) entry, so a re-entrant call cannot double-add.

Changed

  • HetznerCloudResourceManager.createLabelsForServer accepts an optional templateName; the new label lands on VMs provisioned in v103.percona.22+. fetchAllServers keeps the existing two-label selector (cloud-name only) so the same query matches both legacy and v103.percona.22+ VMs.
  • ControllerListener.onOnline defers OrphanedNodesCleaner.doCleanup() by hetzner.rehydrate.grace-period-minutes (default 5) when -Dhetzner.rehydrate.enabled=true. Otherwise the cleanup runs immediately as before. The defer gives the rehydrate pass time to re-adopt VMs before they are evaluated against the empty Jenkins-node list.

Feature flag

  • -Dhetzner.rehydrate.enabled=true to enable both the rehydrate @Initializer and the controller-online cleanup defer. Default false; ps3 master JVM args are the only opt-in for the canary.
  • -Dhetzner.rehydrate.grace-period-minutes=<n> to override the 5-minute defer (consulted only when the feature is on).

Compatibility

  • Fresh upgrade from v103.percona.21: no migration. Without the flag the new code paths are dormant.
  • Downgrade to v103.percona.21: VMs provisioned in v103.percona.22+ keep the template-name label, which prior versions ignore. Safe rollback.

Design decision: no external shared state

Three independent reviews (2 opus subagents + codex gpt-5.5 xhigh) converged: Phase 4b does not need DynamoDB. The jenkins.io/cloud-name=<master> label is a hard partition; each master rehydrates only its own VMs. Every piece of state needed at rehydrate is recoverable locally (Hetzner labels + Jenkins credentials on persistent EBS + in-memory template list). PS-11177 remains the home for any future cross-master shared state.

Added tests

  • HetznerWorkerRehydratorTest covering the matching logic: exact label match, heuristic by serverType + location + prefix, datacenter-vs-location forms, ambiguous twins, prefix disambiguation, no-match, stale-label fallthrough, empty/null template list. End-to-end validation lives on ps3 (kill -9 + boot log + node count + counters).

Added metrics

  • hetzner_rehydrated_workers{cloud} (gauge) - agents rehydrated on the most recent pass.
  • hetzner_rehydrate_attempts_total{cloud} (counter) - VMs evaluated.
  • hetzner_rehydrate_successes_total{cloud,template} (counter).
  • hetzner_rehydrate_failures_total{cloud,reason} (counter) - reason in {no_match, ambiguous, name_collision, add_node, other}.

v103.percona.21 (2026-05-19)

Phase 4a of the ps3 canary resilience plan (PS-11173). DC circuit-breaker state now survives a Jenkins JVM restart, which closes the failure class where a fresh master after spot interrupt would re-attempt every breaker as CLOSED and stampede a still-sick DC.

Added

  • DcHealthTracker.load() reads $JENKINS_HOME/hetzner-dc-health.xml at @Initializer(after=PLUGINS_STARTED) and rehydrates the in-memory BREAKERS map. Missing file is a silent no-op so a clean install or a downgrade to v103.percona.20 is unaffected.
  • DcHealthTracker.save() schedules a write via jenkins.util.Timer.get() whenever recordFailure or recordSuccess mutates state. An AtomicBoolean coalesces concurrent triggers so a burst produces a single write. Deferring to Timer keeps disk I/O off the synchronized breaker lock.
  • DcCircuitBreaker.afterLoad(fallbackLocation, now, staleOpenTtlMs) runs on every breaker after deserialization to restore Prometheus gauges and apply a 30-min TTL: an OPEN breaker whose openedAt is older than 30 min loads as CLOSED with zero consecutive failures, so a transient incident from before the restart does not pin a DC out of rotation.
  • @XStreamAlias("dcCircuitBreaker") on DcCircuitBreaker for a stable XML schema even if the class is later moved across packages.

Changed

  • DcCircuitBreaker.location is no longer final. XStream deserialization assigns it directly, and afterLoad() can fall back to the persisted map key if older XML omits the field. The @Getter and all synchronized accessors are unchanged.

Compatibility

  • Fresh upgrade from v103.percona.20: no migration needed; the XML file is created on first recordFailure/recordSuccess after upgrade.
  • Downgrade to v103.percona.20: the leftover hetzner-dc-health.xml is ignored (the prior version never reads it). Safe rollback path.
  • Persistence skips silently when Jenkins.getInstanceOrNull() returns null so existing unit tests that mock Jenkins.get() without stubbing getRootDir() continue to pass without touching the persistence layer.

Added tests

  • DcHealthPersistenceTest covering savesOnFailure, loadsOnInit, ttl_resetsStaleOpen, missingFile_isNoOp.

Added metrics

  • hetzner_dc_health_loaded_breakers (gauge) - number of breakers restored on the most recent DcHealthTracker.load() call.
  • hetzner_dc_health_saves_total (counter) - successful XmlFile writes.
  • hetzner_dc_health_save_failures_total (counter) - failed writes; non-zero rate is operationally actionable.
  • hetzner_dc_health_stale_open_resets_total{location} (counter) - breakers reset from OPEN to CLOSED on load due to the 30-min stale-OPEN TTL.

v103.percona.20 (2026-05-18)

Deduplicate server entries from paged Hetzner API responses.

Fixed

  • HetznerCloudResourceManager.fetchAllServers() deduplicates servers by ID across pages. The upstream API can return the same server in multiple pages during concurrent list + modify operations; duplicate entries inflated hetzner_running_servers and caused OrphanedNodesCleaner to attempt double-termination of the same VM.

v103.percona.19 (2026-05-18)

Extract Run from PlaceholderExecutable for Pipeline hung-build detection.

Fixed

  • HungBuildDetector now unwraps PlaceholderExecutable (the Jenkins Pipeline step executor wrapper) to reach the underlying Run.isBuilding() check. Without this, Pipeline builds were invisible to the detector and stuck Pipeline builds could run indefinitely without incrementing hetzner_stuck_builds_total.

v103.percona.18 (2026-05-18)

Codex review follow-up on v103.percona.17. The new HungBuildDetector and its three metrics shipped to ps3.cd as a canary; the review surfaced two correctness blockers and one observability fault that would have made fleet rollout silently misleading. v18 fixes all four before promoting beyond ps3.

Fixed

  • HungBuildDetector.scan() is now synchronized(this). The Jenkins PeriodicWork scheduler does not contractually forbid overlapping ticks if a single tick runs longer than the recurrence period (e.g. an executor walk that hangs on a stuck node lookup). The v17 implementation mutated the seen LinkedHashMap and called STUCK_BUILDS_TOTAL.inc() without a lock; two overlapping ticks could double-count a single hung build. Wrapping the whole scan() body in synchronized(this) keeps the dedup contract honest.
  • hetzner_oldest_build_age_seconds now clears stale template children at the end of every tick. v17 populated templatesObserved but never used it; once a hung build finished, the gauge stayed pinned at the last observed peak forever, scaring operators into investigating builds that had already completed. New clearStaleAgeChildren() reads the family's existing samples via the simpleclient collect() API, and calls Gauge.remove(template) on any child whose template label is not in templatesObserved for the current tick.

Changed

  • Dropped the in-plugin master label from hetzner_stuck_builds_total, hetzner_oldest_build_age_seconds, and the renamed hetzner_jenkins_real_busy_executors gauge. Per ADR 0013 in the percona-ci-platform repo, master-side Grafana Alloy injects master="<inst>.cd" via relabel config on the push pipeline, so the plugin-side value was redundant; worse, Alloy's external_labels only fill MISSING labels, not existing empty ones, so the v17 fallback (empty string when JenkinsLocationConfiguration.getUrl() was unset) emitted series dashboards filtering master=~".+\\.cd" would silently drop. Removed HetznerMetricProvider.masterLabel() and the helper deriveMasterFromUrl() since neither has any other caller. All v16-and-earlier hetzner_* metrics already shipped without a plugin-emitted master label; v18 aligns the v17 newcomers with that convention.
  • Renamed hetzner_executor_busy_real to hetzner_jenkins_real_busy_executors. The original name broke the hetzner_<domain>_<thing> convention used by every other Jenkins-side metric in the plugin; the rename keeps the family namespace clean and groups the gauge alongside other hetzner_jenkins_* metrics. No live Grafana panel referenced the v17 name yet (panels 210/211 use hetzner_stuck_builds_total and hetzner_oldest_build_age_seconds), so this is a safe rename.

Added tests

  • HungBuildDetectorTest.concurrent_scan_does_not_double_count -- spawns two threads racing on scan() with the same stuck build; asserts STUCK_BUILDS_TOTAL increments exactly once. Pins the v18 blocker-1 fix.
  • HungBuildDetectorTest.oldest_age_gauge_clears_when_build_finishes -- tick 1 has a hung build on unknown; tick 2 has no hung builds; asserts the gauge family contains NO samples for unknown after tick 2. Pins the v18 blocker-2 fix.
  • HungBuildDetectorTest.oldest_age_gauge_clears_when_template_no_longer_observed -- variant where a stale child for old-tpl is injected; the next tick must remove it while keeping the current template's child intact.
  • HungBuildDetectorTest.ttl_expiry_re_arms_dedup -- injects a fake clock, advances 8 days past the 7-day TTL, and asserts the same hung build re-arms the counter (otherwise a build hung for >7 days would silently stop counting).

Migration notes

  • Any Grafana queries or alerts referencing the master label on hetzner_stuck_builds_total, hetzner_oldest_build_age_seconds, or hetzner_executor_busy_real must drop the master= filter. Alloy adds the same master="<inst>.cd" label downstream via relabel config on the push pipeline, so any external query that previously matched on the in-plugin label keeps working at the Mimir side once that filter is removed (Alloy's value is identical to what the in-plugin helper would have produced when JenkinsLocationConfiguration.getUrl() was set).
  • Grafana queries or alerts referencing hetzner_executor_busy_real must be updated to hetzner_jenkins_real_busy_executors. Within Percona's repos there are no live references to the v17 name; the rename is a pre-emptive cleanup.

v103.percona.17 (2026-05-16)

Detect long-running ("hung") builds where Run.isBuilding() keeps returning true for days, and emit Prometheus metrics so Mimir surfaces the condition before it accretes into a fleet-wide outage.

Fixed

  • New HungBuildDetector (PeriodicWork, period: 10 minutes by default) iterates the controller's executors on every tick, identifies still-building runs older than the configured threshold (24h by default), and increments a dedup-cached counter for each unique build. Without this, the v103.percona.16 + CLI executors --real pipeline trusted Run.isBuilding() to filter zombie executors; production traffic on rel.cd and pmm.cd revealed runs reporting isBuilding=true for 3.9 to 6.4 days, stalling the idle-master plugin-upgrade cron for 6+ hours on 2026-05-15/16.
  • A new gauge hetzner_executor_busy_real{master} mirrors the CLI's --real count plugin-side, so readiness/is-idle probes can hit /hetzner-prometheus directly instead of round-tripping a Groovy snippet through Script Console for every check.

Why

  • The CLI's executors --real flag (shipped 2026-05-16) is the load-bearing filter for the idle-master cron. Trusting Run.isBuilding() alone is insufficient: layering an independent age threshold on top is necessary to distinguish real work from corrupted run state. The plugin owns the data Jenkins doesn't honestly report through its own APIs.

Added metrics

  • hetzner_stuck_builds_total{master, template, threshold_hours} -- Counter. Incremented exactly once per (build, threshold) when the build first crosses the threshold; a 7-day dedup TTL covers the longest plausible stuck-build lifetime before manual intervention. Use increase() / rate() over the total since a JVM restart drops the dedup cache.
  • hetzner_oldest_build_age_seconds{master, template} -- Gauge. Set on every tick to the max elapsed time among still-building runs grouped by Hetzner template. Templates without in-flight builds at the tick boundary are absent from the series for that emit cycle.
  • hetzner_executor_busy_real{master} -- Gauge. Count of executors whose currently-executing Queue.Executable is a Run with isBuilding=true.

Configuration knobs (system properties)

  • hetzner.hung-build.threshold-hours (default 24) -- a build older than this is classified hung. Honored fresh on each tick.
  • hetzner.hung-build.poll-period-minutes (default 10) -- detector recurrence period. Read by Jenkins scheduler on each reschedule.
  • hetzner.master.label (operator override) -- the master label value emitted on the new metrics. Falls back to the leading hostname of JenkinsLocationConfiguration.getUrl() with a .cd suffix (e.g. rel.cd), then empty (Alloy's external_labels on the push pipeline adds the value downstream).

Implementation notes

  • Iterates Jenkins.getComputers()[].getExecutors() rather than Jenkins.getAllItems(Run.class): the executor-walking path is bounded by the executor count (typically <100 per master) and avoids touching cold historical run state.
  • templateOf(Computer) walks computer.getNode() and casts to HetznerServerAgent to recover the template name. Non-Hetzner agents (built-in controller, EC2, manually-added nodes) and post-restart deserialized agents (template is transient) report template="unknown"; dashboards should filter template!="unknown" on per-template panels.
  • A threshold tweak via system property re-arms the counter at the new boundary (the dedup key includes thresholdHours), but does NOT reset the existing dedup cache. A build that already crossed 24h does not re-arm if the threshold drops to 12h. Restart the JVM (or wait 7 days for cache expiry) to re-arm such builds explicitly.
  • Defensive try/catch(Throwable) in doRun() mirrors HetznerMetricsRefresher / OrphanedNodesCleaner: a Jenkins-core or Hetzner-API exception cannot kill the timer.

v103.percona.16 (2026-05-15)

Periodically refresh hetzner_running_servers and hetzner_provisioning_pending regardless of provisioning activity.

Fixed

  • hetzner_running_servers no longer pins to the last-known value when a cloud is idle. New HetznerMetricsRefresher (PeriodicWork, period: 1 minute) walks every configured HetznerCloud and calls a new public HetznerCloud.refreshMetrics() method, which re-queries the Hetzner API for the live server count and re-emits the in-memory pending counter. Observed before the fix: psmdb.cd reported hetzner_running_servers=25 in Mimir while the Hetzner API showed only 2 running servers; the cloud had drained from a recent peak but no new provision attempts triggered a gauge update, so the metric was stuck for hours.
  • The new refreshMetrics() method is also called defensively against PROVISIONING_PENDING from the in-memory pendingProvisions AtomicInteger, so a forgotten set(...) after an atomic mutation does not desync the gauge.

Why

  • Field deployment of v103.percona.15 to the 10-master Percona Jenkins fleet exposed the discrepancy when validating the new dashboard panel 200 ("Active workers per master") at 2026-05-15 09:30 UTC. The new panel would show 25/100 utilisation for psmdb when the truth was 2/100, making the dashboard misleading.

Rate-limit hygiene

  • Per-cloud refresh consumes one Hetzner API call per minute (60/hour). Token budget is 3600/hour, so even 15 templates across 10 masters share the cost comfortably. The refresher skips refresh if the credentials' rate-limiter is open, same pattern as OrphanedNodesCleaner.

v103.percona.15 (2026-05-15)

Surface the actual running Jenkins core version on hetzner_plugin_info.

Changed

  • hetzner_plugin_info metric label jenkins_baseline renamed to jenkins_version. The label value was previously the hard-coded string "2.479" (matching <jenkins.baseline> in pom.xml only by convention); it now reads from Jenkins.getVersion().toString() at class init, so it reports the real running core version on each master (e.g. 2.528.3, 2.541.3). Fully guarded with try/catch; value is "unknown" if the call throws or returns null.

Why

  • Field deployment of 103.percona.14 to the 10-master Percona Jenkins fleet exposed that every master emitted jenkins_baseline="2.479" regardless of its actual running core version, which made the label useless for dashboards intending to break out per-core-version behaviour.

Migration note

  • Any Grafana queries or alerts referencing the old jenkins_baseline label must be updated to jenkins_version. Within Percona's repos this is only the panel-104 transform renameByName in percona-ci-platform/resources/addons/grafana/dashboards/hetzner-plugin.json.

v103.percona.14 (2026-05-12)

Wait for cloud-init to finish before launching the remoting JVM.

Fixed

  • HetznerServerComputerLauncher generates an .agent.start.sh that calls cloud-init status --wait (when present) and verifies java exists on PATH before exec'ing the remoting JVM. Previously the script ran java -jar remoting.jar immediately; on stock images (e.g. Hetzner debian-12) where the user-data script installs openjdk via apt during cloud-init's modules-final stage, java was not yet on PATH and the channel EOFed instantly. Symptom: chronic java.io.EOFException: unexpected stream termination at HetznerServerComputerLauncher.launchAgent with a high bootstrap_io rate (verified on pg.cd and psmdb.cd; cpx62 reproduction showed cloud-init modules-final taking ~49 seconds during which which java returned not-found).
  • The script logs to stderr only (stdout is the remoting channel) and exits with a clear nonzero code on cloud-init failure or missing java, instead of letting the EOF surface as an opaque bootstrap failure.
  • Controller-side logger.info line in launchAgent() now states that the launch script waits for cloud-init, so the Jenkins log shows expected behavior even when remote stderr is not wired through.

Context

  • Pre-baked images (no cloud-init on PATH, java already installed) skip the wait and fall straight through. The aarch64 snapshot path is unaffected.
  • Outer bound is unchanged: NodeCallable.doProvision still wraps computer.connect(false) in a Future bounded by the template's bootDeadline minutes.
  • A long-term follow-up is to pre-bake a Hetzner snapshot with java/docker/awscli installed; that is orthogonal to this fix and will reduce provisioning latency by the cloud-init modules-final time on top of fixing the race.

v103.percona.13 (2026-05-12)

Justfile pin bump.

Changed

  • Bumped justfile version pin to 103.percona.13. No code changes.

v103.percona.12 (2026-05-12)

NodeCallable hardening and metrics endpoint follow-ups. Documented intermediate with no standalone git tag; these changes shipped under the v103.percona.13 release.

Fixed

  • NodeCallable: loopback IP gate prevents connecting to a server whose address resolves to 127.x.
  • DC health gate added at NodeCallable.call() entry (belt-and-suspenders alongside the provision() pre-check from v3).
  • Provisioning outcome enum extended; metrics endpoint loopback follow-ups from v9/v10 review.

v103.percona.11 (2026-05-08)

Make /hetzner-prometheus an UnprotectedRootAction so anonymous loopback callers are not blocked by Jenkins core authorization.

Changed

  • HetznerPrometheusEndpoint now implements UnprotectedRootAction (instead of RootAction). v103.percona.10 dropped the Jenkins.SYSTEM_READ check inside doIndex(), but the request was still rejected with HTTP 403 by GlobalMatrixAuthorizationStrategy before ever reaching doIndex(). Verified anonymous curl http://127.0.0.1:8080/hetzner-prometheus returned 403 on ps3.cd with v103.percona.10 active.
  • Javadoc updated to reflect that the trust boundary is the Jenkins 8080 loopback bind, not core ACLs.

Context

  • Required for ADR 0013's master-side Alloy push model (PS-10997 Phase 2). The Alloy systemd unit on each EC2 master scrapes the endpoint with no credentials; auth lives at the in-cluster alloy-gateway instead.

v103.percona.10 (2026-05-07)

Drop SYSTEM_READ permission gate on /hetzner-prometheus for the push-model rollout.

Changed

  • HetznerPrometheusEndpoint.doIndex() no longer calls Jenkins.get().checkPermission(Jenkins.SYSTEM_READ). The endpoint is unreachable from external clients (Jenkins binds 8080 to 127.0.0.1 on Percona masters); the master-side Grafana Alloy systemd unit is the only consumer. Auth lives at the in-cluster alloy-gateway (NGINX bearer-token sidecar + ALB inbound-CIDRs allowlist), not at the Jenkins endpoint.
  • Javadoc updated to reflect the localhost-only contract and reference ADR 0013 / PS-10997 Phase 2.

Context

  • Companion repo: nogueiraanderson/percona-ci-platform (alloy-gateway addon).
  • Supersedes the prior plan that ran a prom-scraper-svc Jenkins user with API-token basic auth for in-cluster Prometheus to scrape this endpoint over public DNS.

v103.percona.9 (2026-05-07)

Self-contained /hetzner-prometheus Stapler endpoint (PS-10997 Phase 1).

Added

  • HetznerPrometheusEndpoint (RootAction at /hetzner-prometheus): exposes 40+ hetzner_* metric families (DC circuit breaker state, provisioning latency, API rate-limit headroom, CRW iterations, template suppression, anomaly counters) in Prometheus 0.0.4 text format.
  • io.prometheus:simpleclient bundled directly in the plugin, no dependency on the community prometheus-plugin (fleet audit: 0/10 masters had it installed).
  • HetznerMetricProvider: registers all hetzner_* collectors against CollectorRegistry.defaultRegistry.

v103.percona.8 (2026-05-07)

Initial Prometheus metrics scaffolding (PS-10997).

Added

  • Initial hetzner_* metric definitions and HetznerMetricProvider scaffolding: exposes plugin state (provisioning counts, rate-limit headroom, DC breaker state) via Prometheus gauges/counters. Superseded by the self-contained endpoint in v9.

v103.percona.7 (2026-04-07)

SSH retry backoff, log level fixes, and robustness cleanups.

Fixed

  • SSH launcher: retry backoff for transient connection failures during agent launch, reducing spurious bootstrap failures on VMs with slow SSH startup.
  • Noisy log entries lowered to DEBUG; improved context in error messages across the provisioning path.
  • Miscellaneous null guards and defensive catches in the boot and teardown paths.

v103.percona.6 (2026-04-07)

Rate-limit code review follow-ups.

Fixed

  • Addressed Codex static analysis findings on HetznerApiClient and RateLimitInterceptor (v4): missing null checks, rate-limit state transition clarity, exception message alignment with the v1 convention.

v103.percona.5 (2026-04-01)

Retry infrastructure and template error suppression.

Added

  • RetryInterceptor: OkHttp interceptor with exponential backoff and jitter for transient errors (429, 502, 504, socket timeouts). Max 3 retries, honors Retry-After header.
  • TemplateErrorTracker: suppresses templates with persistent config errors (e.g., deprecated image IDs). 3-failure threshold, 30-minute suppression, half-open probe on expiry.
  • Config error detection (invalid_input) aborts DC failover immediately.
  • Rate-limit (429) during DC failover aborts the entire provisioning attempt (token-scoped).

Fixed

  • Boot status polling skips API calls when rate-limited.
  • Improved log coverage, context, and consistency across rate-limit paths.

v103.percona.4 (2026-03-22)

Rate-limit infrastructure and API caching.

Added

  • HetznerApiClient: per-token API client wrapper with rate-limit state tracking, lazy auth via AuthInterceptor, and HTTP 401 client invalidation for token rotation support.
  • RateLimitInterceptor: OkHttp interceptor parsing RateLimit-Limit/Remaining/Reset headers from every API response.
  • Guava response caches: SSH keys (30min TTL), label IDs (15min TTL), server lists (30sec TTL) with recordStats().
  • checkRateLimit() guard before API-intensive operations (createServer, getOrCreateSshKey, fetchAllServers).
  • HetznerCloudResourceManager.getCacheStats() for Script Console observability.
  • OrphanedNodesCleaner skips cleanup cycle when rate-limited.

Fixed

  • destroyServer() catches Exception (not just IOException), hardening the CRW timer death fix for all exception types.

v103.percona.3 (2026-03-19)

Per-datacenter circuit breaker failover.

Added

  • DcCircuitBreaker: per-DC state machine (CLOSED / OPEN / HALF_OPEN). 2-failure threshold trips breaker for 5 minutes, auto-reset to HALF_OPEN for single probe.
  • DcHealthTracker: static registry of breakers per location with sortByHealth() for template ranking. Shuffles within health partitions for load distribution.
  • HetznerProvisioningException: typed RuntimeException carrying HTTP status, Hetzner error code, and DC location. Methods: isRateLimited(), isConfigError(), isResourceUnavailable().
  • HetznerCloud.rankTemplatesByHealth() replaces random pickTemplate().
  • NodeCallable rewritten with DC failover loop: iterates ranked templates, records success/failure in DcHealthTracker, cleans up leaked servers on bootstrap failure.
  • CLI observability via jenkins hetzner commands (health, status, nodes, servers, templates, version, orphans, reset, trip).
  • dc-health-check.groovy for Script Console inspection.

Changed

  • Helper.assertValidResponse() throws HetznerProvisioningException on error responses (typed instead of generic IllegalStateException).

v103.percona.2 (2026-03-19)

Architecture validation, null-safety, and deployment tooling.

Added

  • Architecture validation via NodeCallable.inferArchFromServerType() (CAX* = arm64, else x86_64) and post-boot uname -m check via UnameCallable.
  • Deploy (scripts/deploy.sh), check (scripts/check.sh), and verify (scripts/verify.sh) scripts.
  • justfile recipes: deploy, deploy-all, check, verify.

Fixed

  • Launcher null-safety across DefaultConnectionMethod, DefaultV6ConnectionMethod, PublicAddressOnly, PublicV6AddressOnly with descriptive error messages instead of NPE.
  • AbstractByLabelSelector: .findFirst().get() replaced with .findFirst().orElseThrow() with descriptive error including selector and location.
  • OrphanedNodesCleaner: removed setTemporarilyOffline() call for ghost nodes, added race guard.

v103.percona.1 (2026-03-19)

Retention bug fixes for idle VM accumulation.

Fixed

  • CRW timer death (critical): destroyServer() wrapped IOException in unchecked IllegalStateException, killing the ComputerRetentionWork periodic timer permanently. Changed to log-and-return. Confirmed: CRW was dead for 28 hours on two production instances. This fix was later contributed back to upstream and ships in jenkinsci/hetzner-cloud-plugin as v106 (commit 796d19b).
  • One-directional orphan cleanup: OrphanedNodesCleaner now removes both VMs without Jenkins nodes AND Jenkins nodes without VMs (ghost nodes). Per-item try-catch prevents one failure from blocking cleanup of remaining items.
  • Null transient fields after restart: cloud, template, serverInstance are transient and null after deserialization. Added null guards in _terminate(), isAlive(), getDisplayName().
  • HetznerCloudResourceManager.refreshServerInfo() throws IOException (checked) instead of IllegalStateException.
  • Helper.assertValidResponse() null body guard.