Skip to content

Fix timeout when reusing older Brancher nodes #186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Copilot wants to merge 7 commits into
base: master
Choose a base branch
from
Draft

Fix timeout when reusing older Brancher nodes #186

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
aa827ec
Initial plan
Copilot Jan 7, 2026
fe41291
Add SSH-first availability check for reused Branchers
Copilot Jan 7, 2026
447b1d9
Fix timeout allocation and time tracking issues
Copilot Jan 7, 2026
c3e76e2
Address code review nitpicks
Copilot Jan 7, 2026
dbefe7a
Optimize sleep calls in SSH reachability check
Copilot Jan 7, 2026
2c3448e
Remove trailing whitespace
Copilot Jan 7, 2026
7e30c4b
Simplify SSH check to use fixed 10-second timeout
Copilot Jan 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
194 changes: 154 additions & 40 deletions src/Brancher/BrancherHypernodeManager.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@

class BrancherHypernodeManager
{
/**
* Ratio of timeout to allocate for initial SSH check when reusing a Brancher.
* The remaining time is reserved for logbook flow checks if SSH fails.
*/
private const REUSED_SSH_CHECK_TIMEOUT_RATIO = 0.5;

private LoggerInterface $log;
private HypernodeClient $hypernodeClient;

Expand Down Expand Up @@ -102,6 +108,69 @@ public function createForHypernode(string $hypernode, array $data = []): string
return $this->hypernodeClient->brancherApp->create($hypernode, $data);
}

/**
* Check if brancher Hypernode is reachable via SSH.
*
* @param string $brancherHypernode Name of the brancher Hypernode
* @param int $timeout Maximum time to wait for reachability
* @param int $reachabilityCheckCount Number of consecutive successful checks required
* @param int $reachabilityCheckInterval Seconds between reachability checks
* @return bool True if reachable, false if timed out
*/
private function checkSshReachability(
string $brancherHypernode,
int $timeout,
int $reachabilityCheckCount,
int $reachabilityCheckInterval
): bool {
$startTime = microtime(true);
$consecutiveSuccesses = 0;

while ((microtime(true) - $startTime) < $timeout) {
$connection = @fsockopen(sprintf("%s.hypernode.io", $brancherHypernode), 22);
if ($connection) {
fclose($connection);
$consecutiveSuccesses++;
$this->log->info(
sprintf(
'Brancher Hypernode %s reachability check %d/%d succeeded.',
$brancherHypernode,
$consecutiveSuccesses,
$reachabilityCheckCount
)
);

if ($consecutiveSuccesses >= $reachabilityCheckCount) {
return true;
}

// Only sleep if there's enough time remaining for another check
if ((microtime(true) - $startTime + $reachabilityCheckInterval) < $timeout) {
sleep($reachabilityCheckInterval);
}
} else {
if ($consecutiveSuccesses > 0) {
$this->log->info(
sprintf(
'Brancher Hypernode %s reachability check failed, resetting counter (was at %d/%d).',
$brancherHypernode,
$consecutiveSuccesses,
$reachabilityCheckCount
)
);
}
$consecutiveSuccesses = 0;

// Only sleep if there's enough time remaining for another check
if ((microtime(true) - $startTime + $reachabilityCheckInterval) < $timeout) {
sleep($reachabilityCheckInterval);
}
}
}

return false;
}

/**
* Wait for brancher Hypernode to become available.
*
Expand All @@ -121,6 +190,89 @@ public function waitForAvailability(
int $reachabilityCheckCount = 6,
int $reachabilityCheckInterval = 10
): void {
$this->waitForAvailabilityInternal($brancherHypernode, $timeout, $reachabilityCheckCount, $reachabilityCheckInterval, false);
}

/**
* Wait for reused brancher Hypernode to become available.
* For reused Branchers, first checks SSH connectivity before checking logbook flows,
* as older Branchers may not have recent logbook entries.
*
* @param string $brancherHypernode Name of the brancher Hypernode
* @param int $timeout Maximum time to wait for availability
* @param int $reachabilityCheckCount Number of consecutive successful checks required
* @param int $reachabilityCheckInterval Seconds between reachability checks
* @return void
* @throws CreateBrancherHypernodeFailedException
* @throws HypernodeApiClientException
* @throws HypernodeApiServerException
* @throws TimeoutException
*/
public function waitForReusedAvailability(
string $brancherHypernode,
int $timeout = 1500,
int $reachabilityCheckCount = 6,
int $reachabilityCheckInterval = 10
): void {
$this->waitForAvailabilityInternal($brancherHypernode, $timeout, $reachabilityCheckCount, $reachabilityCheckInterval, true);
}

/**
* Internal method to wait for brancher Hypernode to become available.
*
* @param string $brancherHypernode Name of the brancher Hypernode
* @param int $timeout Maximum time to wait for availability
* @param int $reachabilityCheckCount Number of consecutive successful checks required
* @param int $reachabilityCheckInterval Seconds between reachability checks
* @param bool $checkSshFirst For reused Branchers, check SSH first before logbook
* @return void
* @throws CreateBrancherHypernodeFailedException
* @throws HypernodeApiClientException
* @throws HypernodeApiServerException
* @throws TimeoutException
*/
private function waitForAvailabilityInternal(
string $brancherHypernode,
int $timeout,
int $reachabilityCheckCount,
int $reachabilityCheckInterval,
bool $checkSshFirst
): void {
// For reused Branchers, first check if SSH is already reachable
if ($checkSshFirst) {
$this->log->info(
sprintf(
'Checking if reused brancher Hypernode %s is already reachable via SSH...',
$brancherHypernode
)
);

// Allocate a portion of timeout for initial SSH check to leave time for fallback
$sshCheckTimeout = (int) ($timeout * self::REUSED_SSH_CHECK_TIMEOUT_RATIO);
Copy link
Member

@tdgroot tdgroot Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for fancy calculation, just try this for 10 seconds.

Copy link
Author

Copilot AI Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 7e30c4b - simplified to use a fixed 10-second timeout for the initial SSH check.

$sshCheckStartTime = microtime(true);

if ($this->checkSshReachability($brancherHypernode, $sshCheckTimeout, $reachabilityCheckCount, $reachabilityCheckInterval)) {
$this->log->info(
sprintf(
'Reused brancher Hypernode %s is already reachable!',
$brancherHypernode
)
);
return;
}

// Adjust timeout for remaining checks based on time already spent
$timeSpentOnSshCheck = microtime(true) - $sshCheckStartTime;
$timeout = max(1, (int) ($timeout - $timeSpentOnSshCheck));

$this->log->info(
sprintf(
'Reused brancher Hypernode %s is not yet reachable, proceeding with full availability wait...',
$brancherHypernode
)
);
}

$latest = microtime(true);
$timeElapsed = 0;
$resolved = false;
Expand Down Expand Up @@ -181,46 +333,8 @@ public function waitForAvailability(
);
}

$consecutiveSuccesses = 0;
while ($timeElapsed < $timeout) {
$now = microtime(true);
$timeElapsed += $now - $latest;
$latest = $now;

$connection = @fsockopen(sprintf("%s.hypernode.io", $brancherHypernode), 22);
if ($connection) {
fclose($connection);
$consecutiveSuccesses++;
$this->log->info(
sprintf(
'Brancher Hypernode %s reachability check %d/%d succeeded.',
$brancherHypernode,
$consecutiveSuccesses,
$reachabilityCheckCount
)
);

if ($consecutiveSuccesses >= $reachabilityCheckCount) {
break;
}
sleep($reachabilityCheckInterval);
} else {
if ($consecutiveSuccesses > 0) {
$this->log->info(
sprintf(
'Brancher Hypernode %s reachability check failed, resetting counter (was at %d/%d).',
$brancherHypernode,
$consecutiveSuccesses,
$reachabilityCheckCount
)
);
}
$consecutiveSuccesses = 0;
sleep($reachabilityCheckInterval);
}
}

if ($consecutiveSuccesses < $reachabilityCheckCount) {
$remainingTimeout = $timeout - $timeElapsed;
if (!$this->checkSshReachability($brancherHypernode, $remainingTimeout, $reachabilityCheckCount, $reachabilityCheckInterval)) {
throw new TimeoutException(
sprintf('Timed out waiting for brancher Hypernode %s to become reachable', $brancherHypernode)
);
Expand Down
23 changes: 17 additions & 6 deletions src/DeployRunner.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -296,8 +296,10 @@ private function maybeConfigureBrancherServer(Server $server, bool $reuseBranche

$data = $settings;
$data['labels'] = $labels;
$isReused = false;
if ($reuseBrancher && $brancherApp = $this->brancherHypernodeManager->reuseExistingBrancherHypernode($parentApp, $labels)) {
$this->log->info(sprintf('Found existing brancher Hypernode, name is %s.', $brancherApp));
$isReused = true;
} else {
$brancherApp = $this->brancherHypernodeManager->createForHypernode($parentApp, $data);
$this->log->info(sprintf('Successfully requested brancher Hypernode, name is %s.', $brancherApp));
Expand All @@ -306,12 +308,21 @@ private function maybeConfigureBrancherServer(Server $server, bool $reuseBranche

try {
$this->log->info('Waiting for brancher Hypernode to become available...');
$this->brancherHypernodeManager->waitForAvailability(
$brancherApp,
$timeout,
$reachabilityCheckCount,
$reachabilityCheckInterval
);
if ($isReused) {
$this->brancherHypernodeManager->waitForReusedAvailability(
$brancherApp,
$timeout,
$reachabilityCheckCount,
$reachabilityCheckInterval
);
} else {
$this->brancherHypernodeManager->waitForAvailability(
$brancherApp,
$timeout,
$reachabilityCheckCount,
$reachabilityCheckInterval
);
}
$this->log->info('Brancher Hypernode has become available!');
} catch (CreateBrancherHypernodeFailedException | TimeoutException $e) {
if (in_array($brancherApp, $this->brancherHypernodesRegistered)) {
Expand Down