Skip to content

chore(ci): Add script to cancel SauceLabs jobs #5873

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

chore(ci): Add script to cancel SauceLabs jobs #5873

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 40 additions & 29 deletions .github/workflows/benchmarking.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,49 +130,60 @@ jobs:
SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
with:
script: |
const fs = require('fs');
const { execSync } = require('child_process');
console.log("::debug Checking if the test should be retried");

console.log("Extracting test ID from output log");
const outputLog = fs.readFileSync('output.log', 'utf8');
const path = require('path');
const { shouldRetryTest } = require('./scripts/saucelabs-helpers.js');

// Lookup for the test ID in the output log
// Note: The CLI output might change over time, so this might need to be updated.
const match = outputLog.match(/https:\/\/app\.saucelabs\.com\/tests\/([^\s]+)/);
const testId = match?.[1] ?? '';
const outputLogPath = path.join(__dirname, "output.log");
const result = await shouldRetryTest(outputLogPath);

if (!testId) {
core.warning("No SauceLabs test ID found in CLI output, it might have changed, retrying...");
core.setOutput('RETRY_TEST', 'true');

return;
}

try {
console.log(`Checking if the test exists in SauceLabs: ${testId}`);
execSync(`saucectl jobs get ${testId}`, {
env: process.env,
stdio: 'inherit'
});

console.log("Test exists but failed, not retrying.");
if (result.shouldRetry) {
console.log(`Test should be retried: ${result.reason}`);
core.setOutput('SHOULD_RETRY_TEST', 'true');
} else {
console.log(`Test exists but failed, not retrying: ${result.reason}`);
core.setFailed('Test exists but failed');
} catch (error) {
console.log("Failed to get job, retrying...");
core.setOutput('RETRY_TEST', 'true');
}

- name: Run Benchmarks in SauceLab - Retry 1
id: run-benchmarks-in-sauce-lab-retry-1
# Note: We need to use always() here, because the previous run step might be marked as failed.
if: ${{ always() && steps.should-retry-test.outputs.RETRY_TEST == 'true' }}
if: ${{ always() && steps.should-retry-test.outputs.SHOULD_RETRY_TEST == 'true' }}
env:
SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }}
SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
run: |
echo "::warning SauceLabs benchmark tests need to be retried"
saucectl run \
set -o pipefail && saucectl run \
--select-suite "${{matrix.suite}}" \
--config .sauce/benchmarking-config.yml \
--tags benchmark \
--verbose
--verbose \
2>&1 | tee retry-output.log

- name: Force Cancel SauceLabs Job on Workflow Cancellation
if: ${{ cancelled() }}
uses: actions/github-script@v7
env:
SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }}
SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
with:
# As soon as the workflow is cancelled, we attempt to cancel the SauceLabs jobs.
# This is to avoid wasting expensive device time on SauceLabs, as the results are not used anymore.
script: |
console.log("::warning Workflow was cancelled, attempting to cancel SauceLabs jobs based on the output logs");

const { cancelJobs } = require('./scripts/saucelabs-helpers.js');
const path = require('path');

const outputLogPath = path.join(__dirname, "output.log");
const retryOutputLogPath = path.join(__dirname, "retry-output.log");
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: GitHub Actions Script Path Resolution Error

The __dirname variable in the actions/github-script environment does not correctly resolve to the workflow's working directory. This causes incorrect paths when attempting to access log files like output.log and retry-output.log, which are created in the working directory. process.cwd() or relative paths should be used instead.

Additional Locations (1)
Fix in Cursor Fix in Web


const result = await cancelJobs([outputLogPath, retryOutputLogPath]);

if (result.success) {
console.log("::debug Successfully cancelled SauceLabs jobs");
} else {
console.error("::error Failed to cancel SauceLabs jobs");
}
Loading
Loading