Add support for manually running the real-time benchmark on PRs #19265
+190
−38
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,13 +2,44 @@ name: Real-time Benchmark | |
| on: | ||
| schedule: | ||
| - cron: "30 0 * * *" | ||
| workflow_dispatch: | ||
| inputs: | ||
| repository: | ||
| description: 'Repository name that is going to be benchmarked (e.g. "johndoe/php-src")' | ||
| required: true | ||
| type: string | ||
| branch: | ||
| description: 'Branch name that is going to be benchmarked (e.g. "my-branch")' | ||
| required: true | ||
| type: string | ||
| commit: | ||
| description: 'Commit SHA that is going to be benchmarked (e.g. "123456789a")' | ||
| required: true | ||
| type: string | ||
| baseline_commit: | ||
| description: 'A commit SHA that is compared against the one provided by the "commit" input (e.g. "abcdef123456")' | ||
| required: true | ||
| type: string | ||
| jit: | ||
| description: 'Whether JIT is benchmarked' | ||
| required: false | ||
| type: choice | ||
| options: | ||
| - "0" | ||
| - "1" | ||
| permissions: | ||
| contents: read | ||
| jobs: | ||
| REAL_TIME_BENCHMARK: | ||
| name: REAL_TIME_BENCHMARK | ||
| if: github.repository == 'php/php-src' || github.event_name == 'workflow_dispatch' | ||
| runs-on: ubuntu-22.04 | ||
| env: | ||
| REPOSITORY: ${{ inputs.repository || 'php/php-src' }} | ||
kocsismate marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| BRANCH: ${{ inputs.branch || 'master' }} | ||
| COMMIT: ${{ inputs.commit || github.sha }} | ||
| BASELINE_COMMIT: ${{ inputs.baseline_commit || 'd5f6e56610c729710073350af318c4ea1b292cfe' }} | ||
| JIT: ${{ inputs.jit || '1' }} | ||
| steps: | ||
| - name: Install dependencies | ||
| run: | | ||
|
|
@@ -29,73 +60,143 @@ jobs: | |
| ref: 'main' | ||
| fetch-depth: 1 | ||
| path: 'php-version-benchmarks' | ||
| - name: Checkout php-src (benchmarked version) | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: '${{ env.REPOSITORY }}' | ||
| ref: '${{ env.COMMIT }}' | ||
| fetch-depth: 100 | ||
| path: 'php-version-benchmarks/tmp/php_${{ env.BRANCH }}' | ||
|
||
| - name: Checkout php-src (baseline version) | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: '${{ env.REPOSITORY }}' | ||
| ref: '${{ env.BASELINE_COMMIT }}' | ||
| fetch-depth: 100 | ||
| path: 'php-version-benchmarks/tmp/php_baseline' | ||
| - name: Setup benchmark results | ||
| run: | | ||
| git config --global user.name "Benchmark" | ||
| git config --global user.email "[email protected]" | ||
|
|
||
| rm -rf ./php-version-benchmarks/docs/results | ||
| - name: Checkout benchmark data | ||
| if: github.repository == 'php/php-src' && github.event_name != 'workflow_dispatch' | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: php/real-time-benchmark-data | ||
| ssh-key: ${{ secrets.PHP_VERSION_BENCHMARK_RESULTS_DEPLOY_KEY }} | ||
| path: 'php-version-benchmarks/docs/results' | ||
| - name: Setup infra config | ||
| run: | | ||
| set -e | ||
|
|
||
| cp ./php-version-benchmarks/config/infra/aws/x86_64-metal.ini.dist ./php-version-benchmarks/config/infra/aws/x86_64-metal.ini | ||
| ESCAPED_DOCKER_REGISTRY=$(printf '%s\n' "${{ secrets.PHP_VERSION_BENCHMARK_DOCKER_REGISTRY }}" | sed -e 's/[\/&]/\\&/g') | ||
| sed -i "s/INFRA_DOCKER_REGISTRY=public.ecr.aws\/abcdefgh/INFRA_DOCKER_REGISTRY=$ESCAPED_DOCKER_REGISTRY/g" ./php-version-benchmarks/config/infra/aws/x86_64-metal.ini | ||
| cp ./php-version-benchmarks/build/infrastructure/config/aws.tfvars.dist ./php-version-benchmarks/build/infrastructure/config/aws.tfvars | ||
| sed -i 's/access_key = ""/access_key = "${{ secrets.PHP_VERSION_BENCHMARK_AWS_ACCESS_KEY }}"/g' ./php-version-benchmarks/build/infrastructure/config/aws.tfvars | ||
| sed -i 's/secret_key = ""/secret_key = "${{ secrets.PHP_VERSION_BENCHMARK_AWS_SECRET_KEY }}"/g' ./php-version-benchmarks/build/infrastructure/config/aws.tfvars | ||
| - name: Setup PHP config - baseline PHP version | ||
| if: github.repository == 'php/php-src' && github.event_name != 'workflow_dispatch' | ||
| run: | | ||
| set -e | ||
|
|
||
| BASELINE_SHORT_SHA="$(echo "${{ env.BASELINE_COMMIT }}" | cut -c1-4)" | ||
|
|
||
| cat << EOF > /php-version-benchmarks/config/php/baseline.ini | ||
| PHP_NAME="PHP - baseline@$BASELINE_SHORT_SHA" | ||
| PHP_ID=php_baseline | ||
|
|
||
| PHP_REPO=https://github.com/${{ env.REPOSITORY }}.git | ||
| PHP_BRANCH=${{ env.BRANCH }} | ||
| PHP_COMMIT=${{ env.BASELINE_COMMIT }} | ||
|
|
||
| PHP_OPCACHE=2 | ||
| PHP_JIT=0 | ||
| EOF | ||
| - name: Setup PHP config - baseline PHP version with JIT (manual only) | ||
| if: github.repository == 'php/php-src' && github.event_name == 'workflow_dispatch' && inputs.jit == '1' | ||
| run: | | ||
| set -e | ||
|
|
||
| BASELINE_SHORT_SHA="$(echo "${{ env.BASELINE_COMMIT }}" | cut -c1-4)" | ||
|
|
||
| cat << EOF > /php-version-benchmarks/config/php/baseline.ini | ||
| PHP_NAME="PHP - baseline@$BASELINE_SHORT_SHA (JIT)" | ||
| PHP_ID=php_baseline_jit | ||
|
|
||
| PHP_REPO=https://github.com/${{ env.REPOSITORY }}.git | ||
| PHP_BRANCH=${{ env.BRANCH }} | ||
| PHP_COMMIT=${{ env.BASELINE_COMMIT }} | ||
|
|
||
| PHP_OPCACHE=2 | ||
| PHP_JIT=${{ env.JIT }} | ||
| EOF | ||
|
|
||
| git clone ./php-version-benchmarks/tmp/php_baseline/ ./php-version-benchmarks/tmp/php_baseline_jit | ||
| - name: Setup PHP config - previous PHP version (scheduled only) | ||
| if: github.repository == 'php/php-src' && github.event_name != 'workflow_dispatch' | ||
| run: | | ||
| set -e | ||
|
|
||
| YEAR="$(date '+%Y')" | ||
| DATABASE="./php-version-benchmarks/docs/results/$YEAR/database.tsv" | ||
| if [ -f "$DATABASE" ]; then | ||
| LAST_RESULT_SHA="$(tail -n 2 "$DATABASE" | head -n 1 | cut -f 6)" | ||
| else | ||
| YESTERDAY="$(date -d "-2 day 23:59:59" '+%Y-%m-%d %H:%M:%S')" | ||
| LAST_RESULT_SHA="$(cd ./php-version-benchmarks/tmp/php_${{ env.BRANCH }}/ && git --no-pager log --until="$YESTERDAY" -n 1 --pretty='%H')" | ||
| fi | ||
|
|
||
| cat << EOF > /php-version-benchmarks/config/php/previous.ini | ||
| PHP_NAME="PHP - previous ${{ env.BRANCH }}" | ||
| PHP_ID=php_${{ env.BRANCH }}_previous | ||
|
|
||
| PHP_REPO=https://github.com/${{ env.REPOSITORY }}.git | ||
| PHP_BRANCH=${{ env.BRANCH }} | ||
| PHP_COMMIT=$LAST_RESULT_SHA | ||
|
|
||
| PHP_OPCACHE=1 | ||
| PHP_JIT=0 | ||
| EOF | ||
| - name: Setup PHP config - benchmarked PHP version | ||
| run: | | ||
| set -e | ||
|
|
||
| cat << EOF > /php-version-benchmarks/config/php/this.ini | ||
| PHP_NAME="PHP - ${{ env.BRANCH }}" | ||
| PHP_ID=php_${{ env.BRANCH }} | ||
|
|
||
| PHP_REPO=https://github.com/${{ env.REPOSITORY }}.git | ||
| PHP_BRANCH=${{ env.BRANCH }} | ||
| PHP_COMMIT=${{ env.COMMIT }} | ||
|
|
||
| PHP_OPCACHE=1 | ||
| PHP_JIT=0 | ||
| EOF | ||
| - name: Setup PHP config - benchmarked PHP version with JIT | ||
| if: env.JIT == '1' | ||
| run: | | ||
| set -e | ||
|
|
||
| cat << EOF > /php-version-benchmarks/config/php/this_jit.ini | ||
| PHP_NAME="PHP - ${{ env.BRANCH }} (JIT)" | ||
| PHP_ID=php_${{ env.BRANCH }}_jit | ||
|
|
||
| PHP_REPO=https://github.com/${{ env.REPOSITORY }}.git | ||
| PHP_BRANCH=${{ env.BRANCH }} | ||
| PHP_COMMIT=${{ env.COMMIT }} | ||
|
|
||
| PHP_OPCACHE=1 | ||
| PHP_JIT=${{ env.JIT }} | ||
| EOF | ||
|
|
||
| git clone ./php-version-benchmarks/tmp/php_${{ env.BRANCH }}/ ./php-version-benchmarks/tmp/php_${{ env.BRANCH }}_jit | ||
| - name: Setup test config | ||
| run: | | ||
| set -e | ||
|
|
||
| cp ./php-version-benchmarks/config/test/1_laravel.ini.dist ./php-version-benchmarks/config/test/1_laravel.ini | ||
| cp ./php-version-benchmarks/config/test/2_symfony_main.ini.dist ./php-version-benchmarks/config/test/2_symfony_main.ini | ||
| cp ./php-version-benchmarks/config/test/4_wordpress.ini.dist ./php-version-benchmarks/config/test/4_wordpress.ini | ||
|
|
@@ -104,6 +205,7 @@ jobs: | |
| - name: Run benchmark | ||
| run: ./php-version-benchmarks/benchmark.sh run aws | ||
| - name: Store results | ||
| if: github.repository == 'php/php-src' && github.event_name != 'workflow_dispatch' | ||
| run: | | ||
| set -ex | ||
|
|
||
|
|
@@ -119,6 +221,17 @@ jobs: | |
| fi | ||
| git commit -m "Add result for ${{ github.repository }}@${{ github.sha }}" | ||
| git push | ||
| - name: Comment results | ||
| if: github.repository != 'php/php-src' && github.event_name == 'workflow_dispatch' | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| PR_NUMBER=$(gh pr list --head "${{ inputs.branch }}" --state open --json number --jq '.[0].number') | ||
kocsismate marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| if [ ! -z "$PR_NUMBER" ]; then | ||
| YEAR="$(date '+%Y')" | ||
| NEWEST_RESULT_DIRECTORY=$(ls -td ./php-version-benchmarks/docs/results/$YEAR/*/ | head -1) | ||
| gh pr comment $PR_NUMBER --body-file "$NEWEST_RESULT_DIRECTORY/result.md" | ||
| fi | ||
| - name: Cleanup | ||
| if: always() | ||
| run: | | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not just use ref and baseline_ref? This can be either a branch, tag, commit, etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Possibly, this could even request just the PR number instead of repository, branch, commit, baseline_commit; as we can determine all of these from the PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The branch and the commit is needed for various reasons, mostly displaying them. However, I love the idea of gathering these info automatically from github CLI 😍 I'm about to finish the implementation.
One caveat is that originally I planned that the workflow could be run for branches, tags, or even for master. A required PR number input will prevent these use-cases, but I think the 99% of usecases are still covered, so I'm fine with the loss.