Skip to content

Commit 5f2cc5c

Browse files
authored
chore: dynamically pack the circleci pipeline configuration (#32618)
* chore: dynamically pack the circleci pipeline configuration The ./scripts/pack-ci.sh is the first workhorse, here. It uses the circleci cli to pack any subdirectory of `.circleci/src` into a circleci config using FYAML format. It can pack every subdirectory, a subsection of subdirs that include changes, and use circleci cli to validate the packed yml files. Husky is set up to validate the packed yml file(s) on commit. The packed yml file(s) should not be committed to source control. They are build artifacts. They are configured to be ignored by git in ./.circleci/.gitignore. ./.circleci/config.yml is the second major change. This workflow is modified such that: - it uses workflow filtering to prevent the workflow from triggering unless at least one of the following is true: - the workflow was triggered via api (the 'trigger workflows' button in circleci) - the branch is `develop` - the branch is `release/*` - the branch has an open pull request, and the pull request is not draft - checks out the repository from the HEAD ref of the pr (the branch with the changes), and calculates a checksum of the contents of ./.circleci/src. Packed workflow files are cached by the checksum of the source files that were used to generate the packed workflow. If the cache is restored, the packed pipeline.yml file is continued. If there is no cache for the checksum, it uses ./scripts/pack-ci to pack the source yml files and triggers a continuation with the resulting packed yml file. * rm packed yml file * revert to old draft detection :( other way is only avail on github app * just fail instead of cancelling on drafts, more reliable even if not correct * just halt, since draft->ready does not already trigger a ci run * circleci agent -> circleci-agent * validate main circleci yml file; correct entrypoint for pack workflows job * workflows -> pipeline * Update pack-ci.sh - newline at eof
1 parent 14beae3 commit 5f2cc5c

File tree

15 files changed

+312
-5329
lines changed

15 files changed

+312
-5329
lines changed

.circleci/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
packed

.circleci/README.md

Lines changed: 17 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
# CircleCI Configuration
22

3-
This directory contains CircleCI configuration files that are automatically generated and updated.
3+
This directory contains CircleCI configuration files that use a dynamic workflow packing system for efficient CI development and execution.
44

55
## Prerequisites
66

77
### CircleCI Local CLI
88

9-
The CircleCI Local CLI is required to generate the `pull-request.yml` file from the source configuration.
9+
The CircleCI Local CLI is required to pack the source configurations.
1010

1111
**Installation:**
1212

@@ -30,28 +30,31 @@ The CircleCI Local CLI is required to generate the `pull-request.yml` file from
3030

3131
For more detailed installation instructions, see the [CircleCI Local CLI documentation](https://circleci.com/docs/2.0/local-cli/).
3232

33-
## Lint-Staged Rules
33+
## Pre-commit Validation
3434

35-
When files in this directory are modified, the following lint-staged rule will automatically run:
35+
When files in `.circleci/src/` are modified, the pre-commit hook automatically runs:
3636

3737
```bash
38-
circleci config pack .circleci/workflows-src > .circleci/workflows.yml
38+
yarn pack-ci --verify
3939
```
4040

4141
This command:
42-
1. Takes the source configuration from `./.circleci/workflows-src/`
43-
2. Packs it into a single YAML file
44-
3. Outputs the result to `./circleci/workflows.yml`
42+
1. Scans all directories in `./.circleci/src/` for modifications
43+
2. Packs only the modified directories (e.g., `workflows/``workflows.yml`)
44+
3. Validates the packed configurations
45+
4. Exits with error if validation fails
4546

4647
## File Structure
4748

48-
- `workflows-src/` - Source configuration files (modify these)
49-
- `workflows.yml` - Generated configuration file (auto-generated, do not edit manually)
49+
- `src/` - Source configuration directories (modify these)
50+
- `packed/` - Generated configuration files (ignored by git)
5051

5152
## Development Workflow
5253

53-
1. Make changes to files in `workflows-src/`
54-
2. The lint-staged hook will automatically regenerate `workflows.yml` and stage it
55-
3. Commit both the source changes and the generated file
54+
1. Make changes to files in `src/` directories
55+
2. Stage and commit changes - pre-commit hook automatically validates and packs configurations
56+
3. The jobs defined in `config.yml` will pack these source directories on-the-fly when CI gets kicked off.
5657

57-
**Note:** Always commit both the source files and the generated `workflows.yml` file together to ensure the CircleCI configuration stays in sync.
58+
## `config.yml`
59+
60+
This is the main entrypoint to Cypress CI. It loads packed workflow files from cache, or builds them if necessary. Then it continues to the primary workflow. The main entrypoint to our CI must be available in source control and not packed on-the-fly.

.circleci/cache-version.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
# Bump this version to force CI to re-create the cache from scratch.
2-
9-30-2025
2+
9-30-2026

.circleci/config.yml

Lines changed: 147 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -1,60 +1,159 @@
1-
jobs:
2-
verify-ci-should-run:
3-
docker:
4-
- image: cimg/node:current
5-
resource_class: small
6-
steps:
1+
orbs:
2+
continuation: circleci/[email protected]
3+
circleci-cli: circleci/[email protected]
4+
setup: true
5+
version: 2.1
6+
7+
workflows:
8+
setup-workflow:
9+
jobs:
10+
- pack-workflows
11+
- launch-primary-workflow:
12+
requires:
13+
- pack-workflows
14+
15+
commands:
16+
persist:
17+
steps:
18+
- persist_to_workspace:
19+
root: .
20+
paths:
21+
- .circleci/packed
22+
persist-to-workspace-and-exit-early:
23+
steps:
24+
- run:
25+
name: Check cache status and exit early if needed
26+
command: |
27+
if [[ -d ".circleci/packed" ]] && [[ "$(ls -A .circleci/packed 2>/dev/null)" ]]; then
28+
echo "📦 Cache hit - packed configurations exist"
29+
echo "✅ Persisting packed configurations to workspace and halting job"
30+
echo "CACHE_HIT=true" >> $BASH_ENV
31+
else
32+
echo "📦 Cache miss - continuing with packaging"
33+
echo "CACHE_HIT=false" >> $BASH_ENV
34+
mkdir -p .circleci/packed # persist empty dir to workspace
35+
fi
36+
- persist
37+
- run:
38+
name: End job if cache hit
39+
command: |
40+
if [[ "$CACHE_HIT" == "true" ]] && [[ "$(ls -A .circleci/packed 2>/dev/null)" ]]; then
41+
echo "✅ Cache hit - packed configurations exist"
42+
circleci-agent step halt
43+
else
44+
echo "📦 Cache miss - continuing with packaging"
45+
fi
46+
pack-workflows:
47+
steps:
48+
- circleci-cli/install
49+
- run:
50+
name: Pack workflows
51+
command: |
52+
./scripts/pack-ci.sh --all
53+
validate-workflows:
54+
steps:
55+
- run:
56+
name: Validate workflows
57+
command: |
58+
./scripts/pack-ci.sh --validate --pack=false
59+
60+
# Retrieving and saving packed workflows based on the checksum of the hash
61+
# of the yml files in ./.circleci/src
62+
calc-src-checksum:
63+
steps:
64+
- run:
65+
name: Generate config checksum for cache lookup
66+
# this is saved to a file, so that we can use {{ checksum }} - which
67+
# can only operate on the contents of a single file, not a list of
68+
# files or a directory.
69+
command: |
70+
find .circleci/src -name "*.yml" -type f | \
71+
sort | \
72+
xargs cat | \
73+
md5sum | \
74+
cut -d' ' -f1 > /tmp/config_checksum.txt
75+
restore-src-checksum-cache:
76+
steps:
77+
- calc-src-checksum
78+
- restore_cache:
79+
keys:
80+
- circleci-packed-pipelines-{{ checksum "/tmp/config_checksum.txt" }}
81+
paths:
82+
- .circleci/packed
83+
save-src-checksum-cache:
84+
steps:
85+
- calc-src-checksum
86+
- save_cache:
87+
key: circleci-packed-pipelines-{{ checksum "/tmp/config_checksum.txt" }}
88+
paths:
89+
- .circleci/packed
90+
91+
cancel-draft-prs:
92+
steps:
93+
- when:
94+
condition:
95+
not:
96+
or:
97+
- equal: [<<pipeline.trigger_source>>, 'api']
98+
- equal: [<<pipeline.git.branch>>, 'develop']
99+
- matches:
100+
pattern: /^release\/.*/
101+
value: <<pipeline.git.branch>>
102+
steps:
7103
- run:
104+
name: Cancel CI
8105
command: |
9-
# run CI when manually triggers via CircleCi Dashboard
10-
if [ <<pipeline.trigger_source>> == 'api' ]; then
11-
echo "Always run CI when manually triggered from the UI."
12-
exit 0
13-
fi
14-
15-
if [[ "$CIRCLE_BRANCH" == "develop" || "$CIRCLE_BRANCH" == "release/"* ]]; then
16-
echo "Always run CI for develop and for release candidate branches."
17-
exit 0
18-
fi
106+
cancel_build() {
107+
circleci-agent step halt
108+
}
19109
20-
LAST_COMMIT_MESSAGE=$(curl --silent "https://api.github.com/repos/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/commits/${CIRCLE_BRANCH}" | jq '.commit.message')
110+
if [ ! -z "${CIRCLE_PULL_REQUEST##*/}" ]; then
111+
DRAFT=$(curl --silent "https://api.github.com/repos/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/pulls/${CIRCLE_PULL_REQUEST##*/}" | jq '.draft')
21112
22-
if [[ "$LAST_COMMIT_MESSAGE" =~ "run ci" ]]; then
23-
echo "Always run CI when the commit message includes 'run ci'."
24-
exit 0
113+
if [[ "${DRAFT}" == true ]]; then
114+
echo "Skipping CI; PR is in draft - to trigger CI , click the 'Trigger Pipeline' button in the CircleCI UI."
115+
cancel_build
25116
fi
26117
27-
cancel_build () {
28-
echo "Canceling the CI build..."
29-
circleci-agent step halt
30-
}
118+
echo "Always run CI for PR that is ready for review."
119+
exit 0
120+
fi
31121
32-
TRIGGER_INSTRUCTIONS="to trigger CI , include 'run ci' in the commit message or click the 'Trigger Pipeline' button in the CircleCI UI."
122+
# otherwise, the build is not a PR so we cancel it
123+
cancel_build
33124
34-
if [ ! -z "${CIRCLE_PULL_REQUEST##*/}" ]; then
35-
DRAFT=$(curl --silent "https://api.github.com/repos/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/pulls/${CIRCLE_PULL_REQUEST##*/}" | jq '.draft')
125+
jobs:
126+
pack-workflows:
127+
docker:
128+
- image: cimg/base:stable
129+
resource_class: small
130+
steps:
131+
- cancel-draft-prs
132+
- checkout
133+
- restore-src-checksum-cache
134+
- persist-to-workspace-and-exit-early
135+
- pack-workflows
136+
- save-src-checksum-cache
137+
- persist
36138

37-
if [[ "${DRAFT}" == true ]]; then
38-
echo "Skipping CI; PR is in draft - $TRIGGER_INSTRUCTIONS"
39-
cancel_build
40-
fi
139+
launch-primary-workflow:
140+
docker:
141+
- image: cimg/base:stable
142+
resource_class: small
143+
steps:
144+
- cancel-draft-prs
145+
- attach_workspace:
146+
at: .
147+
- run:
148+
name: Check for primary workflow definition
149+
command: |
150+
if [ ! -f .circleci/packed/pipeline.yml ]; then
151+
echo "❌ Primary workflow definition not found in .circleci/packed/pipeline.yml!"
152+
exit 1
153+
fi
154+
- continuation/continue:
155+
configuration_path: .circleci/packed/pipeline.yml
41156

42-
echo "Always run CI for PR that is ready for review."
43-
exit 0
44-
fi
45157

46-
echo "Skipping CI; branch in progress - $TRIGGER_INSTRUCTIONS"
47-
cancel_build
48-
name: Verify CI should run
49-
- checkout
50-
- continuation/continue:
51-
configuration_path: .circleci/workflows.yml
52-
orbs:
53-
continuation: circleci/[email protected]
54-
setup: true
55-
version: 2.1
56-
workflows:
57-
setup-workflow:
58-
jobs:
59-
- verify-ci-should-run
158+
60159

.circleci/src/config/@config.yml

Lines changed: 0 additions & 4 deletions
This file was deleted.

.circleci/src/config/jobs/verify-ci-should-run.yml

Lines changed: 0 additions & 49 deletions
This file was deleted.

.circleci/src/config/workflows/setup-workflow.yml

Lines changed: 0 additions & 2 deletions
This file was deleted.
File renamed without changes.
File renamed without changes.
File renamed without changes.

0 commit comments

Comments
 (0)