Skip to content

Commit de481d0

Browse files
ndbroadbentndbroadbent
committed
updates and fixes
1 parent b4e8dd6 commit de481d0

File tree

6 files changed

+956
-197
lines changed

6 files changed

+956
-197
lines changed

‎.github/workflows/release.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -191,3 +191,49 @@ jobs:
191191
artifacts/**/*.tar.gz
192192
artifacts/**/*.zip
193193
artifacts/**/*.sha256
194+
195+
docker_image:
196+
name: Build and Push Docker Image
197+
needs: release_create
198+
runs-on: ubuntu-latest
199+
permissions:
200+
contents: read
201+
env:
202+
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
203+
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
204+
steps:
205+
- name: Checkout repository
206+
uses: actions/checkout@v4
207+
with:
208+
fetch-depth: 0
209+
210+
- name: Extract version
211+
id: v
212+
run: |
213+
VERSION=$(grep -E '^version = ' Cargo.toml | head -1 | sed 's/version = "\(.*\)"/\1/')
214+
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
215+
216+
- name: Set up QEMU
217+
uses: docker/setup-qemu-action@v3
218+
219+
- name: Set up Docker Buildx
220+
uses: docker/setup-buildx-action@v3
221+
222+
- name: Log in to Docker Hub
223+
uses: docker/login-action@v3
224+
with:
225+
username: ${{ env.DOCKERHUB_USERNAME }}
226+
password: ${{ env.DOCKERHUB_TOKEN }}
227+
228+
- name: Build and push multi-arch image
229+
run: |
230+
set -euo pipefail
231+
VERSION="${{ steps.v.outputs.version }}"
232+
echo "Building docspringcom/cigen:${VERSION} and :latest"
233+
docker buildx build \
234+
--platform linux/amd64,linux/arm64 \
235+
-f docker/cigen.Dockerfile \
236+
--build-arg CIGEN_VERSION="${VERSION}" \
237+
-t docspringcom/cigen:"${VERSION}" \
238+
-t docspringcom/cigen:latest \
239+
--push .

‎docker/cigen.Dockerfile‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
FROM cimg/base:stable
2+
3+
RUN apt-get update && apt-get install -y --no-install-recommends \
4+
ca-certificates curl jq bash git \
5+
&& rm -rf /var/lib/apt/lists/*
6+
7+
ARG CIGEN_VERSION=0.0.0
8+
ARG TARGETARCH
9+
10+
RUN set -e; \
11+
case "$TARGETARCH" in \
12+
"amd64") ARCH="amd64" ;; \
13+
"arm64") ARCH="arm64" ;; \
14+
*) echo "Unsupported arch: $TARGETARCH" >&2; exit 1 ;; \
15+
esac; \
16+
echo "Installing cigen v${CIGEN_VERSION} for ${ARCH}"; \
17+
curl -fsSL -o /tmp/cigen.tar.gz \
18+
"https://github.com/DocSpring/cigen/releases/download/v${CIGEN_VERSION}/cigen-linux-${ARCH}.tar.gz"; \
19+
tar -xzf /tmp/cigen.tar.gz -C /usr/local/bin; \
20+
chmod +x /usr/local/bin/cigen; \
21+
rm -f /tmp/cigen.tar.gz
22+
23+
ENTRYPOINT ["cigen"]

‎notes/dynamic_setup_redesign.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Dynamic Setup Redesign (Two‑File CircleCI Architecture)
2+
3+
This note summarizes the new direction for CIGen’s CircleCI dynamic setup, aligning with DocSpring’s production needs and replacing the previous per‑workflow split approach.
4+
5+
## Goals
6+
7+
- Use exactly two CircleCI configs:
8+
- `.circleci/config.yml` — entrypoint. Contains:
9+
- workflow `package_updates`: runs immediately when `pipeline.parameters.check_package_versions == true`.
10+
- workflow `staging_postman_tests`: runs immediately when `pipeline.parameters.run_staging_postman_tests == true`.
11+
- workflow `setup`: runs only when neither param is true; performs skip‑gating and posts a filtered continuation.
12+
- `.circleci/main.yml` — CI/CD workflow (tests/build/deploy). Designed for dynamic skip.
13+
- No per‑job runtime skip on CircleCI when `dynamic=true`. Setup is the gate.
14+
- Skip gating in setup uses CircleCI’s cache save/restore only (no Redis).
15+
- Use a dedicated `docspringcom/cigen` runtime image (no Ruby dependency) for setup.
16+
- On CI, ensure `.circleci/config.yml` is up‑to‑date (self‑check), optionally auto‑commit+push and fail if drift is detected (opt‑in).
17+
18+
## Behavior
19+
20+
1. Immediate workflows (in `.circleci/config.yml`)
21+
22+
- `package_updates` runs immediately if `check_package_versions` param is true.
23+
- `staging_postman_tests` runs immediately if `run_staging_postman_tests` param is true.
24+
- Otherwise, neither runs.
25+
26+
2. Dynamic setup workflow (in `.circleci/config.yml`)
27+
28+
- Image: `docspringcom/cigen:latest` (fallback curl installer for cigen until published).
29+
- Steps:
30+
- `checkout`
31+
- `cigen generate` → writes `.circleci/main.yml` (and re‑renders `.circleci/config.yml` for self‑check).
32+
- Self‑check (opt‑in): verify current `.circleci/config.yml` matches what `cigen generate` would produce. If not:
33+
- Optionally `git add && git commit && git push` (opt‑in), then fail the build to force a new run on the updated entrypoint.
34+
- Skip analysis for `main` only:
35+
- For each job+arch in `main` with `source_files`:
36+
- Compute `JOB_HASH` in setup (same hashing logic as jobs).
37+
- Write a variant job‑key: `/tmp/setup_keys/<job_arch>/job-key` (`${JOB_NAME}-${DOCKER_ARCH}-${JOB_HASH}`).
38+
- `restore_cache` with key `job_status-exists-v1-{{ checksum "/tmp/setup_keys/<job_arch>/job-key" }}`
39+
- If `/tmp/cigen_job_exists/done_${JOB_HASH}` exists, append `<job_arch>` to `/tmp/skip/main.txt`.
40+
- Clear `/tmp/cigen_job_exists` before the next restore.
41+
- `cigen generate --workflow main` with `CIGEN_SKIP_JOBS_FILE=/tmp/skip/main.txt` to prune jobs and transitive dependents; write filtered continuation.
42+
- Continue with the filtered `.circleci/main.yml` via the Continuation API.
43+
44+
3. Jobs (CircleCI with dynamic=true)
45+
46+
- Do NOT inject per‑job runtime `restore_cache + halt`. Setup is the gate.
47+
- Still record an “exists” marker at end of job:
48+
- touch `/tmp/cigen_job_exists/done_${JOB_HASH}`
49+
- `save_cache` with key `job_status-exists-v1-{{ checksum "/tmp/cigen_job_status/job-key" }}`; paths `/tmp/cigen_job_exists`
50+
- Keep the job-status cache write if desired, but runtime halt is disabled for CircleCI dynamic.
51+
52+
## CIGen Implementation Outline
53+
54+
- Add `workflows_meta` (internal) to drive generation; but final shape is two files only:
55+
- `config.yml` — contains param‑guarded `package_updates` and `staging_postman_tests`, and a synthesized `setup` workflow.
56+
- `main.yml` — the CI/CD workflow.
57+
- Generator changes:
58+
- Always emit `main.yml` for CircleCI dynamic.
59+
- Synthesize `config.yml` with three workflows as above (no split files for package updates or staging).
60+
- Disable per‑job runtime skip injection when `dynamic=true` and `provider=circleci`; still add exists‑cache save step at end of jobs with `source_files`.
61+
- Add `CIGEN_SKIP_JOBS_FILE` support for `cigen generate --workflow main` to prune jobs and transitive dependents (already implemented).
62+
- Setup generation:
63+
- Uses `docspring/cigen:latest`; fallback curl installer in case the image isn’t on the runner yet.
64+
- Emits a deterministic loop of `restore_cache` probes (one per job variant in `main`) and collects a skip list.
65+
- Emits a self‑check step (opt‑in) that regenerates `config.yml` and verifies drift.
66+
67+
## Self‑Check (Opt‑In)
68+
69+
- Config key (to be added):
70+
71+
```yaml
72+
setup:
73+
self_check:
74+
enabled: true
75+
commit_on_diff: true # optional: add+commit+push then fail
76+
```
77+
78+
- If enabled, setup:
79+
- Regenerates `.circleci/config.yml` into a temp path.
80+
- If different from the existing entrypoint, optionally commit/push and fail the build.
81+
- This ensures the dynamic entrypoint is always up‑to‑date.
82+
83+
## Notes
84+
85+
- This design avoids per‑job runtime halts in CircleCI and does all skip‑gating up front, matching DocSpring’s Ruby implementation goals.
86+
- “Immediate” workflows (package updates, staging postman) run right away via parameters; no hand‑off to setup.
87+
- Scheduled workflows are handled by setting the parameter on the schedule.
88+
- BASE_HASH/file hashing is anchored and cached at the file/content level and is millisecond‑fast.
89+
90+
## Next Steps
91+
92+
- Implement generation of combined `.circleci/config.yml` (package_updates + staging_postman_tests + setup) and `.circleci/main.yml` only.
93+
- Emit the skip‑probe loop in setup with static `restore_cache` steps per job variant in `main`.
94+
- Add setup self‑check (opt‑in) with optional auto‑commit+push.
95+
- Publish `docspring/cigen` and switch setup to use it without fallback.
96+
- Validate end‑to‑end on DocSpring: push branch, confirm pipeline behavior (immediate workflows via params, setup gating, minimal continuation).

‎src/commands/generate.rs‎

Lines changed: 71 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,19 @@ fn generate_from_jobs(
9696
anyhow::bail!("No jobs found for workflow '{}'", workflow_name);
9797
}
9898

99+
// Optional pruning via skip-jobs file (env var for setup use)
100+
if let Ok(skip_file) = std::env::var("CIGEN_SKIP_JOBS_FILE")
101+
&& std::path::Path::new(&skip_file).exists()
102+
{
103+
let content = std::fs::read_to_string(&skip_file)?;
104+
let mut skip: std::collections::HashSet<String> = content
105+
.lines()
106+
.map(|s| s.trim().to_string())
107+
.filter(|s| !s.is_empty())
108+
.collect();
109+
prune_skipped_jobs(&mut workflow_jobs, &mut skip);
110+
}
111+
99112
// Apply package deduplication if any jobs have packages
100113
let has_packages = workflow_jobs.values().any(|job| job.packages.is_some());
101114
if has_packages {
@@ -241,11 +254,35 @@ fn generate_from_jobs(
241254
merged_workflow_configs.insert(workflow_name.clone(), merged);
242255
}
243256

244-
let has_setup = merged_workflow_configs
257+
let mut has_setup = merged_workflow_configs
245258
.values()
246259
.any(|c| c.setup.unwrap_or(false));
247260

248-
if has_setup {
261+
// If dynamic is enabled, force split outputs and synthesized setup
262+
if loaded_config.config.dynamic.unwrap_or(false) {
263+
has_setup = true;
264+
}
265+
266+
if loaded_config.config.provider == "circleci"
267+
&& loaded_config.config.dynamic.unwrap_or(false)
268+
{
269+
// Generate combined two-file dynamic setup (config.yml + main.yml)
270+
provider
271+
.generate_all(
272+
&loaded_config.config,
273+
&workflows,
274+
&loaded_config.commands,
275+
&output_path,
276+
)
277+
.map_err(|e| {
278+
anyhow::anyhow!("Failed to generate dynamic CircleCI config: {}", e)
279+
})?;
280+
281+
println!(
282+
"✅ Generated CircleCI dynamic setup (config.yml + main.yml) to {}",
283+
output_path.display()
284+
);
285+
} else if has_setup {
249286
// Split outputs with inferred filenames
250287
for (workflow_name, mut jobs_copy) in workflows.clone() {
251288
if jobs_copy.values().any(|job| job.packages.is_some()) {
@@ -316,35 +353,44 @@ fn generate_from_jobs(
316353
provider.name(),
317354
output_path.display()
318355
);
319-
320-
// If dynamic mode is enabled but no explicit setup workflow exists, synthesize one
321-
if loaded_config.config.dynamic.unwrap_or(false)
322-
&& !merged_workflow_configs.keys().any(|k| k == "setup")
323-
{
324-
println!("Generating synthesized setup workflow");
325-
// Setup should go to provider default output path (config.yml) unless overridden
326-
let setup_output = if let Some(setup_out) = &loaded_config.config.output_path {
327-
original_dir_path(Path::new(setup_out))
328-
} else {
329-
original_dir_path(Path::new(provider.default_output_path()))
330-
};
331-
// Call provider-specific synthesized setup if available (only CircleCI for now)
332-
// Provider-agnostic support: synthesize CircleCI setup when using CircleCI provider
333-
if loaded_config.config.provider == "circleci" {
334-
cigen::providers::circleci::CircleCIProvider::new()
335-
.generator
336-
.generate_synthesized_setup(&loaded_config.config, &setup_output)
337-
.map_err(|e| {
338-
anyhow::anyhow!("Failed to generate synthesized setup: {}", e)
339-
})?;
340-
}
341-
}
342356
}
343357
}
344358

345359
Ok(())
346360
}
347361

362+
fn prune_skipped_jobs(
363+
jobs: &mut HashMap<String, cigen::models::Job>,
364+
skip: &mut std::collections::HashSet<String>,
365+
) {
366+
// Remove explicitly skipped jobs
367+
jobs.retain(|name, _| !skip.contains(name));
368+
369+
// Transitive pruning: drop jobs whose requires include missing jobs
370+
loop {
371+
let before = jobs.len();
372+
let missing: std::collections::HashSet<String> = jobs
373+
.iter()
374+
.flat_map(|(name, job)| {
375+
job.required_jobs()
376+
.into_iter()
377+
.filter(|r| !jobs.contains_key(r))
378+
.map(move |_| name.clone())
379+
})
380+
.collect();
381+
if missing.is_empty() {
382+
break;
383+
}
384+
for m in missing {
385+
skip.insert(m.clone());
386+
jobs.remove(&m);
387+
}
388+
if jobs.len() == before {
389+
break;
390+
}
391+
}
392+
}
393+
348394
fn generate_with_templates(
349395
outputs: &[cigen::models::OutputConfig],
350396
specific_output: Option<String>,

‎src/models/config.rs‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,8 +75,14 @@ pub struct Config {
7575
#[serde(skip_serializing_if = "Option::is_none")]
7676
pub workflows: Option<HashMap<String, WorkflowConfig>>,
7777

78+
#[serde(skip_serializing_if = "Option::is_none")]
79+
pub workflows_meta: Option<HashMap<String, WorkflowMeta>>,
80+
7881
#[serde(skip_serializing_if = "Option::is_none")]
7982
pub checkout: Option<CheckoutSetting>,
83+
84+
#[serde(skip_serializing_if = "Option::is_none")]
85+
pub setup_options: Option<SetupOptions>,
8086
}
8187

8288
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -263,7 +269,9 @@ impl Default for Config {
263269
docker_build: None,
264270
package_managers: None,
265271
workflows: None,
272+
workflows_meta: None,
266273
checkout: None,
274+
setup_options: None,
267275
}
268276
}
269277
}
@@ -366,6 +374,37 @@ pub struct WorkflowConfig {
366374
pub checkout: Option<CheckoutSetting>,
367375
}
368376

377+
#[derive(Debug, Clone, Serialize, Deserialize)]
378+
pub struct WorkflowMeta {
379+
#[serde(skip_serializing_if = "Option::is_none")]
380+
pub continuation: Option<String>,
381+
382+
#[serde(skip_serializing_if = "Option::is_none")]
383+
pub default: Option<bool>,
384+
385+
#[serde(skip_serializing_if = "Option::is_none")]
386+
pub trigger_param: Option<String>,
387+
388+
#[serde(skip_serializing_if = "Option::is_none")]
389+
pub scheduled: Option<bool>,
390+
}
391+
392+
#[derive(Debug, Clone, Serialize, Deserialize)]
393+
pub struct SetupOptions {
394+
#[serde(skip_serializing_if = "Option::is_none")]
395+
pub self_check: Option<SelfCheckOptions>,
396+
397+
#[serde(skip_serializing_if = "Option::is_none")]
398+
pub runtime_image: Option<String>,
399+
}
400+
401+
#[derive(Debug, Clone, Serialize, Deserialize)]
402+
pub struct SelfCheckOptions {
403+
pub enabled: bool,
404+
#[serde(skip_serializing_if = "Option::is_none")]
405+
pub commit_on_diff: Option<bool>,
406+
}
407+
369408
#[derive(Debug, Clone, Serialize, Deserialize)]
370409
pub struct CheckoutConfig {
371410
#[serde(default = "default_shallow_checkout")]

0 commit comments

Comments
 (0)