Skip to content

Commit 118658e

Browse files
ijac13ijac13
committed
M-erge remote-tracking branch 'origin/main' into feature/pla-502-update-start-free-with-cloud-doc
update review suggestions
2 parents 4ac23ee + 095ec1a commit 118658e

File tree

5 files changed

+222
-8
lines changed

5 files changed

+222
-8
lines changed

‎docs/7-cicd/setup-cd.md‎

Lines changed: 104 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,108 @@
22
title: Setup CD
33
---
44

5-
coming soon
5+
# Setup CD
66

7+
Set up automatic updates for your Recce Cloud base sessions. Keep your data comparison baseline current every time you merge to main, with no manual work required.
8+
9+
## Purpose
10+
11+
**Automated Base Session Management** eliminates manual baseline maintenance.
12+
13+
- **Triggers**: PR merge to main + scheduled updates
14+
- **Action**: Auto-update base Recce session
15+
- **Benefit**: Current comparison baseline for future PRs
16+
17+
## Prerequisites
18+
19+
You need `manifest.json` and `catalog.json` files (dbt artifacts) for Recce Cloud. See [Start Free with Cloud](../2-getting-started/start-free-with-cloud.md) for instructions on preparing these files.
20+
21+
## Implementation
22+
23+
### 1. Core Workflow
24+
25+
Create `.github/workflows/cd-workflow.yml`:
26+
27+
```yaml
28+
name: Update Base Recce Session
29+
30+
on:
31+
push:
32+
branches: ["main"]
33+
schedule:
34+
- cron: "0 2 * * *" # Daily at 2 AM UTC
35+
workflow_dispatch:
36+
37+
concurrency:
38+
group: ${{ github.workflow }}
39+
cancel-in-progress: true
40+
41+
jobs:
42+
update-base-session:
43+
runs-on: ubuntu-latest
44+
timeout-minutes: 30
45+
46+
steps:
47+
- name: Checkout code
48+
uses: actions/checkout@v4
49+
50+
- name: Setup Python
51+
uses: actions/setup-python@v5
52+
with:
53+
python-version: "3.11"
54+
cache: "pip"
55+
56+
- name: Install dependencies
57+
run: |
58+
pip install -r requirements.txt
59+
60+
- name: Prepare dbt artifacts
61+
run: |
62+
# Install dbt packages
63+
dbt deps
64+
65+
# Optional: Build tables to ensure they're materialized and updated
66+
# dbt build --target prod
67+
68+
# Required: Generate artifacts (provides all we need)
69+
dbt docs generate --target prod
70+
env:
71+
DBT_ENV_SECRET_KEY: ${{ secrets.DBT_ENV_SECRET_KEY }}
72+
73+
- name: Update Recce Cloud Base Session
74+
uses: DataRecce/[email protected]
75+
# This action automatically uploads artifacts to Recce Cloud
76+
```
77+
78+
### 2. Artifact Preparation Options
79+
80+
**Default: Fresh Build** (shown in example above)
81+
82+
- `dbt docs generate` is required and provides the needed `manifest.json` and `catalog.json` artifacts.
83+
- `dbt build` is optional but ensures tables are materialized and updated.
84+
85+
**Alternative Methods:**
86+
87+
- **External Download**: Download from dbt Cloud, Paradime, or other platforms
88+
- **Pipeline Integration**: Use existing dbt build workflows
89+
90+
91+
### 3. Verification
92+
93+
#### Manual Trigger Test
94+
95+
1. Go to **Actions** tab in your repository
96+
2. Select "Update Base Recce Session" workflow
97+
3. Click **Run workflow** button
98+
4. Monitor the run for successful completion
99+
100+
#### Verify Success
101+
102+
- ✅ **Workflow completes** without errors in Actions tab
103+
- ✅ **Base session updated** in Recce Cloud dashboard
104+
105+
![Recce Cloud dashboard showing updated base sessions](../assets/images/7-cicd/verify-setup-cd.png){: .shadow}
106+
107+
## Next Steps
108+
109+
**[Setup CI](./setup-ci.md)** to automatically validate PR changes against your updated base session. This completes your CI/CD pipeline by adding automated data validation for every pull request.

‎docs/7-cicd/setup-ci.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
---
2+
title: Setup CI
3+
---
4+
5+
# Setup CI
6+
7+
Automatically validate your data changes in every pull request using Recce Cloud. Catch data issues before they reach production, with validation results right in your PR.
8+
9+
## Purpose
10+
11+
**Automated PR Validation** prevents data regressions before merge.
12+
13+
- **Triggers**: PR opened/updated against main
14+
- **Action**: Auto-update Recce session for PR validation
15+
- **Benefit**: Automated data validation and comparison
16+
17+
## Prerequisites
18+
19+
You need `manifest.json` and `catalog.json` files (dbt artifacts) for Recce Cloud. See [Start Free with Cloud](../2-getting-started/start-free-with-cloud.md) for instructions on preparing these files.
20+
21+
## Implementation
22+
23+
### 1. Core Workflow
24+
25+
Create `.github/workflows/ci-workflow.yml`:
26+
27+
```yaml
28+
name: Validate PR Changes
29+
30+
on:
31+
pull_request:
32+
branches: ["main"]
33+
34+
concurrency:
35+
group: ${{ github.workflow }}-${{ github.ref }}
36+
cancel-in-progress: true
37+
38+
jobs:
39+
validate-changes:
40+
runs-on: ubuntu-latest
41+
timeout-minutes: 45
42+
43+
steps:
44+
- name: Checkout PR branch
45+
uses: actions/checkout@v4
46+
with:
47+
fetch-depth: 2
48+
49+
- name: Setup Python
50+
uses: actions/setup-python@v5
51+
with:
52+
python-version: "3.11"
53+
cache: "pip"
54+
55+
- name: Install dependencies
56+
run: |
57+
pip install -r requirements.txt
58+
59+
# Step 1: Prepare current branch artifacts
60+
- name: Build current branch artifacts
61+
run: |
62+
# Install dbt packages
63+
dbt deps
64+
65+
# Optional: Build tables to ensure they're materialized
66+
# dbt build --target ci
67+
68+
# Required: Generate artifacts for comparison
69+
dbt docs generate --target ci
70+
env:
71+
DBT_ENV_SECRET_KEY: ${{ secrets.DBT_ENV_SECRET_KEY }}
72+
73+
- name: Update Recce PR Session
74+
uses: DataRecce/[email protected]
75+
# This action automatically creates a PR session in Recce Cloud
76+
```
77+
78+
### 2. Artifact Preparation Options
79+
80+
**Default: Fresh Build** (shown in example above)
81+
82+
- `dbt docs generate` is required and provides all needed artifacts.
83+
- `dbt build` is optional but ensures tables are materialized and updated.
84+
85+
**Alternative Methods:**
86+
87+
- **External Download**: Download from dbt Cloud, Paradime, or other platforms
88+
- **Pipeline Integration**: Use existing dbt build workflows
89+
90+
### 3. Verification
91+
92+
#### Test with a PR
93+
94+
1. Create a test PR with small data changes
95+
2. Check **Actions** tab for CI workflow execution
96+
3. Verify validation runs successfully
97+
98+
#### Verify Success
99+
100+
- ✅ **Workflow completes** without errors in Actions tab
101+
- ✅ **PR session updated** in Recce Cloud dashboard
102+
103+
![Recce Cloud showing PR validation session](../assets/images/7-cicd/verify-setup-ci.png){: .shadow}
104+
105+
#### Review PR Session
106+
107+
To analyze the PR changes in detail:
108+
109+
- Go to your [Recce Cloud dashboard](https://cloud.reccehq.com)
110+
- Find the PR session that was created
111+
- Launch Recce instance to explore data differences

‎docs/assets/images/7-cicd/verify-setup-cd.png‎

72.5 KB
Loading

‎docs/assets/images/7-cicd/verify-setup-ci.png‎

48.8 KB
Loading

‎mkdocs.yml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -73,17 +73,17 @@ nav:
7373
- 6-collaboration/checklist.md
7474
- 6-collaboration/share.md
7575
- CI/CD:
76-
#- 7-cicd/setup-cd.md
76+
- 7-cicd/setup-cd.md
7777
- Development:
78-
- 7-cicd/scenario-dev.md
78+
- 7-cicd/scenario-dev.md
7979
#- 7-cicd/recce-debug.md # content outdated
8080
- Continuous Integration:
81-
#- 7-cicd/setup-ci.md
82-
#- 7-cicd/cloud-recce-summary.md
83-
#- 7-cicd/cloud-preset-checks.md
84-
- 7-cicd/scenario-pr-review.md
81+
- 7-cicd/setup-ci.md
82+
#- 7-cicd/cloud-recce-summary.md
83+
#- 7-cicd/cloud-preset-checks.md
84+
- 7-cicd/scenario-pr-review.md
8585
- CI/CD for Open Source:
86-
- 7-cicd/scenario-ci.md
86+
- 7-cicd/scenario-ci.md
8787
#- 7-cicd/recce-summary.md content outdated
8888
- 7-cicd/preset-checks.md
8989
- 7-cicd/best-practices-prep-env.md

0 commit comments

Comments
 (0)