Skip to content

Commit 6671287

Browse files
mendesfabiomendesfabio
authored
Initial commit
0 parents  commit 6671287

38 files changed

+5354
-0
lines changed

‎.cursor/rules/dbt-best-practices.mdc‎

Lines changed: 926 additions & 0 deletions
Large diffs are not rendered by default.

‎.github/workflows/dbt_ci.yml‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
name: dbt CI
2+
3+
# NOTE: Pull request CI jobs are disabled by default. Uncomment to enable standard CI workflows
4+
# Check the README.md to make sure you understand the credit costs associated with running dbt.
5+
# on:
6+
# pull_request:
7+
# types: [opened, synchronize, reopened]
8+
# paths:
9+
# - 'models/**'
10+
# - 'macros/**'
11+
# - 'dbt_project.yml'
12+
# - 'profiles.yml'
13+
# - 'packages.yml'
14+
# - '.github/workflows/dbt_ci.yml'
15+
16+
concurrency:
17+
group: ${{ github.workflow }}-${{ github.ref }}
18+
cancel-in-progress: true
19+
20+
jobs:
21+
dbt-ci:
22+
runs-on: ubuntu-latest
23+
timeout-minutes: 30
24+
25+
env:
26+
DUNE_API_KEY: ${{ secrets.DUNE_API_KEY }}
27+
DUNE_TEAM_NAME: ${{ vars.DUNE_TEAM_NAME || 'dune' }} # Set as GitHub Variable or defaults to 'dune'
28+
DEV_SCHEMA_SUFFIX: pr${{ github.event.pull_request.number }}
29+
30+
steps:
31+
- name: Check out code
32+
uses: actions/checkout@v4
33+
34+
- name: Install uv
35+
uses: astral-sh/setup-uv@v5
36+
with:
37+
enable-cache: true
38+
39+
- name: Install dependencies
40+
run: uv sync --locked
41+
42+
- name: Install dbt packages
43+
run: uv run dbt deps
44+
45+
- name: Download main branch manifest
46+
id: download-manifest
47+
continue-on-error: true
48+
# Using dawidd6/action-download-artifact instead of actions/download-artifact
49+
# because the official action only downloads artifacts from the same workflow run,
50+
# while we need to download the manifest from the most recent successful dbt_deploy run.
51+
uses: dawidd6/action-download-artifact@v6
52+
with:
53+
name: prod-manifest-latest
54+
path: ./state
55+
workflow: dbt_deploy.yml
56+
branch: main
57+
if_no_artifact_found: ignore # Let our verification step handle the error with detailed message
58+
59+
- name: Verify manifest exists
60+
run: |
61+
if [ ! -f "./state/manifest.json" ]; then
62+
echo "❌ ERROR: Main branch manifest not found"
63+
echo ""
64+
echo "The manifest is required for state comparison in PR CI."
65+
echo ""
66+
echo "This typically happens when:"
67+
echo " • This is a new repository (first time setup)"
68+
echo " • The manifest artifact has expired (90-day retention)"
69+
echo ""
70+
echo "To fix this, manually trigger the dbt deploy workflow:"
71+
echo " 1. Go to: Actions → dbt deploy"
72+
echo " 2. Click 'Run workflow' button"
73+
echo " 3. Select the 'main' branch"
74+
echo " 4. Click 'Run workflow'"
75+
echo " 5. Wait for the workflow to complete"
76+
echo " 6. Re-run this PR check"
77+
echo ""
78+
echo "Alternatively, merge any commit to main to trigger it automatically."
79+
exit 1
80+
fi
81+
echo "✅ Main branch manifest found"
82+
83+
- name: Compile feature branch models
84+
run: uv run dbt compile
85+
86+
- name: Run modified models (full refresh)
87+
run: uv run dbt run --select state:modified --state ./state --full-refresh
88+
89+
- name: Test modified models (after full refresh)
90+
run: uv run dbt test --select state:modified --state ./state
91+
92+
- name: Run modified incremental models (incremental run)
93+
run: uv run dbt run --select state:modified,config.materialized:incremental --state ./state
94+
95+
- name: Test modified incremental models (after incremental run)
96+
run: uv run dbt test --select state:modified,config.materialized:incremental --state ./state

‎.github/workflows/dbt_deploy.yml‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
name: dbt deploy
2+
3+
on:
4+
# NOTE: Push and schedule triggers are disabled by default. Uncomment to enable standard CI workflows
5+
# Check the README.md to make sure you understand the credit costs associated with running dbt.
6+
# push:
7+
# branches:
8+
# - main
9+
# paths:
10+
# - 'models/**'
11+
# - 'macros/**'
12+
# - 'dbt_project.yml'
13+
# - 'profiles.yml'
14+
# - 'packages.yml'
15+
# - '.github/workflows/dbt_deploy.yml'
16+
# schedule:
17+
# - cron: '0 0 1 * *' # Run monthly on the 1st to refresh manifest artifact (90-day retention)
18+
workflow_dispatch: # Allow manual triggers
19+
20+
concurrency:
21+
group: dbt-prod-runs # Shared group with dbt_prod.yml to prevent concurrent production runs
22+
cancel-in-progress: false # Don't cancel in-progress runs, queue instead
23+
24+
jobs:
25+
dbt-deploy:
26+
runs-on: ubuntu-latest
27+
timeout-minutes: 30
28+
29+
env:
30+
DUNE_API_KEY: ${{ secrets.DUNE_API_KEY }}
31+
DUNE_TEAM_NAME: ${{ vars.DUNE_TEAM_NAME || 'dune' }} # Set as GitHub Variable or defaults to 'dune'
32+
DBT_TARGET: prod # All dbt commands will use the prod target from profiles.yml
33+
34+
steps:
35+
- name: Check out code
36+
uses: actions/checkout@v4
37+
38+
- name: Install uv
39+
uses: astral-sh/setup-uv@v5
40+
with:
41+
enable-cache: true
42+
43+
- name: Install dependencies
44+
run: uv sync --locked
45+
46+
- name: Install dbt packages
47+
run: uv run dbt deps
48+
49+
- name: Download previous manifest for state comparison
50+
id: download-manifest
51+
continue-on-error: true
52+
# Using dawidd6/action-download-artifact instead of actions/download-artifact
53+
# because the official action only downloads artifacts from the same workflow run,
54+
# while we need to download the manifest from the most recent successful run of this workflow.
55+
uses: dawidd6/action-download-artifact@v6
56+
with:
57+
name: prod-manifest-latest
58+
path: ./state
59+
workflow: dbt_deploy.yml
60+
branch: main
61+
if_no_artifact_found: warn
62+
63+
- name: Compile current models
64+
run: uv run dbt compile
65+
66+
- name: Check if previous state exists
67+
id: check-state
68+
run: |
69+
if [ -f "./state/manifest.json" ]; then
70+
echo "state_exists=true" >> $GITHUB_OUTPUT
71+
echo "✅ Previous manifest found - will run modified models only"
72+
else
73+
echo "state_exists=false" >> $GITHUB_OUTPUT
74+
echo "⚠️ No previous manifest - will run all models (first merge to main)"
75+
fi
76+
77+
- name: Run models
78+
run: |
79+
if [ "${{ steps.check-state.outputs.state_exists }}" = "true" ]; then
80+
echo "Running modified models with full refresh..."
81+
uv run dbt run --select state:modified+ --state ./state --full-refresh
82+
else
83+
echo "First merge to main - running all models..."
84+
uv run dbt run --full-refresh
85+
fi
86+
87+
- name: Test models
88+
run: |
89+
if [ "${{ steps.check-state.outputs.state_exists }}" = "true" ]; then
90+
echo "Testing modified models..."
91+
uv run dbt test --select state:modified+ --state ./state
92+
else
93+
echo "Testing all models..."
94+
uv run dbt test
95+
fi
96+
97+
- name: Upload current manifest for next run
98+
if: always()
99+
uses: actions/upload-artifact@v4
100+
with:
101+
name: prod-manifest-latest
102+
path: target/manifest.json
103+
retention-days: 90
104+
overwrite: true # overwrite the artifact with the latest manifest, ensures only one artifact is uploaded
105+

‎.github/workflows/dbt_prod.yml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: dbt prod incremental
2+
3+
on:
4+
# schedule:
5+
# - cron: '0 * * * *' # Run every hour at minute 0
6+
# NOTE: Schedule is disabled by default. Uncomment when ready to enable hourly production runs.
7+
# Check the README.md to make sure you understand the credit costs associated with running dbt.
8+
workflow_dispatch: # Allow manual triggers
9+
10+
concurrency:
11+
group: dbt-prod-runs # Shared group with dbt_deploy.yml to prevent concurrent production runs
12+
cancel-in-progress: false # Don't cancel in-progress runs, queue instead
13+
14+
jobs:
15+
dbt-prod:
16+
runs-on: ubuntu-latest
17+
timeout-minutes: 30
18+
19+
env:
20+
DUNE_API_KEY: ${{ secrets.DUNE_API_KEY }}
21+
DUNE_TEAM_NAME: ${{ vars.DUNE_TEAM_NAME || 'dune' }} # Set as GitHub Variable or defaults to 'dune'
22+
DBT_TARGET: prod # All dbt commands will use the prod target from profiles.yml
23+
24+
steps:
25+
- name: Check out code
26+
uses: actions/checkout@v4
27+
with:
28+
ref: main
29+
30+
- name: Install uv
31+
uses: astral-sh/setup-uv@v5
32+
with:
33+
enable-cache: true
34+
35+
- name: Install dependencies
36+
run: uv sync --locked
37+
38+
- name: Install dbt packages
39+
run: uv run dbt deps
40+
41+
- name: Run models
42+
run: uv run dbt run --exclude config.materialized:view # views should only run when modified
43+
44+
- name: Test models
45+
run: uv run dbt test --exclude config.materialized:view # views should only be tested when modified

‎.gitignore‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Python
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
*.so
6+
.Python
7+
.venv/
8+
venv/
9+
ENV/
10+
env/
11+
12+
# uv
13+
.python-version
14+
15+
# .user.yml
16+
.user.yml
17+
18+
# dbt
19+
target/
20+
dbt_packages/
21+
logs/
22+
23+
# IDE
24+
.vscode/
25+
.idea/
26+
*.swp
27+
*.swo
28+
*~
29+
.DS_Store
30+
31+
# Cursor - SQL style guide is personal preference, keep gitignored
32+
.cursor/rules/sql-style-guide.mdc
33+
34+
# Environment
35+
.env
36+
.env.local
37+
.envrc
38+
39+
# Compiled files
40+
*.pyc
41+

‎CHANGELOG.md‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Changelog
2+
3+
All notable changes to this template will be documented in this file.
4+
5+
## [v1.3.0] - 2026-01-05
6+
7+
### Changed
8+
- **Partition Configuration Format**: Updated partition strategy to use `properties = { "partitioned_by": "ARRAY['column']" }` format (#50)
9+
- Changed from `partition_by` to proper `properties` configuration for Dune transformations endpoint compatibility
10+
- Updated all model templates (table, merge incremental, delete+insert incremental, append incremental) with partition examples
11+
- Updated documentation with correct partition configuration syntax and examples
12+
- Ensures proper partition assignment and table building using transformations endpoint
13+
14+
## [v1.2.0] - 2025-11-14
15+
16+
### Changed
17+
- **Table Materialization Strategy**: Removed project-level `on_table_exists: replace` config (#46)
18+
- Now uses dbt-trino default strategy (temp table → rename sequence)
19+
- Allows proper schema change handling during full refreshes
20+
- Updated documentation to reflect this change (#48)
21+
22+
### Added
23+
- **Table Maintenance Automation**: Global post-hooks for Delta Lake table optimization (#45)
24+
- Automatic `OPTIMIZE` command after table/incremental materializations
25+
- Automatic `VACUUM` command to clean up old files
26+
- Improves query performance and reduces storage costs
27+
- **Table/View Drop Script**: Python utility script for manual table/view cleanup (#43)
28+
- Supports dropping tables and views across dev/prod environments
29+
- Uses Dune API key for authentication
30+
- Helpful for schema migrations and cleanup tasks
31+
- **Source Read Recommendations**: Documentation for efficient source filtering (#44)
32+
- Best practices for lookback periods on blockchain data
33+
- Guidance on date-based filtering strategies
34+
35+
### Fixed
36+
- **Troubleshooting Documentation**: Added guidance for `DELTA_LAKE_BAD_WRITE` errors when using `on_table_exists: replace` with schema changes
37+
38+
## [1.1.1] - 2025-10-30
39+
40+
### Changed
41+
- **API Endpoint Update**: Updated Dune Trino API host from `dune-api-trino.dune.com` to `trino.api.dune.com` in profiles.yml (#40)
42+
43+
### Added
44+
- **Security Documentation**: Added guidance for public repositories to require workflow approval for outside contributors (#41)
45+
- New section in SETUP_FOR_NEW_TEAMS.md explaining fork pull request workflow permissions
46+
- Protects secrets (DUNE_API_KEY) and prevents unauthorized workflow runs
47+
- Brief reference added to README.md GitHub Setup section
48+
49+
## [1.1.0] - 2025-10-23
50+
51+
### Changed
52+
- **Environment Variable Configuration**: Standardized on environment variables instead of `.env` file approach
53+
- Removed `.env.example` file
54+
- Updated documentation with multiple setup methods (shell profile, session export, inline)
55+
- Simplified getting started guide with link to detailed setup options
56+
57+
### Added
58+
- **GitHub Actions Workflow Enhancements**:
59+
- New `dbt_ci.yml` workflow (renamed from `dbt_run.yml`) for PR validation
60+
- New `dbt_deploy.yml` workflow for deploying modified models on push to main
61+
- Monthly schedule trigger on `dbt_deploy.yml` to prevent manifest artifact expiration (90-day limit)
62+
- Concurrency controls across production workflows to prevent concurrent writes
63+
- Automated manifest generation on first PR if none exists (with clear error handling)
64+
- State comparison logic using manifest artifacts for efficient modified-only runs
65+
66+
### Improved
67+
- **Simplified `dbt_prod.yml`**: Streamlined to focus only on scheduled incremental model runs
68+
- **Workflow naming consistency**: Job names now match workflow file names for clarity
69+
- **Artifact management**: Proper cross-workflow artifact sharing with `dawidd6/action-download-artifact`
70+
- **Documentation**: Updated all workflow references and setup instructions
71+
72+
### Fixed
73+
- Artifact download configuration to properly reference workflow names for manifest retrieval
74+
75+
## [1.0.0] - 2025-10-21
76+
77+
### Added
78+
- Initial dbt template structure for Dune data transformations
79+
- Custom schema naming macro (dev vs prod targets)
80+
- Custom source macro with `delta_prod` database default
81+
- GitHub Actions workflows for CI/CD (PR validation and production runs)
82+
- Complete documentation in `docs/` directory
83+
- Setup guide for new teams
84+
- Python dependency management with `uv`
85+
- Model templates for all materialization types
86+
- Cursor AI rules for dbt best practices
87+
- Production schedule disabled by default for new template users
88+
- Upstream tracking documentation for template updates

0 commit comments

Comments
 (0)