Merge pull request #2 from duneanalytics/update-docs

update docs

Author: jeff-dude

QUICKSTART.md

@@ -1,20 +1,28 @@
 # Quick Start Guide
 
+Get up and running with the Dune DBT Template in minutes.
+
 ## Setup Checklist
 
 - [x] ✅ uv project initialized
 - [x] ✅ dbt-core and dbt-trino installed
-- [x] ✅ dbt project created
-- [ ] ⏳ Activate virtual environment with `source .venv/bin/activate`
-- [ ] ⏳ Configure your Trino connection in `~/.dbt/profiles.yml`
+- [x] ✅ dbt project created with example models
+- [x] ✅ Custom Dune macros configured
+- [x] ✅ GitHub Actions workflow ready
+- [ ] ⏳ Install dependencies with `uv sync`
+- [ ] ⏳ Install dbt packages with `dbt deps`
+- [ ] ⏳ Configure your Dune connection in `~/.dbt/profiles.yml`
 - [ ] ⏳ Test connection with `dbt debug`
 
 ## Next Steps
 
-### 1. Activate Virtual Environment (Recommended)
+### 1. Install Dependencies
 
 ```bash
-# Activate the virtual environment
+# Install Python dependencies
+uv sync
+
+# Activate the virtual environment (recommended)
 source .venv/bin/activate
 
 # Your prompt should now show (.venv) prefix
@@ -23,28 +31,40 @@ source .venv/bin/activate
 
 > **Alternative:** Skip activation and prefix all commands with `uv run` (e.g., `uv run dbt debug`)
 
-### 2. Configure Your Connection
+### 2. Install dbt Packages
+
+```bash
+# Install dbt_utils and other packages
+dbt deps
+```
+
+### 3. Configure Your Dune Connection
 
 ```bash
 # Create dbt config directory
 mkdir -p ~/.dbt
 
 # Copy example profile
-cp profiles.yml.example ~/.dbt/profiles.yml
+cp profiles_example_file.yml ~/.dbt/profiles.yml
 
-# Edit with your Trino connection details
+# Edit with your Dune connection details
 nano ~/.dbt/profiles.yml  # or use your preferred editor
 ```
 
 **Required settings in `profiles.yml`:**
-- `host`: Your Trino/Starburst server hostname
-- `port`: Usually 443 for TLS-enabled clusters
-- `database`: Catalog name in Trino
-- `schema`: Schema within the catalog
-- `user`: Your username
-- `password`: Your password (for LDAP auth)
 
-### 3. Test Your Connection
+For the `dbt_template_api` profile:
+- `host`: `dune-api-trino.dune.com` (prod) or `dune-api-trino.dev.dune.com` (dev)
+- `port`: `443`
+- `method`: `ldap`
+- `catalog`: `delta_prod`
+- `schema`: Your team name (e.g., `dune`)
+- `user`: Your team name
+- `password`: Your Dune API key
+- `session_properties.transformations`: `true` (required!)
+- `http_scheme`: `https`
+
+### 4. Test Your Connection
 
 ```bash
 # With activated venv (recommended):
@@ -59,37 +79,81 @@ You should see:
 All checks passed!
 ```
 
-### 4. Run Your First Models
+### 5. Run the Example Models
 
 ```bash
-# Run the example models (venv activated)
+# Run all models (venv activated)
 dbt run
 
 # Expected output:
-# - 2 models will be created as views in your Trino schema
+# - dbt_template_view_model (view)
+# - dbt_template_table_model (table)
+# - dbt_template_incremental_model (incremental)
+```
+
+### 6. Run Tests
+
+```bash
+# Run all tests
+dbt test
+
+# You should see the unique_combination_of_columns test pass
 ```
 
-### 5. Explore the Example Models
+### 7. Explore the Example Models
+
+The template includes three example models that demonstrate different materializations:
+
+**View Model** (`models/dbt_template_view_model.sql`):
+- Lightweight, always fresh
+- Counts transactions per block for last day
+- Good for fast-changing data
+
+**Table Model** (`models/dbt_template_table_model.sql`):
+- Static snapshot
+- Uses `on_table_exists='replace'` for Dune compatibility
+- Good for static or slowly changing data
 
-Look at these files to understand the structure:
-- `models/example/my_first_dbt_model.sql`
-- `models/example/my_second_dbt_model.sql`
-- `models/example/schema.yml`
+**Incremental Model** (`models/dbt_template_incremental_model.sql`):
+- Efficient updates with merge strategy
+- Processes last 1 day incrementally, last 7 days on full refresh
+- Includes `dbt_utils` test for uniqueness
+- Good for large, append-only datasets
 
-### 6. Create Your Own Models
+### 8. Test Incremental Updates
+
+```bash
+# Run with full refresh
+dbt run --full-refresh
+
+# Run incrementally (only processes last day)
+dbt run
+
+# The incremental model should update efficiently
+```
+
+### 9. Create Your Own Models
 
 ```bash
 # Create a new model file
-cat > models/my_model.sql << 'EOF'
-{{ config(materialized='table') }}
+cat > models/my_ethereum_model.sql << 'EOF'
+{{ config(
+    schema='test_schema',
+    alias='my_ethereum_model',
+    materialized='table'
+) }}
 
 select
-    1 as id,
-    'example' as name
+    block_date,
+    count(distinct "from") as unique_senders,
+    count(*) as tx_count
+from {{ source('ethereum', 'transactions') }}
+where block_date >= now() - interval '7' day
+group by block_date
 EOF
 
 # Run just your new model (venv activated)
-dbt run --select my_model
+dbt run --select my_ethereum_model
 ```
 
 ## Common Commands
@@ -127,22 +191,46 @@ deactivate
 uv run dbt <command>
 ```
 
+## Understanding the CI/CD Pipeline
+
+When you open a pull request, the GitHub Actions workflow automatically:
+
+1. ✅ Checks out your code
+2. ✅ Installs dependencies
+3. ✅ Waits for Trino cluster (up to 10 minutes)
+4. ✅ Compiles all models
+5. ✅ Runs full refresh
+6. ✅ Tests all models
+7. ✅ Runs incremental update
+8. ✅ Tests again to verify incremental logic
+
+All models in CI use the `test_schema` (configured in `generate_schema_name` macro).
+
 ## Troubleshooting
 
 ### "Profile not found" error
 - Make sure `~/.dbt/profiles.yml` exists
-- Verify the profile name matches `dbt_template` (as set in `dbt_project.yml`)
+- Verify the profile name is `dbt_template_api` (as set in `dbt_project.yml`)
+- Check that you copied from `profiles_example_file.yml`
 
 ### Connection errors
-- Check your Trino host, port, and credentials
-- Test connectivity: `curl https://your-trino-host:443`
-- Verify catalog and schema exist and you have access
-
-### "keyring module not found" warning
-This is just a warning. OAuth tokens won't be cached, but LDAP/JWT/other auth methods work fine. To fix:
-```bash
-uv add 'trino[external-authentication-token-cache]'
-```
+- Check your Dune API key and team name
+- Verify `transformations: true` in session properties
+- Test with: `dbt debug`
+- Check host: `dune-api-trino.dune.com` (not `.dev.dune.com` for prod)
+
+### "dbt_utils not found" error
+- Run `dbt deps` to install packages
+- Check that `packages.yml` exists
+
+### Incremental model not updating
+- Verify `unique_key` is set correctly
+- Check incremental logic with: `dbt compile --select dbt_template_incremental_model`
+- Use `--full-refresh` to rebuild: `dbt run --full-refresh`
+
+### CI/CD workflow stuck on "Waiting for runner"
+- Ensure your repo has access to the `spellbook-trino-ci` self-hosted runner
+- Check with your GitHub org admin to enable runner access
 
 ## Documentation