Skip to content

Commit ac60007

Browse files
author
roller100 (BearingNode)
committed
refactor(dbt): Align csv_to_postgres scenario with community conventions
Follow-up to PR OpenLineage#186 feedback addressing final alignment issues: - Rename csv_to_postgres_local to csv_to_postgres (removes 'local' qualifier) - Remove README.md from scenario (community pattern uses scenario.md only) - Update documentation to reflect CI/CD service container deployment model - Correct residual DuckDB references to PostgreSQL Refs: OpenLineage#186 Signed-off-by: roller100 (BearingNode) <[email protected]>
1 parent 053b2c3 commit ac60007

File tree

10 files changed

+10
-11
lines changed

10 files changed

+10
-11
lines changed

producer/dbt/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,7 @@ If you need to debug event generation locally:
154154
```bash
155155
# Using the test runner CLI (same as GitHub Actions uses)
156156
python test_runner/cli.py run-scenario \
157-
--scenario csv_to_postgres_local \
157+
--scenario csv_to_postgres \
158158
--output-dir ./test_output/$(date +%s)
159159
160160
# List available scenarios
File renamed without changes.

producer/dbt/scenarios/csv_to_postgres_local/events/column_lineage_event.json renamed to producer/dbt/scenarios/csv_to_postgres/events/column_lineage_event.json

File renamed without changes.

producer/dbt/scenarios/csv_to_postgres_local/events/lineage_event.json renamed to producer/dbt/scenarios/csv_to_postgres/events/lineage_event.json

File renamed without changes.

producer/dbt/scenarios/csv_to_postgres_local/events/schema_event.json renamed to producer/dbt/scenarios/csv_to_postgres/events/schema_event.json

File renamed without changes.

producer/dbt/scenarios/csv_to_postgres_local/events/sql_event.json renamed to producer/dbt/scenarios/csv_to_postgres/events/sql_event.json

File renamed without changes.

producer/dbt/scenarios/csv_to_postgres_local/maintainers.json renamed to producer/dbt/scenarios/csv_to_postgres/maintainers.json

File renamed without changes.

producer/dbt/scenarios/csv_to_postgres_local/scenario.md renamed to producer/dbt/scenarios/csv_to_postgres/scenario.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
# CSV to PostgreSQL Local Scenario
1+
# CSV to PostgreSQL Scenario
22

33
## Overview
44

5-
This scenario validates dbt's OpenLineage integration compliance using synthetic test data in a controlled CSV → dbt → PostgreSQL pipeline with local file transport.
5+
This scenario validates dbt's OpenLineage integration compliance using synthetic test data in a controlled CSV → dbt → PostgreSQL pipeline with file transport.
66

77
**Purpose**: Compatibility testing and validation, not production use case demonstration.
88

@@ -40,8 +40,8 @@ Synthetic customer analytics scenario designed for validation testing:
4040

4141
- **Source**: Synthetic CSV files with test customer and order data
4242
- **Transform**: dbt models with staging and analytics layers
43-
- **Target**: DuckDB database (local file)
44-
- **Transport**: OpenLineage file transport (JSONL events)
43+
- **Target**: PostgreSQL database (CI/CD service container)
44+
- **Transport**: OpenLineage file transport (JSON Lines format)
4545
- **Validation**: Comprehensive facet compliance testing
4646

4747
## Expected Outputs

producer/dbt/scenarios/csv_to_postgres_local/README.md

Lines changed: 0 additions & 1 deletion
This file was deleted.

producer/dbt/test_runner/README.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -46,11 +46,11 @@ The atomic test runner validates:
4646

4747
1. **Environment Availability**
4848
- dbt command availability
49-
- DuckDB Python package installation
49+
- PostgreSQL adapter package installation
5050

5151
2. **dbt Project Creation**
5252
- Minimal dbt project structure
53-
- Profile configuration for DuckDB
53+
- Profile configuration for PostgreSQL
5454

5555
3. **dbt Execution**
5656
- Model compilation and execution
@@ -62,7 +62,7 @@ The atomic test runner validates:
6262

6363
## CLI Commands
6464

65-
- `check-environment`: Verify dbt and DuckDB availability
65+
- `check-environment`: Verify dbt and PostgreSQL adapter availability
6666
- `run-atomic`: Run all atomic validation tests
6767
- `setup`: Install dependencies (requires virtual environment)
6868

@@ -73,6 +73,6 @@ This test runner provides the foundation for OpenLineage event validation. When
7373
## Troubleshooting
7474

7575
1. **Python Environment Issues**: Use virtual environment as shown above
76-
2. **dbt Not Found**: Install dbt-core and dbt-duckdb in your environment
77-
3. **DuckDB Issues**: Ensure duckdb Python package is installed
76+
2. **dbt Not Found**: Install dbt-core and dbt-postgres in your environment
77+
3. **PostgreSQL Issues**: Ensure psycopg2-binary Python package is installed
7878
4. **Permission Errors**: Make sure scripts are executable (`chmod +x`)

0 commit comments

Comments
 (0)