diff --git a/mintlify/tutorials/manage-projects-with-terraform.mdx b/mintlify/tutorials/manage-projects-with-terraform.mdx index 1270201a8..01cb25f8e 100644 --- a/mintlify/tutorials/manage-projects-with-terraform.mdx +++ b/mintlify/tutorials/manage-projects-with-terraform.mdx @@ -27,24 +27,29 @@ This tutorial is part of the **Bytebase Terraform Provider** series: ## What You'll Learn -Create projects to organize your databases into logical groups. +In this tutorial, you'll learn how to: + +- **Organize** databases into logical project groups for better management +- **Configure** project-specific settings and policies +- **Set up** webhooks for project notifications +- **Choose** between dynamic and explicit database assignment patterns ## Prerequisites Before starting this tutorial, ensure you have: -- **Completed Parts 1 & 2**: Environments and instances configured -- **Built-in instances**: `Test Sample Instance` and `Prod Sample Instance` from sample data -- **Terraform workspace**: From previous tutorials +- Completed [Part 1: Manage Environments with Terraform](/tutorials/manage-environments-with-terraform) and [Part 2: Manage Databases with Terraform](/tutorials/manage-databases-with-terraform) +- Access to the built-in `Test Sample Instance` and `Prod Sample Instance` from Bytebase sample data +- Your Terraform workspace from the previous tutorials ## Step 1 - Understand Current Setup -From previous tutorials, you have: +From the previous tutorials, your Bytebase workspace contains: -- **Test instance** with `hr_test` and `postgres` databases -- **Prod instance** with `hr_prod` and `postgres` databases +- **Test instance**: Contains `hr_test` and `postgres` databases +- **Prod instance**: Contains `hr_prod` and `postgres` databases -Currently, these `hr_test` and `hr_prod` databases are in the default `Sample Project`, while `postgres` databases are unassigned. You can view them in Bytebase by clicking **Databases** on the left sidebar. Let's organize them into new projects. +Currently, the `hr_test` and `hr_prod` databases are assigned to the default `Sample Project`, while the `postgres` databases remain unassigned. You can verify this by navigating to **Databases** in the left sidebar. We'll now organize these databases into dedicated projects for better management. ## Step 2 - Create Projects @@ -56,39 +61,77 @@ Currently, these `hr_test` and `hr_prod` databases are in the default `Sample Pr Create `3-projects.tf`: ```hcl 3-projects.tf -# Project One - All test databases +# Project One - Development/Test Environment +# Uses dynamic assignment to automatically include all test databases resource "bytebase_project" "project_one" { depends_on = [bytebase_instance.test] resource_id = "project-one" title = "Project One" - # Automatically include all databases from test instance + # Project-level configurations for development workflow + allow_modify_statement = true # Allow direct SQL modifications + auto_resolve_issue = false # Manual issue resolution for review + auto_enable_backup = false # Disable automatic backups for test data + + # Dynamically include all databases from the test instance databases = bytebase_instance.test.databases + + # Configure Slack notifications for team collaboration + webhooks { + title = "Sample webhook 1" + type = "SLACK" + url = "https://webhook.site/91fcd52a-39f1-4e7b-a43a-ddf72796d6b1" + notification_types = [ + "NOTIFY_ISSUE_APPROVED", + "NOTIFY_PIPELINE_ROLLOUT", + "ISSUE_CREATE", + ] + } } -# Project Two - Specific production database +# Project Two - Production Environment +# Uses explicit assignment for precise control over production databases resource "bytebase_project" "project_two" { depends_on = [bytebase_instance.prod] resource_id = "project-two" title = "Project Two" - # Explicitly specify which database to include + # Explicitly specify which production databases to include databases = [ "instances/prod-sample-instance/databases/hr_prod" ] } ``` -**Two Assignment Patterns** + +The configuration above demonstrates project-level settings including webhook notifications. For a complete list of available project configurations, refer to the [Terraform provider documentation](https://registry.terraform.io/providers/bytebase/bytebase/latest/docs/data-sources/project). + + You can also configure SQL review policies at the project level. Learn more in [Part 5: Manage SQL Review Rules with Terraform](/tutorials/manage-sql-review-rules-with-terraform). + -1. **Dynamic**: `databases = bytebase_instance.test.databases` +### Database Assignment Patterns - - Automatically includes all databases from an instance - - Good for development/test environments +Choose the appropriate pattern based on your project's needs: + +#### Dynamic Assignment + +```hcl +databases = bytebase_instance.test.databases +``` -1. **Explicit**: `databases = ["instances/.../databases/..."]` - - Manually specify each database - - Better control for production +- **Automatically includes** all databases from a specified instance +- **Best for**: Development and testing environments where databases change frequently +- **Benefit**: Self-maintaining as new databases are automatically included + +#### Explicit Assignment + +```hcl +databases = ["instances/prod-sample-instance/databases/hr_prod"] +``` + +- **Manually specify** each database by its full path +- **Best for**: Production environments requiring precise control +- **Benefit**: Granular control over which databases belong to the project ## Step 3 - Apply Configuration @@ -99,27 +142,33 @@ terraform apply ## Step 4 - Verify Projects -1. Go to **Projects** in Bytebase. +1. Navigate to **Projects** in the Bytebase GUI. -1. You'll see three projects: +1. You should now see three projects organized as follows: - - **Sample Project** (original, now empty) + - **Sample Project** (original default project, now empty after database migration) - - **Project One** (contains `hr_test` and `postgres` which are both from `Test Sample Instance`) + - **Project One** (contains both `hr_test` and `postgres` databases from `Test Sample Instance`) ![project-1](/content/docs/tutorials/manage-projects-with-terraform/bb-project-1.webp) - - **Project Two** (contains `hr_prod` which is from `Prod Sample Instance`) + - **Project Two** (contains the `hr_prod` database from `Prod Sample Instance`) ![project-2](/content/docs/tutorials/manage-projects-with-terraform/bb-project-2.webp) +1. Click on **Project One** and navigate to **Settings** to review the configurations you've applied. + +1. Click **Webhooks** and you'll see the webhook you set. + ## Summary and Next Steps -You've successfully organized databases into projects! Key takeaways: +Congratulations! You've successfully organized your databases into logical projects. Here are the key concepts you've learned: -- **Projects group related databases** for better organization -- **Two assignment patterns**: dynamic (all databases) or explicit (specific databases) -- **Projects span environments**: can include both test and prod databases +- **Project Organization**: Projects serve as logical containers that group related databases for improved management and governance +- **Assignment Flexibility**: Choose between dynamic assignment (automatic inclusion) for development environments or explicit assignment (manual control) for production +- **Cross-Environment Projects**: Projects can span multiple environments, though it's often better to separate test and production databases into different projects +- **Project Configuration**: Customize project behavior with settings like statement modification permissions, automatic issue resolution, and backup policies +- **Notification Integration**: Set up webhooks to keep your team informed about project activities through Slack or other channels - The examples below show some SQL Review rules. For a complete list of available rules and their configurations, see the above terraform provider resource documentation. - + + The examples below demonstrate some SQL Review rules. For a comprehensive list of available rules and their configuration options, refer to the Terraform provider resource documentation linked above. + Create `5-sql-review.tf` with the SQL review configuration: @@ -77,19 +80,19 @@ resource "bytebase_review_config" "sample" { title = "Sample SQL Review Config" enabled = true - # Apply to production environment only + # Apply rules to production environment only resources = toset([ bytebase_setting.environments.environment_setting[0].environment[1].name ]) - # Rule 1: Warn about nullable columns + # Rule 1: Column nullability - warn about nullable columns to improve data integrity rules { type = "column.no-null" engine = "POSTGRES" level = "WARNING" } - # Rule 2: Require specific columns + # Rule 2: Required audit columns - enforce standard tracking fields rules { type = "column.required" engine = "POSTGRES" @@ -97,14 +100,14 @@ resource "bytebase_review_config" "sample" { payload = "{\"list\":[\"id\",\"created_ts\",\"updated_ts\",\"creator_id\",\"updater_id\"]}" } - # Rule 3: Tables must have primary key + # Rule 3: Primary key requirement - ensure all tables have primary keys rules { type = "table.require-pk" engine = "POSTGRES" level = "ERROR" } - # Rule 4: Column naming convention + # Rule 4: Column naming standards - enforce snake_case naming convention rules { type = "naming.column" engine = "POSTGRES" @@ -112,7 +115,7 @@ resource "bytebase_review_config" "sample" { payload = "{\"format\":\"^[a-z]+(_[a-z]+)*$\",\"maxLength\":64}" } - # Rule 5: Limit maximum rows in SELECT + # Rule 5: Query performance protection - limit maximum rows in SELECT statements rules { type = "statement.maximum-limit-value" engine = "POSTGRES" @@ -122,6 +125,10 @@ resource "bytebase_review_config" "sample" { } ``` + + The configuration above applies SQL Review rules specifically to the `Prod` environment. Alternatively, you can scope rules to a specific project by using `bytebase_project.sample_project.name` in the `resources` attribute. + + ### Step 2 - Apply Configuration ```bash @@ -131,81 +138,81 @@ terraform apply ### Step 3 - Verify in Bytebase -1. Go to **CI/CD > SQL Review** on the left sidebar, you should see `Sample SQL Review Config` listed. Note that it's applied to the `Prod` environment. +1. Navigate to **CI/CD > SQL Review** in the left sidebar. You should see `Sample SQL Review Config` listed, showing that it's applied to the `Prod` environment. ![sql-review-list](/content/docs/tutorials/manage-sql-review-rules-with-terraform/bb-sql-review-list.webp) -1. Click on it to view the configured rules. +1. Click on the configuration to examine the rules you've defined. ![sql-review-pg](/content/docs/tutorials/manage-sql-review-rules-with-terraform/bb-sql-review-pg.webp) ### Understanding SQL Review Rules -SQL Review rules are categorized by what they validate: +SQL Review rules are organized into categories based on what they validate: #### 1. Column Rules -- **column.no-null**: Warns about nullable columns to encourage data integrity -- **column.required**: Enforces required columns (id, timestamps, audit fields) for consistency +- **column.no-null**: Issues warnings about nullable columns to promote data integrity and prevent null-related errors +- **column.required**: Mandates the presence of essential columns (ID, timestamps, audit fields) to ensure consistent table structure #### 2. Table Rules -- **table.require-pk**: Ensures every table has a primary key for data integrity and performance +- **table.require-pk**: Guarantees that every table includes a primary key, which is crucial for data integrity, replication, and query performance #### 3. Naming Rules -- **naming.column**: Enforces lowercase snake_case column names for consistency across your schema +- **naming.column**: Enforces consistent lowercase snake_case naming conventions across your database schema, improving readability and maintainability #### 4. Statement Rules -- **statement.maximum-limit-value**: Prevents SELECT queries without proper limits to avoid performance issues +- **statement.maximum-limit-value**: Prevents potentially harmful SELECT queries by requiring appropriate LIMIT clauses, protecting against performance degradation ### Step 4 - Test SQL Review -Create a test SQL statement to see the review in action: +Let's validate that your SQL Review rules are working by testing them with some sample SQL: -1. Go to **Project Two**, and click **Database > Databases** on the left sidebar. -1. Check `hr_prod` and click **Edit Schema**. -1. Try creating a table that violates the rules: +1. Navigate to **Project Two**, then click **Database > Databases** in the left sidebar. +1. Select the `hr_prod` database and click **Edit Schema**. +1. Create a table that intentionally violates multiple rules: ```sql - -- This will trigger multiple violations + -- This table intentionally violates multiple SQL review rules CREATE TABLE BadExample ( - FirstName VARCHAR(50), -- Violates naming convention - LastName VARCHAR(50) -- Missing required columns, no primary key + FirstName VARCHAR(50), -- Violates snake_case naming convention + LastName VARCHAR(50) -- Missing required audit columns and primary key ); ``` Expected violations: - - ❌ Column naming convention (should be first_name) - - ❌ Missing required columns (id, created_ts, etc.) + - ❌ Column naming convention violation (should be `first_name`, `last_name`) + - ❌ Missing required audit columns (`id`, `created_ts`, `updated_ts`, `creator_id`, `updater_id`) - ❌ No primary key defined ![sql-review-violations](/content/docs/tutorials/manage-sql-review-rules-with-terraform/bb-sql-review-violations.webp) -1. Even if you continue to create an issue, it will still show red SQL Review. +1. Even if you proceed to create an issue, the SQL Review violations will be prominently displayed with error indicators. ![issue-sql-review-errors](/content/docs/tutorials/manage-sql-review-rules-with-terraform/bb-issue-sql-review-errors.webp) -1. Try a compliant table: +1. Now try creating a compliant table that follows all your rules: ```sql - -- This follows all rules + -- This table complies with all SQL review rules CREATE TABLE good_example ( - id SERIAL PRIMARY KEY, - first_name VARCHAR(50) NOT NULL, - last_name VARCHAR(50) NOT NULL, - created_ts TIMESTAMP NOT NULL DEFAULT NOW(), - updated_ts TIMESTAMP NOT NULL DEFAULT NOW(), - creator_id INTEGER NOT NULL, - updater_id INTEGER NOT NULL + id SERIAL PRIMARY KEY, -- Required primary key + first_name VARCHAR(50) NOT NULL, -- Snake_case naming, not null + last_name VARCHAR(50) NOT NULL, -- Snake_case naming, not null + created_ts TIMESTAMP NOT NULL DEFAULT NOW(), -- Required audit column + updated_ts TIMESTAMP NOT NULL DEFAULT NOW(), -- Required audit column + creator_id INTEGER NOT NULL, -- Required audit column + updater_id INTEGER NOT NULL -- Required audit column ); ``` Expected result: - - ✅ All required rules pass + - ✅ All SQL review rules pass successfully ![issue-sql-review-pass](/content/docs/tutorials/manage-sql-review-rules-with-terraform/bb-issue-sql-review-pass.webp) @@ -245,12 +252,12 @@ rules { ## Key Points - **Rule Levels**: - - `ERROR`: Blocks the SQL statement from being executed - - `WARNING`: Allows execution but notifies users of potential issues -- **Engine Specific**: Rules can target specific database engines (PostgreSQL, MySQL, etc.) -- **Environment Scoped**: Apply different rules to test vs. production environments -- **Payload Format**: Complex rules use JSON payloads for configuration (e.g., regex patterns, lists) -- **Best Practice**: Start with WARNING level rules and gradually move to ERROR as teams adapt + - `ERROR`: Blocks SQL statement execution, preventing non-compliant changes from proceeding + - `WARNING`: Permits execution while alerting users to potential issues, useful for gradual policy adoption +- **Engine Specificity**: Rules can be tailored to specific database engines (PostgreSQL, MySQL, Oracle, etc.) to accommodate platform differences +- **Environment Scoping**: Apply stricter rules to production while allowing more flexibility in development and testing environments +- **Payload Configuration**: Complex rules utilize JSON payloads for detailed configuration (regex patterns, value lists, thresholds) +- **Implementation Strategy**: Begin with `WARNING` level rules to familiarize teams with standards, then gradually transition to `ERROR` level for enforcement