Load_TPCDS_to_PG

The 'Load_TPCDS_to_PG' project was generated by using the default-python template.

Getting started

1. Install the Databricks CLI from https://docs.databricks.com/dev-tools/cli/databricks-cli.html

2. Authenticate to your Databricks workspace, if you have not done so already:

   $ databricks configure
   

3. To deploy a development copy of this project, type:

   $ databricks bundle deploy --target dev
   

   (Note that "dev" is the default target, so the --target parameter is optional here.)

   This deploys everything that's defined for this project. For example, the default template would deploy a job called [dev yourname] Load_TPCDS_to_PG_job to your workspace. You can find that job by opening your workpace and clicking on Workflows.

4. Similarly, to deploy a production copy, type:

   $ databricks bundle deploy --target prod
   

   Note that the default job from the template has a schedule that runs every day (defined in resources/Load_TPCDS_to_PG.job.yml). The schedule is paused when deploying in development mode (see https://docs.databricks.com/dev-tools/bundles/deployment-modes.html).

5. To run a job or pipeline, use the "run" command:

   $ databricks bundle run
   

6. Optionally, install developer tools such as the Databricks extension for Visual Studio Code from https://docs.databricks.com/dev-tools/vscode-ext.html.

7. For documentation on the Databricks asset bundles format used for this project, and for CI/CD configuration, see https://docs.databricks.com/dev-tools/bundles/index.html.

Databricks Secret Scope Management

This section explains how to create and manage secret scopes in Databricks for securely storing sensitive information like database credentials, API keys, and other configuration values.

Prerequisites

• Databricks CLI installed and configured
• Appropriate permissions in your Databricks workspace (Admin or Workspace Admin role)
• Access to your Databricks workspace

Using Databricks CLI Create Secrets

1. Create a Secret Scope

# Create a new secret scope
databricks secrets create-scope --scope <scope-name>

# Example:
databricks secrets create-scope --scope tpc-ds-config

2. Add Secrets to the Scope

# Add a secret to the scope
databricks secrets put --scope <scope-name> --key <key-name>

# Example - adding database connection string:
databricks secrets put --scope tpc-ds-config --key postgres-connection-string
# You'll be prompted to enter the secret value

# Example - adding database password:
databricks secrets put --scope tpc-ds-config --key postgres-password
# You'll be prompted to enter the secret value

# Example - adding API key:
databricks secrets put --scope tpc-ds-config --key api-key
# You'll be prompted to enter the secret value

3. List Secrets in a Scope

# List all secrets in a scope
databricks secrets list --scope <scope-name>

# Example:
databricks secrets list --scope tpc-ds-config

4. Delete Secrets or Scopes

# Delete a specific secret
databricks secrets delete --scope <scope-name> --key <key-name>

# Delete an entire scope (and all its secrets)
databricks secrets delete-scope --scope <scope-name>