Pydantic-Settings with Google Secret Manager

This repository demonstrates how to integrate pydantic-settings with Google Cloud Platform's Secret Manager to securely manage application configurations and secrets.

Overview

The example shows how to:

• Configure pydantic-settings to use Secret Manager as a settings source
• Retrieve structured configuration data from Secret Manager
• Use these settings in a FastAPI application

Requirements

• Google Cloud Platform account with Secret Manager enabled
• GCP credentials configured in your environment

Installation

1. Clone this repository

2. Install dependencies: If you are running in the devcontianer (or using uv) you can run:

   > uv venv && uv sync

   Otherwise, you can install the required packages using pip

   > pip install -e .
   # or
   > pip install fastapi pydantic-settings[gcp-secret-manager] uvicorn

How It Works

1. Settings Configuration

The settings.py file defines your application's configuration structure using Pydantic models. It configures the settings sources, including Secret Manager.

2. Using Settings in Your Application

The main.py file demonstrates how to use these settings in a FastAPI application.

3. Configuration Sources

pydantic-settings provides a flexible system for loading configuration values from multiple sources:

• Environment variables
• .env files
• Secret files
• JSON and TOML configuration files
• Google Secret Manager (new addition)

These sources can be prioritized and combined as needed. In this example, we demonstrate the integration with Secret Manager while preserving the ability to use other sources.

The key to enabling Secret Manager integration is to override the settings_customise_sources classmethod, instantiate GoogleSecretManagerSettingsSource, and then return the settings sources in the order of priority.

In our example, init_settings, env_settings, dotenv_settings, and file_secret_settings will all take priority of Secret Manager. So if you

4. Setting Up Secret Manager

For this example to work, you need to:

1. Create all of the relevant settings. In our example, we specified a default for environment, but not for the database credentials. Since these are nested attributes, we use the env_nested_delimiter we set - which was __ (double underscore). We either need this as environment variables, dotenv file settings or secrets in Secret Manager with names that match the settings structure:

   • spanner__user
   • spanner__password
   • firestore__user
   • firestore__password

2. Ensure your application has the necessary permissions to access these secrets. The application needs at least secretmanager.secrets.list and secretmanager.secrets.accessor permissions.

   • For example, with the predefined IAM roles,
     1. Secret Manager Viewer (roles/secretmanager.viewer): This role grants the "secretmanager.secrets.list" permission, allowing the user to list secrets and view their metadata.
     2. Secret Manager Secret Accessor (roles/secretmanager.secretAccessor): This role grants the "secretmanager.versions.access" permission, which is needed to access secret data.

Running the Example

1. Set up Secrets in Secret Manager:
   For this example, only the passwords were stored in Secret Manager.
   

2. Sign into your GCP environment locally:

   gcloud auth application-default login

3. Run the FastAPI application:
   Setting the usernames as environment variables.

   SPANNER__USER="FOO" FIRESTORE__USER="BAR" uvicorn main:app --reload

4. Access the /settings endpoint to see your settings:

   curl http://localhost:8000/settings

   The response will include your configuration from Secret Manager:

   

Benefits

• Security: Sensitive credentials never appear in code, environment variables, or config files
• Centralized Management: Manage all secrets in one place (GCP Secret Manager)
• Type Safety: Full type checking and validation of configuration through Pydantic
• Flexibility: Easy to combine with other configuration sources (environment variables, files, etc.)
• Hierarchical Settings: Support for nested configuration structures