Implement core secrets management library

[devin-ai-integration]

Overview

Implement the core airbyte-secrets library as a helper library for managing secrets across Airbyte tools (PyAirbyte, CDK, connector-builder-mcp).

Requested by: Aaron ("AJ") Steers (@aaronsteers)

Implementation Details

Core Components

SensitiveDict class

• Dict subclass indicating values may be sensitive
• sanitize() method to mask SecretStrings for safe logging
• Recursive handling of nested dicts

SecretString enhancements

• is_config() method to detect JSON/dotenv configs
• as_config() method returning SensitiveDict with auto-format detection
• Preserves SecretString values through nested structures

SecretManager base class

• Lowest common denominator interface
• Single get_secret(secret_name: str) -> SecretString | None method

SecretManagerSet

• Priority-based manager registry
• Methods: add(manager, priority), remove(), get_secret(), list_managers(), clear()
• Managers checked in priority order (highest first)

PrivateBinSecretManager

• Fetch secrets from PrivateBin URLs
• Password support via env var or constructor
• Content caching with dotenv parsing

Dotenv utilities

• Nested notation support (credentials.password=value → nested dict)
• Auto-detection of JSON vs dotenv format
• parse_dotenv_to_nested_dict() and merge_nested_dicts()

Project Setup

Package manager: uv (not Poetry)
Type checker: pyrefly (not mypy) - https://pyrefly.org/
Linter/formatter: ruff
Testing: pytest with pytest-cov

Dependencies:

• Core: python-dotenv>=1.0.0
• Optional: privatebin>=0.3.0, google-cloud-secret-manager>=2.16.0
• Dev: pytest, pytest-cov, ruff

Testing

• Comprehensive test suite for all core functionality
• Tests for SecretString, SensitiveDict, SecretManagerSet, dotenv parsing
• Mock-based tests for PrivateBinSecretManager

Documentation

• Updated README with examples
• Type hints throughout (py.typed marker)
• Docstrings for all public APIs

Design Principles

1. Helper library pattern: No global state management - consuming libraries create their own global instances
2. Type safety: Full type hints with SensitiveDict for nested configs
3. Security first: Masked representations, no secret exposure in logs
4. Flexibility: Priority-based manager ordering, pluggable implementations

Future Work (separate issues)

• DotenvSecretManager (reads from .env files)
• GoogleGSMSecretManager (Google Secret Manager integration)
• Additional secret manager implementations