Add json yaml toml supports

Author: lzjever

CHANGELOG.md

@@ -5,6 +5,58 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+- **File-Based Sources**: New YAML, JSON, and TOML sources for loading configuration from files
+  - `sources.YAML(file_path, ...)` - Load configuration from YAML files
+  - `sources.JSON(file_path, ...)` - Load configuration from JSON files
+  - `sources.TOML(file_path, ...)` - Load configuration from TOML files
+  - All file sources support nested configuration structures (automatically flattened to dot notation)
+  - All file sources support `required=False` parameter for graceful handling of missing files
+  - Missing files return empty dict and show "Not Available" status in `--check-variables`
+- **Source ID System**: Enhanced source identification for multiple sources of the same type
+  - Each source now has a unique `id` property (in addition to `name`)
+  - Custom source IDs can be specified via `source_id` parameter
+  - Automatic ID generation based on source type and key parameters
+  - `PriorityPolicy` now supports both source names and source IDs
+  - Multiple sources of the same type can be used with different priorities
+- **Enhanced Diagnostic Table**: Improved `--check-variables` output
+  - Added "Status" column showing source load status ("Active", "Not Available", "Failed: ...")
+  - Better error messages for missing files (shows "Not Available" instead of "Error")
+  - Source status tracking via `load_status` and `load_error` properties
+- **Improved Error Messages**: Required field errors now include field descriptions from metadata
+  - `RequiredFieldError` messages now show field descriptions when available
+  - More user-friendly error messages with actionable guidance
+
+### Changed
+- **Dependencies**: Moved `python-dotenv`, `pyyaml`, and `tomli` from optional to core dependencies
+  - `dotenv`, `yaml`, and `toml` sources are now always available
+  - Only `etcd` remains as an optional dependency
+- **Etcd Source**: Removed `Etcd.from_env()` method
+  - All parameters must now be passed explicitly via `__init__`
+  - Users should read environment variables themselves and pass to `Etcd()`
+  - This aligns with the principle that the library should not implicitly read environment variables for its own configuration
+- **Source Base Class**: Enhanced `Source` base class with status tracking
+  - Added `_load_status` attribute ("success", "not_found", "failed", "unknown")
+  - Added `_load_error` attribute for error messages
+  - Added `load_status` and `load_error` properties
+  - Modified `load()` to wrap `_do_load()` with proper error handling
+  - Subclasses should implement `_do_load()` instead of `load()`
+- **Key Mapping**: File-based sources (YAML, JSON, TOML) use recursive flattening
+  - Nested dictionaries are automatically flattened to dot notation (e.g., `{"db": {"host": "localhost"}}` → `{"db.host": "localhost"}`)
+  - Consistent with existing key mapping rules (`__` → `.`, `_` preserved, lowercase)
+- **Examples**: Updated all examples to use best practices
+  - Use nested dataclasses instead of double-underscore fields
+  - Include field descriptions in metadata
+  - Proper error handling with `RequiredFieldError`
+  - Support for `--help` and `-cv` flags
+
+### Fixed
+- Fixed "Unknown" status in diagnostic table for Env, CLI, Defaults, and DotEnv sources
+- Fixed source status tracking to correctly show "Active", "Not Available", or "Failed" status
+- Improved test coverage for file-based sources and multiple sources of the same type
+
 ## [0.5.0] - 2026-01-07
 
 ### Fixed

docs/source/examples/file_sources_example.rst

@@ -0,0 +1,9 @@
+File Sources Example
+====================
+
+Example demonstrating YAML, JSON, and TOML file sources.
+
+.. literalinclude:: ../../../examples/file_sources_example.py
+   :language: python
+   :linenos:
+

docs/source/examples/index.rst

@@ -11,5 +11,6 @@ Real-world usage examples.
    validation_example
    nested_validation_example
    logging_example
+   file_sources_example
    etcd_example
 

docs/source/user_guide/etcd.rst

@@ -53,22 +53,33 @@ Create an Etcd source directly:
 From Environment Variables (Recommended)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The recommended approach is to use ``Etcd.from_env()`` to read configuration from environment variables. This solves the "bootstrap problem" of how to configure the etcd source itself.
+The recommended approach is to read environment variables yourself and pass them to ``Etcd()``. This aligns with the principle that the library should not implicitly read environment variables for its own configuration.
 
 .. code-block:: python
 
    from varlord import Config
    from varlord.sources import Etcd
    from dataclasses import dataclass, field
+   import os
 
    @dataclass
    class AppConfig:
        host: str = field()
        port: int = field(default=8000)
        debug: bool = field(default=False)
 
-   # From environment variables
-   etcd_source = Etcd.from_env(prefix="/app/")
+   # Read environment variables and pass to Etcd
+   etcd_source = Etcd(
+       host=os.environ.get("ETCD_HOST", "127.0.0.1"),
+       port=int(os.environ.get("ETCD_PORT", "2379")),
+       prefix=os.environ.get("ETCD_PREFIX", "/app/"),
+       ca_cert=os.environ.get("ETCD_CA_CERT"),
+       cert_key=os.environ.get("ETCD_CERT_KEY"),
+       cert_cert=os.environ.get("ETCD_CERT_CERT"),
+       user=os.environ.get("ETCD_USER"),
+       password=os.environ.get("ETCD_PASSWORD"),
+       watch=os.environ.get("ETCD_WATCH", "").lower() in ("true", "1", "yes", "on"),
+   )
 
    cfg = Config(
        model=AppConfig,
@@ -77,6 +88,8 @@ The recommended approach is to use ``Etcd.from_env()`` to read configuration fro
 
    app = cfg.load()
 
+**Note**: The ``Etcd.from_env()`` method has been removed. All parameters must be passed explicitly via ``__init__``.
+
 Environment Variables
 ---------------------
 
@@ -138,11 +151,20 @@ Then load it with python-dotenv:
 
    load_dotenv()  # Load .env file
 
+   import os
+   
+   etcd_source = Etcd(
+       host=os.environ.get("ETCD_HOST", "127.0.0.1"),
+       port=int(os.environ.get("ETCD_PORT", "2379")),
+       prefix=os.environ.get("ETCD_PREFIX", "/app/"),
+       ca_cert=os.environ.get("ETCD_CA_CERT"),
+       cert_key=os.environ.get("ETCD_CERT_KEY"),
+       cert_cert=os.environ.get("ETCD_CERT_CERT"),
+   )
+   
    cfg = Config(
        model=AppConfig,
-       sources=[
-           Etcd.from_env(),  # Read from environment variables
-       ],
+       sources=[etcd_source],
    )
 
 TLS Configuration
@@ -257,7 +279,15 @@ Enable watch support for dynamic configuration updates. Etcd source can watch fo
    cfg = Config(
        model=AppConfig,
        sources=[
-           Etcd.from_env(prefix="/app/", watch=True),
+           Etcd(
+               host=os.environ.get("ETCD_HOST", "127.0.0.1"),
+               port=int(os.environ.get("ETCD_PORT", "2379")),
+               prefix="/app/",
+               watch=True,
+               ca_cert=os.environ.get("ETCD_CA_CERT"),
+               cert_key=os.environ.get("ETCD_CERT_KEY"),
+               cert_cert=os.environ.get("ETCD_CERT_CERT"),
+           ),
        ],
    )
 
@@ -341,7 +371,15 @@ Watch events include:
    cfg = Config(
        model=AppConfig,
        sources=[
-           Etcd.from_env(prefix="/app/", watch=True),
+           Etcd(
+               host=os.environ.get("ETCD_HOST", "127.0.0.1"),
+               port=int(os.environ.get("ETCD_PORT", "2379")),
+               prefix="/app/",
+               watch=True,
+               ca_cert=os.environ.get("ETCD_CA_CERT"),
+               cert_key=os.environ.get("ETCD_CERT_KEY"),
+               cert_cert=os.environ.get("ETCD_CERT_CERT"),
+           ),
        ],
    )
 
@@ -371,7 +409,15 @@ For automatic updates, use ``ConfigStore`` which handles watch automatically:
    cfg = Config(
        model=AppConfig,
        sources=[
-           Etcd.from_env(prefix="/app/", watch=True),
+           Etcd(
+               host=os.environ.get("ETCD_HOST", "127.0.0.1"),
+               port=int(os.environ.get("ETCD_PORT", "2379")),
+               prefix="/app/",
+               watch=True,
+               ca_cert=os.environ.get("ETCD_CA_CERT"),
+               cert_key=os.environ.get("ETCD_CERT_KEY"),
+               cert_cert=os.environ.get("ETCD_CERT_CERT"),
+           ),
        ],
    )
 
@@ -402,7 +448,14 @@ Etcd source can be combined with other sources. Later sources override earlier o
    cfg = Config(
        model=AppConfig,
        sources=[
-           Etcd.from_env(prefix="/app/"),  # Load from etcd
+           Etcd(
+               host=os.environ.get("ETCD_HOST", "127.0.0.1"),
+               port=int(os.environ.get("ETCD_PORT", "2379")),
+               prefix="/app/",
+               ca_cert=os.environ.get("ETCD_CA_CERT"),
+               cert_key=os.environ.get("ETCD_CERT_KEY"),
+               cert_cert=os.environ.get("ETCD_CERT_CERT"),
+           ),  # Load from etcd
            Env(),                           # Env can override etcd
            CLI(),                           # CLI can override all
        ],
@@ -478,7 +531,7 @@ Nested Configuration
 Best Practices
 --------------
 
-1. **Use Environment Variables**: Use ``Etcd.from_env()`` instead of hardcoding connection parameters
+1. **Use Environment Variables**: Read environment variables yourself and pass them to ``Etcd()`` instead of hardcoding connection parameters
 2. **Use .env Files**: Manage configuration in development with ``.env`` files
 3. **Enable Watch**: Enable ``watch=True`` for configurations that need dynamic updates
 4. **Use Prefixes**: Use different etcd prefixes for different applications to avoid key conflicts