Migrate configuration to Pydantic by jsimonetti · Pull Request #484 · corneel27/day-ahead

jsimonetti · 2025-12-11T13:39:41Z

Pydantic Configuration Migration

Overview

This PR migrates the configuration system from manual JSON parsing to Pydantic models with automatic validation, type safety, and documentation generation.

Why Pydantic

The previous implementation used manual dictionary access with scattered validation logic throughout the codebase. Pydantic centralizes validation in model definitions, catches configuration errors at startup rather than runtime and automatically generates documentation from the same source. This eliminates the maintenance burden of keeping documentation synchronized with code, reduces bugs from typos or missing fields, and makes the configuration API self-documenting.

This migration also establishes the foundation for future improvements. The versioned configuration models enable proper migration paths when configuration changes are needed. The next migration can be to change the config to be compatible with jsonforms usage of json schema (migrating from database ha to database_ha. This migration can be handled completely by the config module without user intervention.

Review Guidance

Please review with caution - errors and issues are expected. This is a substantial refactoring of the configuration layer. The author will handle all issues until the maintainer is satisfied to merge.

Documentation Effort

Best effort has been made to document all configuration models by:

Scraping existing code for field usage and validation logic
Extracting information from DOCS.md
Inferring field purposes from context and variable names
Adding descriptions to all fields where possible

Some field descriptions may be incomplete or imprecise. Corrections and improvements are welcome.

What Changed

New Features

Type Safety: All configuration fields are now typed and validated
JSON Schema: Auto-generated schema for IDE autocomplete (config_schema.json)
Documentation: Auto-generated SETTINGS.md from model metadata
Migrations: Automatic migration from legacy configuration format
Validation: GitHub Actions ensures all fields have documentation
Fallback Mode: If validation fails, falls back to legacy behavior

New Files

dao/prog/config/ - Pydantic model definitions organized by category
dao/prog/config/versions/v0.py - Main configuration model
dao/prog/config/migrations/ - Migration system
dao/prog/config/generate_docs.py - Documentation generator
SETTINGS.md - Auto-generated configuration reference
config_schema.json - JSON Schema for IDE support
DEVELOPER_GUIDE.md - Guide for extending configuration
.github/workflows/generate-docs.yml - CI for documentation validation

Modified Files

dao/prog/da_config.py - Updated to use Pydantic models
dao/requirements.txt - Added pydantic dependency
.gitignore - Ignore Python cache files

Testing

The implementation preserves backward compatibility:

Existing configurations should work without changes
New type validation catches errors early
Fallback mode prevents breaking existing deployments

Known Limitations

Some field descriptions may need refinement
Migration from very old configurations may need testing
Not all edge cases may be covered

Future Work

Improve field descriptions based on maintainer feedback
Add more comprehensive validation rules
Extend documentation with examples

Author commitment: I will address all issues, bugs, and feedback until this is merge-ready.

Assorted list of things to change to the config after this merge, before we add save logic:
(This is just a TODO list from the author, no actions are required by the maintainer)

Migrate scheduler freeform setting to a SchedulerConfig model
Migrate Boolean values to be actual booleans instead of str | boolean
Add json schema validation logic and checks
Change model and field populate_by_name to make saving the field name instead of alias (possibly with spaces)
Create wiki pages from the configuration models similar to SETTINGS.md, but push them to the wiki instead.
Alter calls from DAO in to settings to make full use of the pydantic model, instead of the limited compatibility layer
Save options.json back to disk after migration

jsimonetti · 2025-12-11T13:54:32Z

I am unsure why CodeQL fails, since I did not touch the relevant db_manager.py file.

corneel27 · 2025-12-11T15:23:51Z

I am unsure why CodeQL fails, since I did not touch the relevant db_manager.py file.

First time same fail for my.
I investigated the message and I come to the conclusion: "false alarm"
The password will never printed to the logfile and the logfile stays locale.

You can dismiss the alerts.

corneel27 · 2025-12-11T15:41:00Z

I will investigate your work but not in a short time.
My "roadmap" is as follows:

Finish the rework for boiler/heatpump (this week, week 50)
Make a test-version of the implementation of your solar prediction module (next week, week 51)
Implement off user definable formules for the tariffs for consumption and production (the weeks of Christmas and New Year, weeks 52/2025 and week 1/2026)
This migration (week 2/2026)

Is that OK for you?

What is the best way to test?
Can I make a local branche of https://github.com/jsimonetti/day-ahead/tree/pydantic_configuration?

jsimonetti · 2025-12-11T16:13:28Z

This is absolutely fine!
You can directly checkout a PR.

See https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

jsimonetti · 2025-12-20T14:01:32Z

I have added even more help and ui-section/ui-group hints to the schema. Those ui hints can be ignored for now and serve as example for future usage. The help and validation hints will need to be looked at more carefully to make sure they make sense to an end user.
Perhaps we can ask a third person to review those parts, so you can focus on the technical implementation part.

…documentation Replaces manual JSON configuration parsing with Pydantic models for type safety, validation, and automatic documentation generation. Includes migration system for backward compatibility. Changes: - Pydantic models for all configuration sections (battery, solar, EV, pricing, etc.) - Automatic migration from legacy configuration format - JSON Schema generation for IDE autocomplete - Auto-generated SETTINGS.md documentation - GitHub Actions workflow for documentation validation - Developer guide for extending configuration Technical details: - Zero hardcoded knowledge: all documentation generated from Pydantic JSON schema - Validation ensures all model fields have descriptions - Fallback mode preserves existing functionality if validation fails - Supports secrets management via !secret syntax