Skip to content

Commit 35a64b6

Browse files
corneel27jsimonettigithub-actions[bot]
authored
Migrate configuration to Pydantic (#591)
* Migrate configuration to Pydantic with validation and auto-generated documentation Replaces manual JSON configuration parsing with Pydantic models for type safety, validation, and automatic documentation generation. Includes migration system for backward compatibility. Changes: - Pydantic models for all configuration sections (battery, solar, EV, pricing, etc.) - Automatic migration from legacy configuration format - JSON Schema generation for IDE autocomplete - Auto-generated SETTINGS.md documentation - GitHub Actions workflow for documentation validation - Developer guide for extending configuration Technical details: - Zero hardcoded knowledge: all documentation generated from Pydantic JSON schema - Validation ensures all model fields have descriptions - Fallback mode preserves existing functionality if validation fails - Supports secrets management via !secret syntax * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * Use Type colums for link instead Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Use proper description for SecretStr fields Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * Use same pydantic version in CI Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Fix requirements installation for CI Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Fix CI schema Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Use a better way to check for model validation Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * Add additional validations Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * auto migrate to latest defined version This is a more explicit pattern Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Add extensive documentation and hints This will add in the self-describing of the code and also in the upcoming GUI. Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Create better settings.md based on additional fields We use the newly provided json extensions to generate better documentation. Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * Add better ui hints Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * update config models and call sites Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * update some more field defaults Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * update docs Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * add scheduler migration Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * update x- helpers Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * update dev docs Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * update to get ready for review Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * update db call sites that where missed Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * move backup action only when writing config on save Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * simplify migration logic Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * fix documentation output path Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * only migrate data if config version is not current Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * make loading more explicit Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Fix SecretStr resolving This not automatically resolves secrets with a plain text fallback. Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * Also fix FlexValue resolution Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * clarify secretstr serialisation Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * handle list types in models instead of at call site Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * fix encoding thing Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * fix stategy being a FlexValue Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * more cleanup and removal of dead code Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * fix CI Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * use model defaults Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * move sentinel add to config model Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * Add explicit loader errors Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Move config models to prog Also revert changed to pred to its 2026.03.0 state Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * add default and make some keys optional Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * made a few adjustments ["name"] to .name and dynamic defaults * adjusted hp_power from W to kW * changed obj['name'] to obj.name several places * made predictor work * implemented singleton for config and loader * removed all miplib stuff from run.sh * add save logic to migrations Save pre-migration config to options_{version}.json before migrating Save migrated config back to disk after migration Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * lock config during migration To prevent any races (unlikely, but possible) we lock the config file during migration. Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * remove obsolute method the load_secrets method is not used publicly, so we make that private Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * use pydantic model dump for save This ensures we save using proper key names, that do not conflict with json schema / jsonforms Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * rearranged the da-prediction stuff * don't dump null values we shouldn't save null values Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * add config file unit tests Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * coearse strings to booleans Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * update CI to not fail when updating docs Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * trigger doc rebuild Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * add concurrency lock to gen docs Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * trigger another doc build Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * another trigger Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Auto-generate documentation from Pydantic models - Updated SETTINGS.md - Updated config_schema.json Generated by GitHub Actions workflow * change CI to only comment when docs need updating Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * trigger rebuild Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Update generated documentation Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * use more comments to search Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * trigger again Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Update generated documentation Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * use workflow failure message in action Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Revert "use workflow failure message in action" This reverts commit c1841cb. Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * trigger again Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * Update generated documentation Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * change name of workflow to better indicate its function Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> * delete miplib archives * Version 2026.03.3.rc1 * update changelog * update prediction stuff * update Dockerfile: deleted line with miplib extraction --------- Signed-off-by: Jeroen Simonetti <jeroen@simonetti.nl> Co-authored-by: Jeroen Simonetti <jeroen@simonetti.nl> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
1 parent c369b63 commit 35a64b6

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

67 files changed

+12531
-904
lines changed
Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
name: Check Model Documentation
2+
3+
on:
4+
push:
5+
paths:
6+
- 'dao/prog/config/**/*.py'
7+
- 'scripts/generate_docs.py'
8+
pull_request:
9+
paths:
10+
- 'dao/prog/config/**/*.py'
11+
- 'scripts/generate_docs.py'
12+
- 'SETTINGS.md'
13+
- 'config_schema.json'
14+
15+
concurrency:
16+
group: ${{ github.workflow }}-${{ github.head_ref || github.ref_name }}-${{ github.event_name }}
17+
cancel-in-progress: true
18+
19+
jobs:
20+
check-docs:
21+
runs-on: ubuntu-latest
22+
permissions:
23+
contents: read
24+
pull-requests: write
25+
26+
steps:
27+
- name: Checkout repository
28+
uses: actions/checkout@v4
29+
30+
- name: Set up Python
31+
uses: actions/setup-python@v5
32+
with:
33+
python-version: '3.11'
34+
cache: 'pip'
35+
36+
- name: Install dependencies
37+
run: |
38+
python -m pip install --upgrade pip
39+
grep '^pydantic' dao/requirements.txt | xargs pip install
40+
41+
- name: Generate schema and documentation (with validation)
42+
id: generate
43+
run: |
44+
echo "Generating JSON schema and markdown documentation..."
45+
if python scripts/generate_docs.py; then
46+
echo "SUCCESS: Schema and documentation generated successfully"
47+
echo "validation=success" >> $GITHUB_OUTPUT
48+
else
49+
echo "ERROR: Generation failed - some fields missing descriptions"
50+
echo "validation=failed" >> $GITHUB_OUTPUT
51+
exit 1
52+
fi
53+
54+
- name: Check for changes
55+
id: check_changes
56+
run: |
57+
git diff --exit-code SETTINGS.md config_schema.json || echo "changed=true" >> $GITHUB_OUTPUT
58+
59+
- name: Manage PR comment for outdated docs
60+
if: github.event_name == 'pull_request'
61+
uses: actions/github-script@v7
62+
with:
63+
script: |
64+
const MARKER = '## \u26a0\ufe0f Documentation is out of date';
65+
const docsOutdated = '${{ steps.check_changes.outputs.changed }}' === 'true';
66+
67+
const { data: comments } = await github.rest.issues.listComments({
68+
owner: context.repo.owner,
69+
repo: context.repo.repo,
70+
issue_number: context.issue.number,
71+
per_page: 100,
72+
});
73+
74+
const existing = comments.find(c => c.body.includes(MARKER));
75+
76+
if (docsOutdated) {
77+
const body = [
78+
MARKER,
79+
'',
80+
'The generated files `SETTINGS.md` and `config_schema.json` are out of sync with the current Pydantic models.',
81+
'',
82+
'Please run the following commands locally and push the updated files:',
83+
'',
84+
'```bash',
85+
'python scripts/generate_docs.py',
86+
'git add SETTINGS.md config_schema.json',
87+
'git commit -m "Update generated documentation"',
88+
'git push',
89+
'```',
90+
].join('\n');
91+
92+
if (existing) {
93+
await github.rest.issues.updateComment({
94+
owner: context.repo.owner,
95+
repo: context.repo.repo,
96+
comment_id: existing.id,
97+
body,
98+
});
99+
} else {
100+
await github.rest.issues.createComment({
101+
owner: context.repo.owner,
102+
repo: context.repo.repo,
103+
issue_number: context.issue.number,
104+
body,
105+
});
106+
}
107+
} else if (existing) {
108+
await github.rest.issues.deleteComment({
109+
owner: context.repo.owner,
110+
repo: context.repo.repo,
111+
comment_id: existing.id,
112+
});
113+
}
114+
115+
- name: Fail if docs are outdated
116+
if: steps.check_changes.outputs.changed == 'true'
117+
run: |
118+
echo "ERROR: SETTINGS.md and/or config_schema.json are out of sync with the models."
119+
exit 1
120+
121+

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,3 +18,4 @@ __pycache__/
1818
../CODEOWNERS
1919
dao/data/*.csv
2020
/dao/data/batteries.json
21+
.venv

DEVELOP.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -43,17 +43,17 @@ Creating a virtual environment isolates your project dependencies from your syst
4343

4444
**On macOS/Linux:**
4545
```bash
46-
python3 -m venv venv
47-
source venv/bin/activate
46+
python3 -m venv .venv
47+
source .venv/bin/activate
4848
```
4949

5050
**On Windows:**
5151
```bash
52-
python -m venv venv
53-
venv\Scripts\activate
52+
python -m venv .venv
53+
.venv\Scripts\activate
5454
```
5555

56-
You should see `(venv)` in your terminal prompt, indicating the virtual environment is active.
56+
You should see `(.venv)` in your terminal prompt, indicating the virtual environment is active.
5757

5858
### 3. Install Dependencies
5959

@@ -146,9 +146,9 @@ From the project root (`day-ahead/`):
146146
cd /path/to/day-ahead
147147

148148
# Activate your virtual environment if not already active
149-
source venv/bin/activate # On macOS/Linux
149+
source .venv/bin/activate # On macOS/Linux
150150
# or
151-
venv\Scripts\activate # On Windows
151+
.venv\Scripts\activate # On Windows
152152

153153
# In case of manual mip install:
154154
export PMIP_CBC_LIBRARY=~/day-ahead/dao/prog/miplib/lib/libCbc.so

0 commit comments

Comments
 (0)