improved doc

Author: albanpuech

.gitignore

@@ -20,3 +20,4 @@ opf_data/data
 test_data/
 scripts/datasets_sampled/
 pfdelta/data/
+site/

.pre-commit-config.yaml

@@ -7,7 +7,7 @@ repos:
     -   id: check-yaml
     -   id: debug-statements
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.13.3
+  rev: v0.14.10
   hooks:
     - id: ruff-check
     - id: ruff-format
@@ -17,7 +17,7 @@ repos:
     -   id: flake8
         args: ["--ignore=E501,W503,E203"]
 -   repo: https://github.com/asottile/add-trailing-comma
-    rev: v3.2.0
+    rev: v4.0.0
     hooks:
     -   id: add-trailing-comma
 -   repo: https://github.com/ibm/detect-secrets

.secrets.baseline

@@ -3,7 +3,7 @@
     "files": "^.secrets.baseline$",
     "lines": null
   },
-  "generated_at": "2025-10-03T08:52:13Z",
+  "generated_at": "2025-12-28T17:31:41Z",
   "plugins_used": [
     {
       "name": "AWSKeyDetector"

README.md

@@ -13,7 +13,44 @@
 ![License](https://img.shields.io/badge/license-Apache%202.0-blue)
 
 
-This library is brought to you by the GridFM team to generate power flow data to train machine learning and foundation models.
+---
+
+# GridFM DataKit
+
+**GridFM DataKit** (`gridfm-datakit`) is a Python library for generating realistic, diverse, and scalable synthetic datasets for power flow (PF) and optimal power flow (OPF) machine learning solvers. It unifies state-of-the-art methods for perturbing loads, generator dispatches, network topologies, and branch parameters, addressing limitations of existing data generation libraries.
+
+## Key Features
+
+* **Scalable**: Supports grids with up to 30,000 buses for PF and 10,000 buses for OPF. Compatible with MATPOWER (`.m`) files and the PGLib dataset.
+* **Realistic load scenarios**: Combines global scaling from real-world aggregated profiles with localized per-bus noise, preserving temporal and spatial correlations.
+* **Flexible topology perturbations**: Handles arbitrary (N-k) outages for lines, transformers, and generators, ensuring feasible network states.
+* **Generator cost diversity**: Permutes or randomly scales generator cost functions when solving OPF to produce diverse dispatches and improve generalization across different cost conditions.
+* **Out-of-operating-limits scenarios for PF**: PF datasets include realistic violations of operating limits (e.g., voltage or branch overloads) resulting from topology and load perturbations without re-optimizing generator dispatch.
+* **Admittance perturbations**: Randomly scales branch resistances and reactances to enhance diversity.
+* **Structured outputs for ML**: Per-bus, per-branch, and per-generator data ready for training neural PF/OPF solvers, with pre-computed DC-PF and DC-OPF baselines and runtime.
+* **Data validation and benchmarking**: Includes CLI tools for consistency checks, statistics, and constraint validation.
+
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/gridfm/gridfm-datakit/refs/heads/main/docs/figs/comparison_table.png" alt="Comparison table" style="width: 40%; height: auto;"/>
+  <br/>
+</p>
+
+## Citation
+
+Please cite the library when using it in your work:
+
+```bibtex
+@misc{puech2025gridfmdatakitv1pythonlibraryscalable,
+      title={gridfm-datakit-v1: A Python Library for Scalable and Realistic Power Flow and Optimal Power Flow Data Generation},
+      author={Alban Puech and Matteo Mazzonelli and Celia Cintas and Tamara R. Govindasamy and Mangaliso Mngomezulu and Jonas Weiss and Matteo Baù and Anna Varbella and François Mirallès and Kibaek Kim and Le Xie and Hendrik F. Hamann and Etienne Vos and Thomas Brunschwiler},
+      year={2025},
+      eprint={2512.14658},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2512.14658},
+}
+```
 
 ---
 

docs/components/admittance_perturbation.md

@@ -0,0 +1,18 @@
+# Admittance Perturbations
+
+
+## Base Classes
+
+### `AdmittanceGenerator`
+
+::: gridfm_datakit.perturbations.admittance_perturbation.AdmittanceGenerator
+
+## Concrete Implementations
+
+### `NoAdmittancePerturbationGenerator`
+
+::: gridfm_datakit.perturbations.admittance_perturbation.NoAdmittancePerturbationGenerator
+
+### `PerturbAdmittanceGenerator`
+
+::: gridfm_datakit.perturbations.admittance_perturbation.PerturbAdmittanceGenerator

docs/components/generator_perturbation.md

@@ -0,0 +1,21 @@
+# Generator Perturbations
+
+## Base Classes
+
+### `GenerationGenerator`
+
+::: gridfm_datakit.perturbations.generator_perturbation.GenerationGenerator
+
+## Concrete Implementations
+
+### `NoGenPerturbationGenerator`
+
+::: gridfm_datakit.perturbations.generator_perturbation.NoGenPerturbationGenerator
+
+### `PermuteGenCostGenerator`
+
+::: gridfm_datakit.perturbations.generator_perturbation.PermuteGenCostGenerator
+
+### `PerturbGenCostGenerator`
+
+::: gridfm_datakit.perturbations.generator_perturbation.PerturbGenCostGenerator

docs/components/power_balance.md

@@ -0,0 +1,17 @@
+# Power Balance
+
+This module provides functions for computing power flow balance and branch admittances.
+
+## Functions
+
+### `compute_branch_admittances`
+
+::: gridfm_datakit.utils.power_balance.compute_branch_admittances
+
+### `compute_branch_powers_vectorized`
+
+::: gridfm_datakit.utils.power_balance.compute_branch_powers_vectorized
+
+### `compute_bus_balance`
+
+::: gridfm_datakit.utils.power_balance.compute_bus_balance

docs/components/utils.md

@@ -2,6 +2,16 @@
 
 This module provides utility functions for the project.
 
+## Functions
+
+### `get_num_scenarios`
+
+::: gridfm_datakit.utils.utils.get_num_scenarios
+
+### `read_partitions`
+
+::: gridfm_datakit.utils.utils.read_partitions
+
 ### `write_ram_usage_distributed`
 
 ::: gridfm_datakit.utils.utils.write_ram_usage_distributed