|
| 1 | +--- |
| 2 | +name: dpgen-simplify |
| 3 | +description: Prepare, explain, validate, and run DP-GEN simplify workflows for reducing repeated or redundant DeepMD datasets. Use when the user wants to generate or modify `param.json` and `machine.json`, run `dpgen simplify param.json machine.json`, organize repeated simplify experiments, or inspect simplify outputs. |
| 4 | +compatibility: Requires a runnable environment with Python and an activated DP-GEN runtime where `dpgen` is available in PATH for the outer simplify command. Real execution also requires DeePMD-kit and any backend-specific software required by the selected `fp_style`. For scheduler execution, each stage environment must be explicitly activated in `resources.source_list`. |
| 5 | +license: LGPL-3.0-or-later |
| 6 | +metadata: |
| 7 | + author: hyb1109 |
| 8 | + version: 0.2.0 |
| 9 | + repository: https://github.com/deepmodeling/dpgen |
| 10 | +--- |
| 11 | + |
| 12 | +# DP-GEN Simplify |
| 13 | + |
| 14 | +Use this skill when the user wants to prepare, explain, validate, or execute the `dpgen simplify` workflow. |
| 15 | + |
| 16 | +This skill is for dataset simplification workflows where the user already has candidate data in DeepMD-compatible format and wants to reduce repeated or redundant structures through iterative selection. |
| 17 | + |
| 18 | +## Core Rule (Critical) |
| 19 | + |
| 20 | +DP-GEN simplify always uses **two parameter classes** and therefore **two JSON files**: |
| 21 | + |
| 22 | +- **Workflow parameters** -> `param.json` |
| 23 | +- **Execution / machine parameters** -> `machine.json` |
| 24 | + |
| 25 | +Run exactly: |
| 26 | + |
| 27 | +```bash |
| 28 | +dpgen simplify param.json machine.json |
| 29 | +``` |
| 30 | + |
| 31 | +Environment boundary rule: |
| 32 | + |
| 33 | +- Outer layer: run `dpgen simplify param.json machine.json` in an activated environment where `dpgen --version` works. |
| 34 | +- Inner layer: for scheduler stages, explicitly activate runtime in `resources.source_list` on the server side. |
| 35 | + |
| 36 | +## Agent responsibilities |
| 37 | + |
| 38 | +When using this skill, the agent should: |
| 39 | + |
| 40 | +1. confirm that the task is a simplify workflow |
| 41 | +1. check whether existing configs or templates are already available |
| 42 | +1. collect only the missing dataset, training, FP, and machine inputs |
| 43 | +1. generate or patch `param.json` |
| 44 | +1. generate or patch `machine.json` |
| 45 | +1. explain important simplify parameters in plain language when asked |
| 46 | +1. validate the workflow before execution |
| 47 | +1. provide the exact command for running simplify |
| 48 | +1. if requested, help structure repeated experiments |
| 49 | +1. after execution, summarize outputs and next inspection targets |
| 50 | + |
| 51 | +## Working policy |
| 52 | + |
| 53 | +### 1. Ask only for missing inputs |
| 54 | + |
| 55 | +Do not ask the user for everything if part of the configuration is already available. |
| 56 | + |
| 57 | +If the user already provides: |
| 58 | + |
| 59 | +- a partial `param.json` |
| 60 | +- a partial `machine.json` |
| 61 | +- a known training template |
| 62 | +- a known cluster template |
| 63 | + |
| 64 | +then patch those files instead of rebuilding everything from scratch. |
| 65 | + |
| 66 | +### 2. Preserve the user's scientific choices |
| 67 | + |
| 68 | +Do not silently change: |
| 69 | + |
| 70 | +- descriptor family |
| 71 | +- fitting net structure |
| 72 | +- fp backend |
| 73 | +- trust thresholds |
| 74 | +- `type_map` ordering |
| 75 | + |
| 76 | +If a value looks scientifically questionable, explain the concern instead of silently replacing it. |
| 77 | + |
| 78 | +### 3. Keep local and scheduler execution explicit |
| 79 | + |
| 80 | +If the user wants local execution, produce local-friendly commands. |
| 81 | + |
| 82 | +If the user wants scheduler execution, produce scheduler-friendly commands and keep queue, partition, and resource requests explicit. |
| 83 | + |
| 84 | +Do not invent scheduler module names or executable paths. |
| 85 | + |
| 86 | +### 4. Do not invent environment activation commands |
| 87 | + |
| 88 | +If the user already has a working activation command such as: |
| 89 | + |
| 90 | +- `conda activate ...` |
| 91 | +- `module load ...` |
| 92 | +- `source ...` |
| 93 | + |
| 94 | +reuse it exactly. |
| 95 | + |
| 96 | +If execution is requested and the activation method is unknown, ask the user for the precise activation command. |
| 97 | + |
| 98 | +Do not guess conda environment names, module names, or site-specific paths. |
| 99 | + |
| 100 | +### 4.1 Outer launcher policy |
| 101 | + |
| 102 | +Use an activated DP-GEN environment and verify with: |
| 103 | + |
| 104 | +```bash |
| 105 | +dpgen --version |
| 106 | +``` |
| 107 | + |
| 108 | +Do not start simplify from a shell where `dpgen` is unavailable. |
| 109 | + |
| 110 | +### 4.2 Outer vs inner runtime boundaries (critical) |
| 111 | + |
| 112 | +Treat simplify execution as two separate environment layers: |
| 113 | + |
| 114 | +1. Outer layer: the shell that launches `dpgen simplify param.json machine.json` (must have `dpgen` in PATH) |
| 115 | +1. Inner layer: stage tasks dispatched by DP-GEN (`train` / `model_devi` / `fp`) on server/runtime side |
| 116 | + |
| 117 | +Even if the outer layer is correct, inner stage tasks still need explicit runtime setup in `machine.json`. |
| 118 | +Do not assume the outer shell environment will be inherited by dispatched stage jobs. |
| 119 | +For scheduler-style execution, `resources.source_list` must explicitly activate the required runtime environment. |
| 120 | + |
| 121 | +### 5. Prefer reproducible output layout |
| 122 | + |
| 123 | +When generating a simplify workflow, keep files organized and predictable. |
| 124 | + |
| 125 | +Recommended structure: |
| 126 | + |
| 127 | +```text |
| 128 | +project/ |
| 129 | +├── param.json |
| 130 | +├── machine.json |
| 131 | +├── run.sh |
| 132 | +├── logs/ |
| 133 | +└── summary/ |
| 134 | +``` |
| 135 | + |
| 136 | +For repeated experiments: |
| 137 | + |
| 138 | +```text |
| 139 | +project/ |
| 140 | +├── base/ |
| 141 | +├── exp_01/ |
| 142 | +├── exp_02/ |
| 143 | +├── exp_03/ |
| 144 | +└── summary/ |
| 145 | +``` |
| 146 | + |
| 147 | +## Minimum required inputs |
| 148 | + |
| 149 | +Collect the following information before generating files. |
| 150 | + |
| 151 | +### Dataset information |
| 152 | + |
| 153 | +- `pick_data` |
| 154 | +- `sys_configs` |
| 155 | +- `init_data_prefix` |
| 156 | +- `init_data_sys` |
| 157 | +- `sys_batch_size` |
| 158 | +- dataset format |
| 159 | +- `type_map` |
| 160 | +- `mass_map` if needed |
| 161 | +- `labeled` |
| 162 | + |
| 163 | +### Simplify controls |
| 164 | + |
| 165 | +- `init_pick_number` |
| 166 | +- `iter_pick_number` |
| 167 | +- `model_devi_f_trust_lo` |
| 168 | +- `model_devi_f_trust_hi` |
| 169 | +- `model_devi_e_trust_lo` / `model_devi_e_trust_hi` if energy trust is used |
| 170 | +- `numb_models` if not already specified |
| 171 | + |
| 172 | +### Training setup |
| 173 | + |
| 174 | +- `train_backend` if required by environment (for example `pytorch`) |
| 175 | +- `default_training_param` |
| 176 | + - descriptor settings |
| 177 | + - fitting network settings |
| 178 | + - learning rate settings |
| 179 | + - loss settings |
| 180 | + - training step settings |
| 181 | + |
| 182 | +### FP setup |
| 183 | + |
| 184 | +- `fp_style` |
| 185 | +- If data is already labeled (energy/force/virial available) and no re-labeling is requested, set `fp_style` to `none`. |
| 186 | +- if `fp_style != "none"`, collect matching FP runtime settings such as: |
| 187 | + - `fp_task_max` |
| 188 | + - `fp_task_min` |
| 189 | + - `fp_params` |
| 190 | + - pseudopotential or backend file paths if required |
| 191 | + |
| 192 | +### Execution setup |
| 193 | + |
| 194 | +For each stage `train`, `model_devi`, and `fp`, collect or preserve: |
| 195 | + |
| 196 | +- `command` |
| 197 | +- `machine.batch_type` |
| 198 | +- `machine.context_type` |
| 199 | +- `machine.local_root` |
| 200 | +- `machine.remote_root` |
| 201 | +- `resources.number_node` |
| 202 | +- `resources.cpu_per_node` |
| 203 | +- `resources.gpu_per_node` |
| 204 | +- `resources.group_size` |
| 205 | +- `resources.source_list` (required for scheduler jobs; use it to activate environment explicitly) |
| 206 | +- any explicit queue / partition / custom scheduler flags if the user already uses them |
| 207 | + |
| 208 | +Choose a runtime profile first, then fill the matching template: |
| 209 | + |
| 210 | +- server-local Slurm: `assets/machine.template.server-local-slurm.json` |
| 211 | +- local machine -> remote Slurm via SSH: `assets/machine.template.ssh-remote-slurm.json` |
| 212 | +- pure local shell testing: `assets/machine.template.local-shell.json` |
| 213 | + |
| 214 | +## How to build `param.json` |
| 215 | + |
| 216 | +Construct `param.json` around these logical blocks: |
| 217 | + |
| 218 | +1. element and mass definitions |
| 219 | +1. data source and batch settings |
| 220 | +1. model ensemble count |
| 221 | +1. default DeePMD training parameters |
| 222 | +1. FP backend settings |
| 223 | +1. simplify pick settings |
| 224 | +1. trust thresholds |
| 225 | + |
| 226 | +Key fields usually include: |
| 227 | + |
| 228 | +- `type_map` |
| 229 | +- `mass_map` |
| 230 | +- `pick_data` |
| 231 | +- `init_data_prefix` |
| 232 | +- `init_data_sys` |
| 233 | +- `sys_batch_size` |
| 234 | +- `numb_models` |
| 235 | +- `default_training_param` |
| 236 | +- `fp_style` |
| 237 | +- `shuffle_poscar` |
| 238 | +- `fp_task_max` |
| 239 | +- `fp_task_min` |
| 240 | +- `fp_pp_path` |
| 241 | +- `fp_pp_files` |
| 242 | +- `fp_params` |
| 243 | +- `init_pick_number` |
| 244 | +- `iter_pick_number` |
| 245 | +- `model_devi_f_trust_lo` |
| 246 | +- `model_devi_f_trust_hi` |
| 247 | + |
| 248 | +If the user is doing grid experiments, keep a base template and derive variants from it. |
| 249 | + |
| 250 | +Official reference example (QM7-style, adapted with path placeholders): |
| 251 | + |
| 252 | +- `assets/param.example.qm7.from-official-docs.json` |
| 253 | + |
| 254 | +## How to build `machine.json` |
| 255 | + |
| 256 | +Construct `machine.json` with separate stage blocks for: |
| 257 | + |
| 258 | +- `train` |
| 259 | +- `model_devi` |
| 260 | +- `fp` |
| 261 | + |
| 262 | +For each stage, keep the following explicit: |
| 263 | + |
| 264 | +- `command` |
| 265 | +- machine or context configuration |
| 266 | +- resources |
| 267 | +- queue or partition if needed |
| 268 | +- cpu and gpu counts |
| 269 | +- custom scheduler flags |
| 270 | +- environment activation commands |
| 271 | + |
| 272 | +Do not merge all stages into one vague machine block. |
| 273 | + |
| 274 | +## Validation before run |
| 275 | + |
| 276 | +Before execution, validate the workflow in this order: |
| 277 | + |
| 278 | +1. confirm outer-layer `dpgen` is available: |
| 279 | + |
| 280 | +```bash |
| 281 | +dpgen --version |
| 282 | +``` |
| 283 | + |
| 284 | +2. validate JSON syntax: |
| 285 | + |
| 286 | +```bash |
| 287 | +python -m json.tool param.json |
| 288 | +python -m json.tool machine.json |
| 289 | +``` |
| 290 | + |
| 291 | +3. verify required dataset paths exist |
| 292 | +1. verify stage commands match the selected software stack |
| 293 | +1. if `fp_style` is `none`, do not require FP-specific backend settings |
| 294 | +1. only then run: |
| 295 | + |
| 296 | +```bash |
| 297 | +dpgen simplify param.json machine.json |
| 298 | +``` |
| 299 | + |
| 300 | +## Output contract |
| 301 | + |
| 302 | +Always provide: |
| 303 | + |
| 304 | +1. final absolute paths to `param.json` and `machine.json` |
| 305 | +1. the exact simplify command to run (`dpgen simplify param.json machine.json`) |
| 306 | +1. a short pre-run checklist |
| 307 | +1. any unresolved required fields |
| 308 | +1. if execution was performed, the main output locations and next files to inspect |
| 309 | + |
| 310 | +## Guardrails |
| 311 | + |
| 312 | +- Never merge workflow and machine parameters into one file. |
| 313 | +- Never run `dpgen simplify` before both JSON files are present. |
| 314 | +- Never hardcode personal cluster, account, queue, or path settings as universal defaults. |
| 315 | +- Never silently change the user's scientific choices. |
| 316 | +- Keep `type_map` ordering consistent with dataset typing. |
| 317 | +- If required inputs are missing, stop and ask instead of guessing. |
| 318 | +- If `fp_style` is `none`, skip FP-specific prompts and keep FP-specific settings disabled or unset. |
| 319 | +- If data is already labeled and the user does not request new labels, enforce `fp_style = "none"` and do not require active FP runtime fields. |
| 320 | +- Do not assume outer-shell activation is inherited by stage jobs; for scheduler execution, require explicit `source_list` per stage. |
| 321 | +- If the user already has working templates, patch them rather than overwriting them blindly. |
| 322 | + |
| 323 | +## References and bundled files |
| 324 | + |
| 325 | +Use these bundled files: |
| 326 | + |
| 327 | +- `assets/param.template.json` |
| 328 | +- `assets/param.example.qm7.from-official-docs.json` |
| 329 | +- `assets/machine.template.json` |
| 330 | +- `assets/machine.template.server-local-slurm.json` |
| 331 | +- `assets/machine.template.ssh-remote-slurm.json` |
| 332 | +- `assets/machine.template.local-shell.json` |
| 333 | +- `references/param-fields.md` |
| 334 | +- `references/machine-fields.md` |
| 335 | +- `references/workflow-notes.md` |
| 336 | + |
| 337 | +External references: |
| 338 | + |
| 339 | +- DP-GEN simplify overview: https://docs.deepmodeling.com/projects/dpgen/en/latest/simplify/simplify.html |
| 340 | +- simplify parameter definitions: https://docs.deepmodeling.com/projects/dpgen/en/latest/simplify/simplify-jdata.html |
| 341 | +- simplify machine definitions: https://docs.deepmodeling.com/projects/dpgen/en/latest/simplify/simplify-mdata.html |
0 commit comments