Simplified namelists, common simulation classes, PORTALS refinements, MAESTRO enhancements, MEGPY and FIBE

This is a major refactor of MITIM to generalize the way optimization workflows, PORTALS and MAESTRO interact with the user (fully defined in YAML namelists now). The implementation of new simulation codes has now been made easier thanks to the creation of a parent simulation class, and CGYRO, TGLF, NEO and GX inherit from it. We introduced the MEGPY package as our equilibrium interpreter, and FIBE has been added as an option to initialize MAESTRO runs.

This release is highly incompatible with previous MITIM-fusion scripts due to all the refactoring. I promise it is worth upgrading your codes to interact with this new MITIM 4.0 version, much more robust and faster.

New Features

• 💥 Full YAML-based configuration for PORTALS, MAESTRO, optimization, and simulation models (including helpers to write YAML).
• 💥 Transport models in PORTALS now all interact via write/read JSON files with flux information. This simplifies the way to interact with externally-run codes.
• 💥 megpy as substitution of omfit_classes for geqdsk file readers, to simplify installation (omfit_classes required old numpy and scipy packages, breaking other compatibilities). @gsnoep @aaronkho
• 💥 MAESTRO runs can now be initialized using the fibe fixed boundary solver instead of freegs, for added control on boundary shape. @aaronkho @AudreySaltzman
• 💥 Added a new entry point: mitim_run_portals simple runner.
• 💥 EPED standalone capabilities, including squareness. @AudreySaltzman @aaronkho
• 💥 EPED initializer can now be different from the full EPED in MAESTRO. @AudreySaltzman
• 💥 NEO standalone capabilities.
• 💥 First automatic implementation of CGYRO and GX workflows, including their PORTALS interfaces. @xavior-wxc @nthoward
• 💥 Refined capability to write local input files to common simulation frameworks directly from MITIMstate.
• 💥 Added ability to send additional files (e.g., CGYRO/GX inputs) with runs, for restarts.
• 💥 Better loggers to keep track of machine resources and how the simulation was farmed out.
• 💥 code_settings now accepts labels (not numbers, although for back-compatibility the TGLF module can still receive numbers).
• 💥 [Experimental] Capability to reuse TGLF scan in subsequent PORTALS iterations that fall within the specified delta. In some cases, this speeds up convergence. @amsanabria

Bug Fixes

• 🐛 Corrected TGLF input handling: ensured zeff/z_eff column consistency, enforced dr/dx=1.0 to avoid a known TGLF bug, and corrected sign in to_tglf.
• 🐛 Fixed TGLF error bar calculation in PORTALS to avoid incorrect values.
• 🐛 TRANSP now enforces exclusive node requirement, avoiding shared-node failures. Removed unnecessary interactive warning in TRANSP exclusive-node check (no longer halts runs).
• 🐛 Fixed multiple Slurm/job farming issues: submitter launch paths, job array limit options in EPED tools.
• 🐛 PORTALS–CGYRO bug fix: prevented use of CGYRO results in prep mode. Fixed and defaulted to prep.
• 🐛 Fixed path quoting issues (folder names, double quotes, backslashes) across the code base. @jo-hall
• 🐛 Various MITIMstate corrections, including mass-weighted velocity profiles when lumping species. @gsnoep

Changes for developers (internal execution)

• 🔎 Generalized way that PORTALS interacts with transport models, to allow for multi-fidelity and passing the same options as standalone runs.
• 🔎 Transitioned namelists/configs to YAML, including what previously needed to happen in prep() for PORTALS. PORTALS settings are fully contained in namelist now.
• 🔎 Deprecated PORTALS functions removed; older TGYRO+PORTALS implementation dropped.
• 🔎 The type of targets in PORTALS is now represented as a list of strings.
• 🔎 Renamed tr_neo → tr_neoc for clarity and consistency with tr_turb.
• 🔎 Extracted MAESTRO–PORTALS post-processing logic into the namelist.
• 🔎 Improved tools for debugging PORTALS standalone runs and TGLF scans.
• 🔎 Improved namelist comparison by showing relative differences and in order.
• 🔎 Generalized simulation readers and added CGYRO as a first-class simulation object (via inheritance).
• 🔎 Avoided forcing installation of optional packages (e.g., onnx2pytorch).

Back-compatibility considerations and defaults

• 🔮 Migration to YAML configuration is required: all configs for PORTALS/MAESTRO/optimization now expect YAML, and exploration ranges are specified in YAML instead of via prep(). The name of some options has also changed, so be careful now.
• 🔮 Removal of INIT parameters for PORTALS. It was decided that the user must prepare the input.gacode in advance to use in PORTALS.
• 🔮 input.gacode is now always initialized with Deuterium reference mass (no longer uses “first ion”).
• 🔮 The TGYRO+PORTALS path has been removed as part of cleanup.
• 🔮 cgyroing_file capability dropped. Now it should be done via JSON files.

Bug fixes, new capabilities and MAESTRO for reactor design

This release provides bug fixes to PORTALS and MAESTRO and introduces new features, with a focus on MAESTRO for power plant design exercises.

Bug Fixes

Major:

• 🪲 Because zeff | - column in input.gacode must always exist before running TGYRO, MITIM now ensures its existence by deriving it with PROFILEStools.
• 🪲 Solved bug that wrote input.gacode with calculated targets always, regardless of whether PORTALS was run with, e.g., fixed radiation. @cjperks7
• 🪲 EPED temperature now uses the total ion density, not thermal, to translate from pressure to temperatures. Important when minority RF scenario has 3-5% "fast" minority even if thermalized at pedestal top.
• 🪲 Bug fix in ion particle flux scan (TGLFtools) using wrong indeces. @cjperks7

Minor:

• 🐛 Fixed bug that raised an exception if PORTALS converges at iteration #0 (possible in the last beats of a MAESTRO run).
• 🐛 Fixed plotting issues if PORTALS has avoided points during convergence.
• 🐛 Fixed bug that wouldn't let PORTALS reuse surrogates in MAESTRO.
• 🐛 Bug fix in maestro where neped was not passed trans-beats from EPED initializer.
• 🐛 Bug fix in PORTALS when restarting a case with different radii.
• 🐛 Several pathlib related bugs were fixed.
• 🐛 Several local singularity execution (TRANSP) fixes.
• 🐛 Several bug fixes related to job farming: number of cores, folder names and memory requirements.
• 🐛 Several minimal bug fixes in RAPIDS and POWERTORCH.
• 🐛 Corrected mass of impurities passed from TRANSP to profiles, particularly important for MAESTRO runs that have W.
• 🐛 Fixed backslash in f string error for python 3.11 @jo-hall

New Features

• 💥 Capability to use namelist (json) to initialize MAESTRO simulations, together with run_maestro script. @jo-hall
• 💥 Capability to specify Ti/Te at pedestal top for EPED beat in MAESTRO. @jo-hall
• 💥 Catching random exceptions in TGLF scans when running in PORTALS: automatic repetition if failure (e.g. cluster issues) and second attempt at paramiko's connection.
• 💥 Capability to insert profiles in r/a or psin coordinate for MAESTRO initialization, with warnings if more than one are provided. @kaitlyn-yanna
• 💥 Added default namelists for PORTALS, PORTALS soft, TRANSP and TRANSP soft in MAESTRO.
• 💥 Automatic MAESTRO folder retrieval from remove cluster, with capability to "fix" the pickle if it was run on a different operating system (but available also if local @AudreySaltzman).
• 💥 Allow PORTALS interpretation and plotting of folders without writing permissions.
• 💥 Added maximum number of codes a given code can use in parallel (specified in config_user.json as cores_per_node), useful for local runs.
• 💥 MAESTRO: Added betaN multiplier and capability to match 995 shaping values.
• 💥 MAESTRO: Fast species from TRANSP are brought back to plasma state if PORTALS removes them for transport calcualtions. @jo-hall
• 💥 TGLF scans: Added capability to scan more variables together.
• 💥 MAESTRO: Capability to check status of runs in terminal via alias.
• 💥 PORTALS: Added capability to only consider flux file in TGLF to reduce IO overhead. Remove temporary TGLF folder
• 💥 MITIM warnings are now printed more clearly, for easy grabbing from log files.
• 💥 Provide capability to enforce closest-Z impurity radiation if impurity not available in file (but not as default yet).

Changes for developers (internal execution)

• 🔎 Added new variables to powertorch needed for additional transport models, IMAS renormalization and gyrokinetic databases interaction @aaronkho
• 🔎 Re-routing all unpickling through custom one for error catcher
• 🔎 Beat finalization and merging repeated even if restart, to ensure that variables are properly "frozen" and brought back even if restart has happened. Removed MAESTRO finalization method.
• 🔎 TRANSP: If NUBEAM legacy corrector didn't work, continue without it
• 🔎 Cleaned up astra to gacode conversion tools and new astra naming convention. @jo-hall

Back-compatibility considerations and defaults

• 🔮 Stopping criterion in between optimizations in PORTALS reduced by default.
• 🔮 Back to using 5 coefficients to avoid TRANSP curvature issues in MAESTRO initializations. @Javier-Pimentel-Aldaz
• 🔮 Longer TRANSP beats by default.
• 🔮 Radiation file also contains atomic charge.

Paths! MAESTRO! PORTALS!

This release aggregates hundreds of commits that collectively refactor and update various components of MITIM-fusion. The changes span several functional areas, and chatGPT o3-mini-high was used to summarize the changes and provide the list below. This release would not have been possible without the work of the MFE-IM team., with particular mention to @AudreySaltzman, @aaronkho and @amsanabria.

New Features

• 💥 Full Pathlib Migration: Almost all filesystem operations now utilize pathlib and shutil. This update enhances cross-platform compatibility (including Windows) and simplifies operations like directory creation, file removal, and path globbing. @aaronkho @AudreySaltzman

• 💥 Enhanced Workflow Modules: Major improvements have been made to TGLF and TGYRO workflows. Test scripts now pass consistently, and additional functionality (such as an estimator for low-Z content in TRANSP) has been integrated, contributing to more robust end-to-end runs.

• 💥 MAESTRO Integration Enhancements: Merged MAESTRO-relevant updates allow for tighter integration with related simulation workflows. These changes include better handling of remote HPC retrieval and refined “beats” (e.g., for TRANSP, PORTALS, and EPED_NN), expanding the overall capability of MITIM.

• 💥 Improved HPC & Remote Execution: The PR introduces unique scratch folder generation for remote HPC runs, enhanced job array handling (defaulting to single-core to prevent system overload), and improved error handling with retry mechanisms during SSH/scp transfers.

• 💥 Optimizer & Algorithmic Upgrades: Restructured optimization namelists and streamlined optimizer options, resulting in faster execution of PORTALS runs and a more efficient overall simulation pipeline.

• 💥 Faster, more efficient PORTALS for TGLF. Introduction of best Ricci metric parameters and calculation of TGLF error bars based on variations in input parameters. @amsanabria

• 💥 New RAPIDS tool has been introduced, to study the effect of geometry, pedestal and dilution on reactor performance in stiff conditions.

• 💥 POWERTORCH solver has been refined to be used standalone to solve transport equations efficiently with surrogate models.

• 💥 Dynamic simple relaxation method has been introduced, which allows for more efficient flux-matching (steady-state) solutions when using fast surrogate models for transport fluxes.

Bug Fixes

• 🐛 Path Handling & Naming Issues: Fixed multiple issues where initialization files, configuration paths, and namelists were not read or updated properly. The comprehensive migration from os.path to pathlib.Path now prevents folder creation errors, broken symlinks, and incorrect expansion of environment variables.

• 🐛 Logging & Remote Execution Anomalies: Addressed bugs such as logger routines not closing correctly and fixed problematic nested shell calls (e.g., in FARMINGtools.py) that were affecting remote job submissions via SSH/scp. These fixes help prevent random connection drops and ensure smoother remote operations.

• 🐛 HPC & Workflow Corner Cases: Corrected edge-case issues in HPC workflows—resolving failures in job arrays (such as unexpected behavior in TGLF calls) and fixing boundary-related errors (e.g., negative q values in input files and issues with low-Z estimators in TRANSP). These changes ensure that all computational modules behave reliably in high-performance environments.

• 🐛 Inconsistent Naming & Module References: Removed outdated references (from PRF to MITIM) and standardized naming conventions across the codebase. This unification reduces confusion between local and remote execution contexts and streamlines the user experience.

• 🐛 Namelist Writing & Environment Handling: Resolved subtle issues where configuration files (namelists) were not being correctly overwritten or updated—especially during remote executions and job array submissions—ensuring that environment variables are properly propagated to all modules.

Changes for developers (internal execution)

• 🔎 Refactoring & Modernization: Replaced legacy os.system calls with higher-level Python functions (like shutil.copy, Path.unlink, and Path.replace). This refactoring reduces OS-specific edge cases and makes the codebase more maintainable.

• 🔎 Consolidated HPC Logic: Centralized HPC-related checks, environment variable expansions, and Slurm job submission routines into shared methods. This standardization minimizes redundancy and simplifies the configuration for remote and local runs.

• 🔎 Cleaner Module Layout: Removed deprecated SSH-based code and outdated references (such as old ASTRA workflows), cleaning up the module structure. This streamlining improves readability and paves the way for future development.

• 🔎 Enhanced Logging & Diagnostics: Overhauled internal logging and timing functions (notably in IOtools.py) to provide better runtime diagnostics and facilitate easier debugging during development and production runs.

• 🔎 Consistent Naming & Imports: Continued the refactor of naming conventions (transitioning from PRF to MITIM) and removed extraneous imports, leading to a more consistent and clutter-free codebase.

Back-compatibility considerations

• 🔮 Optimization JSON Format Update: The JSON file for MITIM optimization has been completely refactored. Users with custom scripts relying on the old format should update their configurations accordingly.

• 🔮 Transition to Pathlib: Scripts that depend on legacy os.path calls within MITIM modules may need to be revised. Despite this change, the external API remains largely unchanged.

• 🔮 Adjusted HPC Defaults: TGLF job arrays now default to single-core executions to avoid overloading systems. Users requiring multi-core operations must explicitly configure cores in their Slurm settings or adjust the namelist options.

• 🔮 Stricter Environment Variable Resolution: The new environment handling approach now enforces stricter resolution of paths and variable expansions. This might expose misconfigurations in legacy setups—updating or removing references to old configuration files (such as “config_user.json”) is recommended.

• 🔮 Packages requirements lead to the need of python 3.10, 3.11 or 3.12, as <=3.9 is not compatible with the latest botorch requirements and >=3.13 is not compatible with tensorflow or omfit_classes

Full Changelog: v2.1.0...v3.0.0

MAESTRO, TRANSP and flux surfaces

Bug Fixes

• 🐛 Relaxed requirements of pip packages to avoid installation issues.
• 🐛 Corrupted TRANSP CDF files can now be read by removing bad time slices.
• 🐛 Improved interaction with TRANSP singularity to avoid trlook and trfinish issues, and to be able to run locally.
• 🐛 Improved logic for TRANSP cases that failed. Errors are now properly reported and process stopped.
• 🐛 areal elongation is now properly calculated in PROFILEStools.py, fixing H98 calculations. @josephb-hall

New Features

• 💥 Flux surfaces now handled always using their own class mitim_flux_surfaces. Parameterization, including Miller Extended Harmonics (MXH) now happens as a class method. @josephb-hall
• 💥 Capability to easily run "fixed-boundary" millerized equilibrium using the FreeGS code. The freegs pip package is now part of the requirements.
• 💥 Better check for missing or wrong flags in TGLF, TGYRO and PORTALS.
• 💥 Various conversion routines: profiles to transp, transp to profiles, freegs to transp, geqdsk to profiles, etc. All part of their own objects.
• 💥 TRANSP Helper object can now interact with namelist and prepare UFILES.
• 💥 Start of the common Neural Network surrogate class, and implementation of EPED NN. @josephb-hall
• 💥 First release of MAESTRO (Modular and Accelerated Engine for Simulation of Transport and Reactor Optimization) workflow. Implementation of TRANSP, PORTALS and EPED_NN beats. Ready for stress-testing. @AudreySaltzman @josephb-hall.
• 💥 Surrogates in PORTALS can now receive any other variable that are not included in gradients or profiles, such as q and shear. This will be useful when combining cases from different plasmas or scenarios.
• 💥 TRANSP reader mitim_plot_transp can now work with folder as input, and it will find the CDF
• 💥 if omfit_classes geqdsk reader cannot find the separatrix, now MITIM will increase resolution and go find it again. Seems to be working in some extreme cases.
• 💥 TRANSP runs do not need to use MRY anymore, now we can build RFS and ZFS files, which removes the need for scruncher.
• 💥 Further improvements to ASTRA class. @josephb-hall @AudreySaltzman @mmuraca
• 💥 TGLF output class has now a to_xarray() converter now. @aaronkho

Changes for developers (internal execution)

• 🔎 New contexts: timers and loggers. All part of IOtools.py
• 🔎 Removed deprecated functions and freegsu references to SPARC. Code moved to new locations.
• 🔎 Old "im" module moved out of MITIM-fusion. Too many changes to keep it up to date.
• 🔎 Naming changes to physicsBasedParams in PORTALS.

Back-compatibility considerations

• 🔮 It has been decided that PORTALS-TGLF runs by default use only the botorch optimizers, instead of root and ga that was used before. Only in expensive situations (e.g. CGYRO), one should try to refine solutions. This makes the convergence faster in wall-time.
• 🔮 config_user.json is now removed from the package folder. A template is provided in templates but the user can choose to place their config_user.json anywhere. Follow the read-the-docs for instructions: https://mitim-fusion.readthedocs.io/en/latest/installation.html#user-configuration @aaronkho @josephb-hall
• 🔮 machine-specific bashrc "recommendations" are now private and not part of the MITIM-fusion repository.

Improvements in efficiency, structure and physics

Bug Fixes

• 🐛 Optimization stopping criteria changed to follow maximization logic, not residual minimization.
• 🐛 Fixed bug that wrote surrogate data file with wrong nuei.
• 🐛 Removed references to pablorf folders.
• 🐛 Reading and plotting of geqdsk now more robust, fixed bug in removeCoils function.
• 🐛 Fixed bug that made waveform calculation in TGLF unstable.
• 🐛 PROFILES variable $\epsilon$ (inverse aspect ratio) now correctly defined as $r/R$.
• 🐛 Corrected betae for 100 times lower (before, it was a percentage)
• 🐛 Keep botorch==0.9.4 until fix is found.

New Features

• 💥 First release of basic ASTRA class (@AudreySaltzman, @mmuraca).
• 💥 Equilibrium and flux surfaces to MXH coefficients conversion routine available (@josephb-hall).
• 💥 Users will now be warned if PORTALS, TGLF or BO option they selected is not available.
• 💥 POWERSTATE class now receives custom methods for transport and target calculations, in view of SMARTSsolver future development (@cholland). These can be provided externally as long as they inherit from parent classes.
• 💥 POWERSTATE flux-matching class now works fine outside of surrogate-based optimization methods, for quick evaluation of steady-state profiles with analytical or surrogated transport coefficients. Future development will include direct TGLF call, resembling a pythonic TGYRO solver.
• 💥 BO and PORTALS results now written as csv files and handled as dataframes.
• 💥 Much faster POWERSTATE class (changed logic for detachment and unsqueezing of tensors) and much faster derivedQuantities calculation of PROFILES via parallelization of geometric variables calculations.
• 💥 Zeff in PROFILES can now be modified directly without the need to handle manually the individual ion densities.
• 💥 Maximization objective now added as part of the optimization_data results file.
• 💥 Direct PROFILES to TGLF in-memory converter ready to go (prep_direct_tglf() method). No need for "iteration=0" of TGYRO. In next release, this will be default.
• 💥 POPCON matching tools available (@josephb-hall).

Changes for developers (internal execution)

• 🔎 Significant renaming of internal methods, functions and classes to follow better standards.
• 🔎 Only POWERSTATE class is needed for PORTALS plotting and analysis, which eases implementation with non-TGYRO transport methods.
• 🔎 Generalized update_var function towards a channel-agnostic framework
• 🔎 Generally, better organization of: parameterization of POWERSTATE, targets calculations (including fine targets). Removal of deprecated options and methods.
• 🔎 Radiation coefficients for ADAS radiation now are part of external csv file instead of hardcoded in python.

Back-compatibility considerations

• 🔮 Changed name of options variables and dictionaries in TGYRO, PORTALS and BO.
• 🔮 Fine targets resolution in PORTALS is now enabled by default.
• 🔮 Changed TGLF settings default in solvers.
• 🔮 Changed PORTALS radial resolution by default.
• 🔮 POWERSTATE now must receive arrays without rho=0.0, which is added by default.
• 🔮 Folders aux and exe renamed to utils and scripts to avoid issues with Windows machines and github cloning.

Misc bug fixes and faster TGLF scans

Bug Fixes

• 🐛 High-resolution fluxes in standard PORTALS plotting now have the correct coordinate system instead of defaulting to rho(-).
• 🐛 Plotting of PORTALS initialization stage now shows next profile to evaluate.
• 🐛 Reading of geqdsk files does not crash that often: robust to empty lines and option for several time slices in same file.
• 🐛 Reading of input.gacode files more robust: allows time row and non scientific notation.
• 🐛 User is now warned if PORTALS is run with an ion specie not included in radiation calculations.
• 🐛 PORTALS does not crash if surrogate specified with no transformation.
• 🐛 Fixed routines for singularity execution of TRANSP.
• 🐛 removed scipy modules causing problems for certain numpy version.

New Features

• 💥 Generalization of TGLF scans class to allow for parallel execution of scans (including for analysis purposes and waveform generation).
• 💥 CGYRO class development has commenced! Linear runs possible, but still in development phase. Use at your own risk.
• 💥 CSV tables easily created from PROFILES class for quick communication of global parameter results (e.g. PORTALS predictions).

Changes for developers (internal execution)

• Changed use of (most) lambda functions in PORTALS to "def" functions.
• Generalization of handler of GACODE settings (TGLF, CGYRO, TGYRO).
• Capability to tell the execution farming to check files inside output folder when coming back from remote machine.

Refactored remote execution handler

Bug Fixes

• 🐛 Particle flux in GB units was multiplied by a factor of 3/2 when written into data tables and extracted surrogates. Now, that factor is removed. This does not change results of past profile predictions (output data is standardized anyway) but it changes the interpretation of output tables for the particle flux.
• 🐛 Fixed situation that made MITIM crash when added data points (from previous runs) to new surrogates have different input dimensions.
• 🐛 Input features are made equal when the changes are very close to machine precision, to solve problem with dimensions that should not be fitted (issue appeared only when PORTALS restarting from previous surrogates, from a different input.gacode).
• 🐛 Fixed issue with zero particle flux with powerstate targets when plotting PORTALS.
• 🐛 Solved issue that caused MITIM-TGYRO to crash if anchor point was chosen past rho>0.9, and catch case where numerical scheme cannot run with zero iterations.
• 🐛 Path canonicalization solved issues (encountered in some clusters) solved.

New Features

• 💥 More robust handling of external command execution, using paramiko and tqdm packages. Transferred files are now first included in a tarball to send and receive, and they do not stop at tunnel anymore.
• 💥 Slurm setup (account, partition, constraints, memory requirements) now are part of JSON. Code classes may change memory requirements, but a default can now be added.
• 💥 TGLF settings (defaults and TGLFsettings options) are now handled as part of JSON and control files in templates/. Added SAT3 as option 6.
• 💥 PORTALS plotter can now combine several runs in a single notebook.
• 💥 TRANSP plotter can now combine several runs in a single notebook, and plotting occurs automatically from alias mitim_plot_transp.
• 💥 General improvements to documentation, including new config file specifications, FAQ and clear instructions about the role of machine's bashrc files.
• 💥 Capability to extract PORTALS surrogates in an elegant way, together with the possibility to run standalone TGLF and TGYRO, extracted from specific evaluations of PORTALS during the BO workflow.
• 💥 Fluxes produced during a PORTALS-CGYRO run can now be easily (and automatically in PORTALS plotting) compared to TGLF.
• 💥 Notebooks improvements: No need to block terminal, more robust to segmentation faults, and added colors to tabs.
• 💥 User config file can now have argument of "modules" to send to each machine, to execute before any code.

Changes for developers (internal execution)

• New NEO class handles VGEN runs for calculation of Er and rotation from input.gacode files.
• PORTALS plotting separated from analysis, and much cleaner interface with general plotting routines.
• Default physics-based params are now part of the class, to retrieve again if eliminated before restarting.
• Start of the CGYRO class and NERSC perlmutter connection (in the works).

First complete version of MITIM-fusion, Hello World

This release signifies the first version of the MITIM-fusion repository to be released publicly. It is ready for prime time!