Profile data dumping (aiidateam#6723)

Data dumping for groups and profiles (aiidateam#6723)

This PR adds functionality to incrementally dump profile and group data
into a human-readable output folder, and refactors the internal logic of
the recently released process dumping.

### Public API

The data dumping feature can be used from the CLI via the 
`verdi {profile|group|process} dump` commands. Furthermore, 
the classes `aiida.manage.configuration.Profile`, `aiida.orm.Group` and
`aiida.orm.ProcessNode` are extended by a new public member function
`dump` that takes the same dumping options as the CLI entry points.

The internal implementation of the feature is contained in the private
module `aiida.tools._dumping`, which is currently excluded from
`codecov`. Further testing and modifications will be applied based on
user feedback in smaller, more manageable PRs.

### Configuration of dumping

To organize the extensive options, a data class for the config options is
created using `pydantic` `BaseModel` in the `aiida.tools._dumping.config`
module. For each type of dumping (process/group/profile) different
options are available contained in three config classes
`ProcessDumpConfig`, `ProfileDumpConfig`, `GroupDumpConfig` all
inheriting from `BaseDumpConfig`. The `*DumpConfig` classes use the
mixin pattern to organize different options via the `TimeFilterMixin`,
`EntityFilterMixin`, `ProcessHandlingMixin` and `GroupManagementMixin`
since they are not available for each type of entity being dumped. The
new CLI entry points and the new member function `dump` all map their
inputs to the respective config class. By mapping both inputs to the
`*DumpConfig` classes the validation process is unified, reducing code
duplication.

### State of dumping folder

The dumping functionality tracks the state of the dumped folder wrt. to
the AiiDA database. This requires a persistent storage of the current
state of the dumping folder as well as logic comparing the state of the
dumping folder and the database. To prevent expensive file reads of the
dumping folder, the state is stored in a `json` file after the dumping
process. The logic to evaluate and track the state and compare it with
the database is contained in the modules `aiida.tools._dumping.tracking`
and `aiida.tools._dumping.mapping`, and changes since the last dump
(new/deleted nodes/groups, relabeled groups, node membership changes,
etc.) are then picked up via the module `aiida.tools._dumping.detector`.
This enables incremental dumping of data in a way that the
human-readable output folder of the dumping feature tracks the state of
the AiiDA DB as it evolves.

### Execution of the dump

The `aiida.tools._dumping.engine` module is responsible for the
top-level orchestration of the dumping process (including reading in the
`json` state file from the previous dump, or deleting it, if overwrite
mode is selected), as well as common setup and teardown operations. For
group and profile dumping, changes in AiiDA's DB since the last dump
that are not yet reflected in the dumping output folder, are then
carried out incrementally. This includes deleting output directories of
nodes and groups that were previously dumped but were since deleted from
AiiDA's DB, applying group relabeling carried out by the user, as well
as dumping new nodes and groups. This functionality is contained in the
`aiida.tools._dumping.executors` that provides the `DeletionExecutor`,
`ProcessDumpExecutor`, `GroupDumpExecutor`, and `ProfileDumpExecutor`,
classes and presents the meat of the feature implementation.

Finally, code that was previously contained in the `ProcessDumper` class
is now moved to the `ProcessDumpExecutor`, while a `ProcessDumper`
"facade" class is still provided via the `aiida.tools._dumping.facades`
module, and exposed as public API for backwards compatibility.

Author: GeigerJ2

codecov.yml

@@ -15,3 +15,6 @@ coverage:
     patch:
       default:
         threshold: 0.1%
+
+ignore:
+- src/aiida/tools/_dumping/**/*

docs/source/howto/data.rst

@@ -223,6 +223,7 @@ Below is a list with all available subcommands.
       create        Create an empty group with a given label.
       delete        Delete groups and (optionally) the nodes they contain.
       description   Change the description of a group.
+      dump          Dump data of an AiiDA group to disk.
       list          Show a list of existing groups.
       move-nodes    Move the specified NODES from one group to another.
       path          Inspect groups of nodes, with delimited label paths.
@@ -397,6 +398,7 @@ Below is a list with all available subcommands.
     Commands:
       configure-rabbitmq  Configure RabbitMQ for a profile.
       delete              Delete one or more profiles.
+      dump                Dump all data in an AiiDA profile's storage to disk.
       list                Display a list of all available profiles.
       set-default         Set a profile as the default profile.
       setdefault          (Deprecated) Set a profile as the default profile.

docs/source/reference/command_line.rst

@@ -279,6 +279,11 @@ Documentation = 'https://aiida.readthedocs.io'
 Home = 'http://www.aiida.net/'
 Source = 'https://github.com/aiidateam/aiida-core'
 
+[tool.coverage.run]
+omit = [
+  "src/aiida/tools/_dumping/**/*"
+]
+
 [tool.flit.module]
 name = 'aiida'
 

pyproject.toml

@@ -137,7 +137,7 @@ def group_move_nodes(source_group, target_group, force, nodes, all_entries):
 
     if not force:
         click.confirm(
-            f'Are you sure you want to move {len(nodes)} nodes from {source_group} ' f'to {target_group}?', abort=True
+            f'Are you sure you want to move {len(nodes)} nodes from {source_group} to {target_group}?', abort=True
         )
 
     source_group.remove_nodes(nodes)
@@ -325,6 +325,11 @@ def group_relabel(group, label):
         echo.echo_critical(str(exception))
     else:
         echo.echo_success(f"Label changed to '{label}'")
+        msg = (
+            'Note that if you are dumping your profile data to disk, to reflect the relabeling of the group, '
+            'run your `verdi profile dump` command again.'
+        )
+        echo.echo_report(msg)
 
 
 @verdi_group.command('description')
@@ -632,3 +637,106 @@ def group_path_ls(path, type_string, recursive, as_table, no_virtual, with_descr
             if no_virtual and child.is_virtual:
                 continue
             echo.echo(child.path, bold=not child.is_virtual)
+
+
+@verdi_group.command('dump')
+@arguments.GROUP()
+@options.PATH()
+@options.DRY_RUN()
+@options.OVERWRITE()
+@options.PAST_DAYS()
+@options.START_DATE()
+@options.END_DATE()
+@options.FILTER_BY_LAST_DUMP_TIME()
+@options.ONLY_TOP_LEVEL_CALCS()
+@options.ONLY_TOP_LEVEL_WORKFLOWS()
+@options.DELETE_MISSING()
+@options.SYMLINK_CALCS()
+@options.INCLUDE_INPUTS()
+@options.INCLUDE_OUTPUTS()
+@options.INCLUDE_ATTRIBUTES()
+@options.INCLUDE_EXTRAS()
+@options.FLAT()
+@options.DUMP_UNSEALED()
+@click.pass_context
+@with_dbenv()
+def group_dump(
+    ctx,
+    group,
+    path,
+    dry_run,
+    overwrite,
+    past_days,
+    start_date,
+    end_date,
+    filter_by_last_dump_time,
+    delete_missing,
+    only_top_level_calcs,
+    only_top_level_workflows,
+    symlink_calcs,
+    include_inputs,
+    include_outputs,
+    include_attributes,
+    include_extras,
+    flat,
+    dump_unsealed,
+):
+    """Dump data of an AiiDA group to disk."""
+
+    import traceback
+    from pathlib import Path
+
+    from aiida.cmdline.utils import echo
+    from aiida.tools._dumping.utils import DumpPaths
+
+    warning_msg = (
+        'This is a new feature which is still in its testing phase. '
+        'If you encounter unexpected behavior or bugs, please report them via Discourse or GitHub.'
+    )
+    echo.echo_warning(warning_msg)
+
+    try:
+        if path is None:
+            group_path = DumpPaths.get_default_dump_path(group)
+            dump_base_output_path = Path.cwd() / group_path
+            echo.echo_report(f'No output path specified. Using default: `{dump_base_output_path}`')
+        else:
+            dump_base_output_path = Path(path).resolve()
+            echo.echo_report(f'Using specified output path: `{dump_base_output_path}`')
+
+        # --- Logical Checks ---
+        if dry_run and overwrite:
+            msg = (
+                '`--dry-run` and `--overwrite` selected (or set in config). Overwrite operation will NOT be performed.'
+            )
+            echo.echo_warning(msg)
+
+        # Run the dumping
+        group.dump(
+            output_path=dump_base_output_path,
+            dry_run=dry_run,
+            overwrite=overwrite,
+            past_days=past_days,
+            start_date=start_date,
+            end_date=end_date,
+            filter_by_last_dump_time=filter_by_last_dump_time,
+            only_top_level_calcs=only_top_level_calcs,
+            only_top_level_workflows=only_top_level_workflows,
+            symlink_calcs=symlink_calcs,
+            include_inputs=include_inputs,
+            include_outputs=include_outputs,
+            include_attributes=include_attributes,
+            include_extras=include_extras,
+            flat=flat,
+            dump_unsealed=dump_unsealed,
+        )
+
+        if not dry_run:
+            msg = f'Raw files for group `{group.label}` dumped into folder `{dump_base_output_path.name}`.'
+            echo.echo_success(msg)
+        else:
+            echo.echo_success('Dry run completed.')
+
+    except Exception as e:
+        msg = f'Unexpected error during dump of group {group.label}:\n ({e!s}).\n'
+        echo.echo_critical(msg + traceback.format_exc())

src/aiida/cmdline/commands/cmd_group.py

@@ -14,6 +14,7 @@
 from aiida.cmdline.params import arguments, options, types
 from aiida.cmdline.params.options.overridable import OverridableOption
 from aiida.cmdline.utils import decorators, echo
+from aiida.cmdline.utils.decorators import with_dbenv
 from aiida.common.log import LOG_LEVELS, capture_logging
 
 REPAIR_INSTRUCTIONS = """\
@@ -583,98 +584,94 @@ def process_repair(manager, broker, dry_run):
 @verdi_process.command('dump')
 @arguments.PROCESS()
 @options.PATH()
+@options.DRY_RUN()
 @options.OVERWRITE()
-@click.option(
-    '--include-inputs/--exclude-inputs',
-    default=True,
-    show_default=True,
-    help='Include the linked input nodes of the `CalculationNode`(s).',
-)
-@click.option(
-    '--include-outputs/--exclude-outputs',
-    default=False,
-    show_default=True,
-    help='Include the linked output nodes of the `CalculationNode`(s).',
-)
-@click.option(
-    '--include-attributes/--exclude-attributes',
-    default=True,
-    show_default=True,
-    help='Include attributes in the `.aiida_node_metadata.yaml` written for every `ProcessNode`.',
-)
-@click.option(
-    '--include-extras/--exclude-extras',
-    default=True,
-    show_default=True,
-    help='Include extras in the `.aiida_node_metadata.yaml` written for every `ProcessNode`.',
-)
-@click.option(
-    '-f',
-    '--flat',
-    is_flag=True,
-    default=False,
-    show_default=True,
-    help='Dump files in a flat directory for every step of the workflow.',
-)
-@click.option(
-    '--dump-unsealed',
-    is_flag=True,
-    default=False,
-    show_default=True,
-    help='Also allow the dumping of unsealed process nodes.',
-)
-@options.INCREMENTAL()
+@options.INCLUDE_INPUTS()
+@options.INCLUDE_OUTPUTS()
+@options.INCLUDE_ATTRIBUTES()
+@options.INCLUDE_EXTRAS()
+@options.FLAT()
+@options.DUMP_UNSEALED()
+@click.pass_context
+@with_dbenv()
 def process_dump(
+    ctx,
     process,
     path,
+    dry_run,
     overwrite,
     include_inputs,
     include_outputs,
     include_attributes,
     include_extras,
     flat,
     dump_unsealed,
-    incremental,
 ) -> None:
     """Dump process input and output files to disk.
 
     Child calculations/workflows (also called `CalcJob`s/`CalcFunction`s and `WorkChain`s/`WorkFunction`s in AiiDA
     jargon) run by the parent workflow are contained in the directory tree as sub-folders and are sorted by their
-    creation time.  The directory tree thus mirrors the logical execution of the workflow, which can also be queried by
+    creation time. The directory tree thus mirrors the logical execution of the workflow, which can also be queried by
     running `verdi process status <pk>` on the command line.
 
     By default, input and output files of each calculation can be found in the corresponding "inputs" and
     "outputs" directories (the former also contains the hidden ".aiida" folder with machine-readable job execution
     settings). Additional input and output files (depending on the type of calculation) are placed in the "node_inputs"
     and "node_outputs", respectively.
 
-    Lastly, every folder also contains a hidden, human-readable `.aiida_node_metadata.yaml` file with the relevant AiiDA
+    Lastly, every folder also contains a hidden, human-readable `aiida_node_metadata.yaml` file with the relevant AiiDA
     node data for further inspection.
     """
+    import traceback
+    from pathlib import Path
 
+    from aiida.cmdline.utils import echo
+    from aiida.tools._dumping.utils import DumpPaths
     from aiida.tools.archive.exceptions import ExportValidationError
-    from aiida.tools.dumping.processes import ProcessDumper
-
-    process_dumper = ProcessDumper(
-        include_inputs=include_inputs,
-        include_outputs=include_outputs,
-        include_attributes=include_attributes,
-        include_extras=include_extras,
-        overwrite=overwrite,
-        flat=flat,
-        dump_unsealed=dump_unsealed,
-        incremental=incremental,
+
+    warning_msg = (
+        'This is a new feature which is still in its testing phase. '
+        'If you encounter unexpected behavior or bugs, please report them via Discourse or GitHub.'
     )
+    echo.echo_warning(warning_msg)
 
+    # Check for dry_run + overwrite
+    if overwrite and dry_run:
+        msg = 'Both `dry_run` and `overwrite` set to true. Operation will NOT be performed.'
+        echo.echo_warning(msg)
+        return
+
+    if path is None:
+        process_path = DumpPaths.get_default_dump_path(process)
+        dump_base_output_path = Path.cwd() / process_path
+        msg = f'No output path specified. Using default: `{dump_base_output_path}`'
+        echo.echo_report(msg)
+    else:
+        echo.echo_report(f'Using specified output path: `{path}`')
+        dump_base_output_path = Path(path).resolve()
+
+    if dry_run:
+        echo.echo_success('Dry run completed.')
+        return
+
+    # Execute dumping
     try:
-        dump_path = process_dumper.dump(process_node=process, output_path=path)
-    except FileExistsError:
-        echo.echo_critical(
-            'Dumping directory exists and overwrite is False. Set overwrite to True, or delete directory manually.'
+        process.dump(
+            output_path=dump_base_output_path,
+            dry_run=dry_run,
+            overwrite=overwrite,
+            include_inputs=include_inputs,
+            include_outputs=include_outputs,
+            include_attributes=include_attributes,
+            include_extras=include_extras,
+            flat=flat,
+            dump_unsealed=dump_unsealed,
         )
+
+        msg = f'Raw files for process `{process.pk}` dumped into folder `{dump_base_output_path.name}`.'
+        echo.echo_success(msg)
     except ExportValidationError as e:
-        echo.echo_critical(f'{e!s}')
+        echo.echo_critical(f'Data validation error during dump: {e!s}')
     except Exception as e:
-        echo.echo_critical(f'Unexpected error while dumping {process.__class__.__name__} <{process.pk}>:\n ({e!s}).')
-
-    echo.echo_success(f'Raw files for {process.__class__.__name__} <{process.pk}> dumped into folder `{dump_path}`.')
+        msg = f'Unexpected error during dump of process {process.pk}:\n ({e!s}).\n'
+        echo.echo_critical(msg + traceback.format_exc())