Added serialisation to docs

Author: tclose

docs/source/index.rst

@@ -85,7 +85,7 @@ workflows,
 * :ref:`Python-tasks`
 * :ref:`Shell-tasks`
 * :ref:`Workflows`
-* :ref:`Canonical task form`
+* :ref:`Canonical form and serialisation`
 
 Examples
 ~~~~~~~~

docs/source/tutorial/7-canonical-form.ipynb

@@ -4,7 +4,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Canonical task form\n",
+    "# Canonical form and serialisation\n",
+    "\n",
+    "## Canonical task form\n",
     "\n",
     "Under the hood, all Python, shell and workflow tasks generated by the\n",
     "`pydra.compose.*.define` decorators/functions are translated to\n",
@@ -149,33 +151,32 @@
     "from pathlib import Path\n",
     "from fileformats import generic\n",
     "from pydra.compose import shell\n",
-    "from pydra.utils.typing import MultiInputObj\n",
+    "\n",
+    "\n",
+    "# the \"copied\" output is magically passed to this function because the name matches\n",
+    "def get_file_size(copied: Path) -> int:\n",
+    "    \"\"\"Calculate the file size\"\"\"\n",
+    "    result = os.stat(copied)\n",
+    "    return result.st_size\n",
     "\n",
     "\n",
     "@shell.define\n",
-    "class CpWithSize(shell.Task[\"CpWithSize.Outputs\"]):\n",
+    "class CpFileWithSize(shell.Task[\"CpFileWithSize.Outputs\"]):\n",
     "\n",
     "    executable = \"cp\"\n",
     "\n",
-    "    in_fs_objects: MultiInputObj[generic.FsObject]\n",
-    "    recursive: bool = shell.arg(argstr=\"-R\")\n",
-    "    text_arg: str = shell.arg(argstr=\"--text-arg\")\n",
-    "    int_arg: int | None = shell.arg(argstr=\"--int-arg\")\n",
-    "    tuple_arg: tuple[int, str] | None = shell.arg(argstr=\"--tuple-arg\")\n",
+    "    in_file: generic.File  # = shell.arg() is assumed\n",
+    "    archive_mode: bool = shell.arg(argstr=\"-a\", default=False)\n",
     "\n",
     "    class Outputs(shell.Outputs):\n",
     "\n",
-    "        @staticmethod\n",
-    "        def get_file_size(out_file: Path) -> int:\n",
-    "            \"\"\"Calculate the file size\"\"\"\n",
-    "            result = os.stat(out_file)\n",
-    "            return result.st_size\n",
+    "        copied: generic.File = shell.outarg(\n",
+    "            position=-1, path_template=\"{in_file}_copied\"\n",
+    "        )\n",
+    "        file_size: int = shell.out(callable=get_file_size)\n",
     "\n",
-    "        copied: generic.FsObject = shell.outarg(path_template=\"copied\")\n",
-    "        out_file_size: int = shell.out(callable=get_file_size)\n",
     "\n",
-    "\n",
-    "print_help(CpWithSize)"
+    "print_help(CpFileWithSize)"
    ]
   },
   {
@@ -197,7 +198,6 @@
     "import typing as ty\n",
     "import re\n",
     "from pydra.compose import python, workflow\n",
-    "from pydra.compose.base import is_set\n",
     "from pydra.utils import print_help, show_workflow\n",
     "\n",
     "\n",
@@ -237,6 +237,121 @@
     "print_help(CanonicalWorkflowTask)\n",
     "show_workflow(CanonicalWorkflowTask)"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Serialisation\n",
+    "\n",
+    "As well as the dataclass-like canonical form, it is also possible to represent all tasks\n",
+    "in a nested dictionary form, which could be written to a static file (e.g. in JSON or\n",
+    "YAML format). The dictionary form of a class can be generated by the `pydra.utils.task_def_as_dict`\n",
+    "function. For example, the following shell command"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "MyCmd = shell.define(\n",
+    "    \"my-cmd <in_file> <out|out_file> --an-arg <an_arg?> \"\n",
+    "    \"--a-flag<a_flag> --arg-with-default <arg_with_default:int=3>\"\n",
+    ")\n",
+    "\n",
+    "print_help(MyCmd)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Can be converted into a serialised dictionary form"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pprint import pprint\n",
+    "from pydra.utils import task_def_as_dict\n",
+    "\n",
+    "my_cmd_dict = task_def_as_dict(MyCmd)\n",
+    "\n",
+    "pprint(my_cmd_dict)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Noting that there is still a little more work has to be done to serialise some Python\n",
+    "objects, such as the `type` field and `xor` before it could be written to JSON/YAML. \n",
+    "This is more challenging for task definitions containing Python functions such as Python\n",
+    "and Workflow tasks or Shell tasks with converters, validators or callables, e.g."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cp_with_size_dict = task_def_as_dict(CpFileWithSize)\n",
+    "\n",
+    "pprint(cp_with_size_dict)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To unserialize the general dictionary form back into a Task class, you can use the\n",
+    "`pydra.utils.task_def_from_dict` method"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pydra.utils import task_def_from_dict\n",
+    "\n",
+    "ReloadedCpFileWithSize = task_def_from_dict(cp_with_size_dict)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "which should run just as before"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import tempfile\n",
+    "from pydra.utils import task_as_dict\n",
+    "\n",
+    "tmp_dir = Path(tempfile.mkdtemp())\n",
+    "\n",
+    "a_file = tmp_dir / \"hello-world.txt\"\n",
+    "a_file.write_text(\"Hello world\")\n",
+    "\n",
+    "cp_file_with_size = ReloadedCpFileWithSize(in_file=a_file)\n",
+    "outputs = cp_file_with_size(cache_root=tmp_dir / \"cache\")\n",
+    "\n",
+    "pprint(task_as_dict(outputs))"
+   ]
   }
  ],
  "metadata": {