Refactoring docs and internals to improve readability (PolusAI#339)

Author: vjaganat90

docs/userguide.md

@@ -10,7 +10,7 @@ See [overview](overview.md)
 
 Many software packages have a way of automatically discovering files which they can use. (examples: [pytest](https://docs.pytest.org/en/latest/explanation/goodpractices.html#conventions-for-python-test-discovery) [pylint](https://pylint.pycqa.org/en/latest/user_guide/usage/run.html))
 
-By default, wic will recursively search for tools / workflows within the directories (and subdirectories) listed in the config file's json tags `search_paths_cwl` and `search_paths_wic`. The paths listed can be absolute or relative. The default `config.json` is shown.
+By default, sophios will recursively search for tools / workflows within the directories (and subdirectories) listed in the config file's json tags `search_paths_cwl` and `search_paths_wic`. The paths listed can be absolute or relative. The default `config.json` is shown.
 
 ***`We strongly recommend placing all repositories of tools / workflows in the same parent directory.`***
 
@@ -39,7 +39,7 @@ By default, wic will recursively search for tools / workflows within the directo
 .....
 ```
 
-If you do not specify config file using the command line argument `--config`, it will be automatically created for you the first time you run wic in `~/wic/global_config.json`. (Because of this, the first time you run wic you should be in the root directory of any one of your repos.) Then you can manually edit this file with additional sources of tools / workflows.
+If you do not specify config file using the command line argument `--config`, it will be automatically created for you the first time you run sophios in `~/wic/global_config.json`. (Because of this, the first time you run sophios you should be in the root directory of any one of your repos.) Then you can manually edit this file with additional sources of tools / workflows.
 
 To avoid dealing with relative file paths in YAML files, by default
 
@@ -74,7 +74,7 @@ steps:
       message: !ii Hello World
 ```
 
-Note that this is one key difference between WIC and CWL. In CWL, all inputs must be given in a separate file. In WIC, inputs can be given inline with !ii and after compilation they will be automatically extracted into the separate file.
+Note that this is one key difference between sophios and CWL. In CWL, all inputs must be given in a separate file. In sophios, inputs can be given inline with !ii and after compilation they will be automatically extracted into the separate file.
 
 (NOTE: raw CWL is still supported with the --allow_raw_cwl flag.)
 
@@ -288,7 +288,7 @@ Similar to `scatter`, `when` is a **special (and optional)** attribute to any st
 The `when` attribute of a step object exposes the exact same js embedded syntax of `when` tag of the YAML/CWL syntax. One has to be careful about appropriate escaping in the string input of `when` in Python API. In the above case the comparison is between two strings so "" is around the literal 27 (i.e. value after `toString` step).
 ## Partial Failures
 
-In running workflows at scale, sometimes it is the case that one of the workflow steps may crash due to a bug causing the entire workflow to crash. In this case can use `--partial_failure_enable` flag. For special cases when the exit status of a workflow step isn't 1, and a different error code is returned (for example 142), then the user can supply the error code to wic as a success code to prevent workflow from crashing with `--partial_failure_success_codes 0 1 142`. By default partial failure flag will consider only 0 and 1 as success codes. An example line snippet of the error code being printed is shown below.
+In running workflows at scale, sometimes it is the case that one of the workflow steps may crash due to a bug causing the entire workflow to crash. In this case can use `--partial_failure_enable` flag. For special cases when the exit status of a workflow step isn't 1, and a different error code is returned (for example 142), then the user can supply the error code to sophios as a success code to prevent workflow from crashing with `--partial_failure_success_codes 0 1 142`. By default partial failure flag will consider only 0 and 1 as success codes. An example line snippet of the error code being printed is shown below.
 ```
 [1;30mWARNING[0m [33m[job compare_extract_protein_pdbbind__step__4__topology_check] exited with status: 139[0m
 ```

docs/validation.md

@@ -2,7 +2,7 @@
 
 # Formal Schemas
 
-What is validation? Validation is the process of checking whether the contents of a file agree with a formal specification. A popular modern choice for formal specification language is jsonschema, which is what wic uses. (Don't worry, you do not need to learn jsonschema. See below.)
+What is validation? Validation is the process of checking whether the contents of a file agree with a formal specification. A popular modern choice for formal specification language is jsonschema, which is what sophios uses. (Don't worry, you do not need to learn jsonschema. See below.)
 
 ## Different levels of strictness
 Notice that I said `a` formal specification, not `the` formal specification. That's because there are many different kinds of schemas, which may be more or less strict.
@@ -19,33 +19,33 @@ Next, there are various basic schemas for simply checking that a yml file is syn
 
 Since CWL files are just special yml files, there are [validators](https://github.com/common-workflow-language/cwl-utils/blob/main/cwl_utils/parser/cwl_v1_0.py) for these special yml files.
 
-Getting better, but wic workflows aren't just any yml or cwl files, they are even more special. So how can we create a special schema that will only accept wic workflows?
+Getting better, but sophios workflows aren't just any yml or cwl files, they are even more special. So how can we create a special schema that will only accept sophios workflows?
 
-## The WIC schema
+## The sophios schema
 
 At the opposite end of the spectrum (from `{}`), we could create the most strict schema possible. This would be a schema which only accepts inputs from ***`all known tools and subworkflows`***. In other words, we can use a ***`whitelist`*** instead of a blacklist. We can assume a closed world instead of an open world. Where does this whitelist come from? It comes from the [auto-discovery](userguide.md#auto-discovery) mechanism!
 
-So the command `wic --generate_schemas`
+So the command `sophios --generate_schemas`
 * makes a list of every single tool and workflow available
 * generates a separate sub-schema for each tool / workflow individually (this is why the flag says schemas, plural)
 * combines all of the sub-schemas into a single giant disjoint union!
 
-In other words, each step in a WIC workflow had better be chosen from a list of valid steps!
+In other words, each step in a sophios workflow had better be chosen from a list of valid steps!
 
 ### Stale schemas ###
 
-A direct consequence of using the most strict possible schema is that, as you add and/or modify tools and workflows, you have to keep re-generating the WIC schema so it doesn't become stale. Otherwise you will get validation errors, because you will be attempting to validate against an old schema which no longer reflects the tool and workflows that are currently available.
+A direct consequence of using the most strict possible schema is that, as you add and/or modify tools and workflows, you have to keep re-generating the sophios schema so it doesn't become stale. Otherwise you will get validation errors, because you will be attempting to validate against an old schema which no longer reflects the tool and workflows that are currently available.
 
 The exact same thing happens in Python when you are in the middle of editing a file. Until you push the save button, VSCode will keep attempting to validate your python code against the functions and methods that were available the last time you saved. It isn't a surprise that you get a red squiggly line while you are still typing. Makes sense, right?
 
-That said, for technical and performance reasons, (for now) we do not automatically generate a new wic schema.
+That said, for technical and performance reasons, (for now) we do not automatically generate a new sophios schema.
 
 ## TL;DR ##
 
 You must periodically run this command!
 
 ```
-rm -rf autogenerated/schemas/ && wic --generate_schemas
+rm -rf autogenerated/schemas/ && sophios --generate_schemas
 ```
 
 If you are getting validation errors, try re-running this command!
@@ -54,15 +54,15 @@ If you are getting validation errors, try re-running this command!
 
 So why are we going through all this trouble to create the most strict possible schema?
 
-The answer is that thanks to the excellent hypothesis-jsonschema library, we can use the WIC schema to perform property-based integration testing
+The answer is that thanks to the excellent hypothesis-jsonschema library, we can use the sophios schema to perform property-based integration testing
 
-***`on the entire WIC language, for every single tool and workflow simultaneously!`***
+***`on the entire sophios language, for every single tool and workflow simultaneously!`***
 
 We can randomly generate ***`entire synthetic workflows!`***
 
-This has [already been implemented](https://github.com/PolusAI/workflow-inference-compiler/blob/master/tests/test_fuzzy_compile.py), and it has been incredibly useful for finding a few (and thankfully only a few!) bugs. It also revealed a few subtle design issues (which have long ago been fixed).
+This has [already been implemented](https://github.com/PolusAI/sophios/blob/master/tests/test_fuzzy_compile.py), and it has been incredibly useful for finding a few (and thankfully only a few!) bugs. It also revealed a few subtle design issues (which have long ago been fixed).
 
-The [Fuzzy Compile CI logs](https://github.com/PolusAI/workflow-inference-compiler/actions/workflows/fuzzy_compile_weekly.wic) should speak for themselves.
+The [Fuzzy Compile CI logs](https://github.com/PolusAI/sophios/actions/workflows/fuzzy_compile_weekly.wic) should speak for themselves.
 
 ## Well what about ... !?!
 

src/sophios/post_compile.py

@@ -18,17 +18,22 @@ def find_output_dirs(data: Union[RoseTree, Dict, list]) -> list:
         list: A list of location values.
     """
     results = []
-    if isinstance(data, Dict):
-        if "class" in data and data["class"] == "Directory" and "location" in data:
-            if isinstance(data["location"], dict) and "wic_inline_input" in data["location"]:
-                results.append(data["location"]["wic_inline_input"])
-            else:
-                results.append(data["location"])
-        for value in data.values():
-            results.extend(find_output_dirs(value))
-    elif isinstance(data, list):
-        for item in data:
-            results.extend(find_output_dirs(item))
+    match data:
+        case dict() as data_dict:
+            match data_dict:
+                case {"class": "Directory", "location": {"wic_inline_input": val}, **rest_data_dict}:
+                    results.append(val)
+                case {"class": "Directory", "location": dl, **rest_data_dict}:
+                    results.append(dl)
+                case _:
+                    pass
+            for value in data_dict.values():
+                results.extend(find_output_dirs(value))
+        case list(l):
+            for item in l:
+                results.extend(find_output_dirs(item))
+        case _:
+            pass
 
     return results
 
@@ -79,6 +84,8 @@ def remove_entrypoints(container_engine: str, rose_tree: RoseTree) -> RoseTree:
     # Requires root, so guard behind CLI option
     if container_engine == 'docker':
         plugins.remove_entrypoints_docker()
-    if container_engine == 'podman':
+    elif container_engine == 'podman':
         plugins.remove_entrypoints_podman()
+    else:
+        pass
     return plugins.dockerPull_append_noentrypoint_rosetree(rose_tree)

src/sophios/run_local.py

@@ -9,7 +9,7 @@
 import shutil
 import platform
 import traceback
-from typing import Dict, List, Optional
+from typing import List, Optional
 from datetime import datetime
 
 try:
@@ -413,20 +413,23 @@ def stage_input_files(yml_inputs: Yaml, root_yml_dir_abs: Path,
         FileNotFoundError: If throw and it any of the input files do not exist.
     """
     for key, val in yml_inputs.items():
-        if isinstance(val, Dict) and val.get('class', '') == 'File':
-            path = root_yml_dir_abs / Path(val['path'])
-            if not path.exists() and throw:
-                # raise FileNotFoundError(f'Error! {path} does not exist!')
-                print(f'Error! {path} does not exist!')
-                sys.exit(1)
-
-            relpath = Path('autogenerated/') if relative_run_path else Path('.')
-            pathauto = relpath / Path(val['path'])  # .name # NOTE: Use .name ?
-            pathauto.parent.mkdir(parents=True, exist_ok=True)
-
-            if path != pathauto:
-                cmd = ['cp', str(path), str(pathauto)]
-                proc = sub.run(cmd, check=False)
+        match val:
+            case {'class': 'File', **rest_of_val}:
+                path = root_yml_dir_abs / Path(val['path'])
+                if not path.exists() and throw:
+                    # raise FileNotFoundError(f'Error! {path} does not exist!')
+                    print(f'Error! {path} does not exist!')
+                    sys.exit(1)
+
+                relpath = Path('autogenerated/') if relative_run_path else Path('.')
+                pathauto = relpath / Path(val['path'])  # .name # NOTE: Use .name ?
+                pathauto.parent.mkdir(parents=True, exist_ok=True)
+
+                if path != pathauto:
+                    cmd = ['cp', str(path), str(pathauto)]
+                    _ = sub.run(cmd, check=False)
+            case _:
+                pass
 
 
 def cwltool_main() -> int:

src/sophios/utils_cwl.py

@@ -229,15 +229,19 @@ def canonicalize_type(type_obj: Any) -> Any:
     Returns:
         Any: The JSON canonical normal form associated with type_obj
     """
-    if isinstance(type_obj, str):
-        if len(type_obj) >= 1 and type_obj[-1:] == '?':
-            return ['null', canonicalize_type(type_obj[:-1])]
-        if len(type_obj) >= 2 and type_obj[-2:] == '[]':
-            return {'type': 'array', 'items': canonicalize_type(type_obj[:-2])}
-    if isinstance(type_obj, Dict):
-        if type_obj.get('type') == 'array':
-            return {**type_obj, 'items': canonicalize_type(type_obj['items'])}
-    return type_obj
+    match type_obj:
+        case str() as str_obj:
+            if len(str_obj) >= 1 and str_obj[-1:] == '?':
+                return ['null', canonicalize_type(str_obj[:-1])]
+            if len(str_obj) >= 2 and str_obj[-2:] == '[]':
+                return {'type': 'array', 'items': canonicalize_type(str_obj[:-2])}
+            return str_obj
+        case dict() as dict_obj:
+            if dict_obj.get('type') == 'array':
+                return {**dict_obj, 'items': canonicalize_type(dict_obj['items'])}
+            return dict_obj
+        case _:
+            return type_obj
 
 
 def canonicalize_steps_list(steps: Yaml) -> List[Yaml]: