Merge pull request #1 from compspec/add-transformers

wip: creation of subsystem solver as an interface

Author: vsoch

README.md

@@ -18,6 +18,18 @@ We want to:
 
 See [examples/fractale](examples/fractale) for a detailed walk-through of the above.
 
+For graph tool:
+
+```bash
+conda install -c conda-forge graph-tool
+```
+
+## Questions
+
+- Should other subsystem types have edges? How used?
+- Should we try to map them to nodes in the graph or use another means (or assume global across cluster nodes as we do now)?
+- Can we simplify spack subsystem graph (it's really big...)
+
 <!-- ⭐️ [Documentation](https://compspec.github.io/fractale) ⭐️ -->
 
 ## License

examples/fractale/README.md

@@ -25,6 +25,11 @@ fractale generate --cluster A spack /home/vanessa/Desktop/Code/spack
 
 ## Satisfy Request
 
+Satisfy asks two questions:
+
+1. Which clusters have the subsystem resources that I need?
+2. Which clusters have the job resources that I need (containment subsystem)?
+
 This is the step where we want to say "Run gromacs on 2-4 nodes with these requirements." Since we haven't formalized a way to do that, I'm going to start with a flux jobspec, and then add attributes that can be used to search our subsystems. For example, I generated [software-gromacs.json](software-gromacs.json) with:
 
 ```bash
@@ -47,6 +52,40 @@ fractale satisfy ./examples/fractale/software-curl.yaml
 fractale satisfy ./examples/fractale/software-curl.json
 ```
 
+Try the graph backend, install [graph-tool](https://graph-tool.skewed.de/installation.html) then:
+
+```bash
+export PYTHONPATH=/home/vanessa/anaconda3/lib/python3.12/site-packages
+fractale satisfy --solver graph ./examples/fractale/software-curl.yaml
+fractale satisfy --solver graph ./examples/fractale/software-curl.json
+```
+```console
+=> 🍇 Loading cluster "a" subsystem "containment"
+=> 🍇 Loading cluster "a" subsystem "modules"
+=> 🍇 Loading cluster "a" subsystem "spack"
+    => Exploring cluster "a" containment subsystem
+          (1/1) satisfied resource core 
+Cluster "a" is a match
+```
+
+Here is for a nested slot:
+
+```bash
+fractale satisfy --solver graph ./examples/fractale/software-nested-slot.yaml 
+```
+```console
+=> 🍇 Loading cluster "a" subsystem "containment"
+=> 🍇 Loading cluster "a" subsystem "modules"
+=> 🍇 Loading cluster "a" subsystem "spack"
+    => Exploring cluster "a" containment subsystem
+          (1/1) satisfied resource socket 
+          (1/4) found resource core 
+          (2/4) found resource core 
+          (3/4) found resource core 
+          (4/4) satisfied resource core 
+Cluster "a" is a match
+```
+
 By default, the above assumes subsystems located in the fractale home. If you want to adjust that, set `fractale --config-dir=<path> satisfy...` to adjust that (and note you will need to have generated the tree here. What we basically do with satisfy is build a database with tables for:
 
 - clusters
@@ -67,4 +106,32 @@ And right now the search is just over attributes to find matching clusters. E.g.
 }
 ```
 
+Here is a jobspec that can't be satisfied because we ask for too many resources given the cluster [containment-subsystem.json]([containment-subsystem.json).
+
+```bash
+fractale satisfy ./examples/fractale/jobspec-containment-unsatisfied.yaml 
+```
+```console
+=> 🍇 Loading cluster "a" subsystem "containment"
+=> 🍇 Loading cluster "a" subsystem "modules"
+=> 🍇 Loading cluster "a" subsystem "spack"
+       (2) SELECT * from subsystems WHERE type = 'software';
+=> No Matches due to containment
+```
 We likely want to have a more structured query syntax that can handle AND, OR, and other specifics. The actual search should remain general to support any generic key/value pair of attributes. My database structure and queries are also bad. 
+
+## Save
+
+We can save an image of our subystem for a cluster. E.g.,
+
+```bash
+$ fractale save a --out ./examples/fractale/cluster-a-containment.png
+```
+```console
+=> 🍇 Loading cluster "a" subsystem "containment"
+=> 🍇 Loading cluster "a" subsystem "modules"
+=> 🍇 Loading cluster "a" subsystem "spack"
+Saving to "./examples/fractale/cluster-a-containment.png"
+```
+
+<img src="./cluster-a-containment.svg">

examples/fractale/cluster-a-containment-graph.svg

@@ -0,0 +1,21 @@
+version: 1
+resources:
+- type: slot
+  count: 1
+  with:
+  - type: core
+    count: 100
+  label: task
+tasks:
+- command:
+  - gmx
+  slot: task
+  count:
+    per_slot: 1
+attributes:
+  system:
+    duration: 0
+    requires:
+      software:
+        - name: curl
+          type: binary

examples/fractale/cluster-a-containment.png

@@ -0,0 +1,24 @@
+version: 1
+resources:
+- type: slot
+  count: 2
+  with:
+  - type: socket
+    count: 1
+    with:
+      - type: core
+        count: 4
+  label: task
+tasks:
+- command:
+  - gmx
+  slot: task
+  count:
+    per_slot: 1
+attributes:
+  system:
+    duration: 0
+    requires:
+      software:
+        - name: curl
+          type: binary

examples/fractale/jobspec-containment-unsatisfied.yaml

@@ -8,6 +8,7 @@
 from compspec.plugin.registry import PluginRegistry
 
 import fractale
+import fractale.defaults as defaults
 from fractale.logger import setup_logger
 
 # Generate the plugin registry to add parsers
@@ -71,15 +72,31 @@ def get_parser():
     generate.add_argument("-c", "--cluster", help="cluster name")
 
     # run.add_argument("-t", "--transform", help="transformer to use", default="flux")
+    save = subparsers.add_parser(
+        "save",
+        formatter_class=argparse.RawTextHelpFormatter,
+        description="save a picture of a subsystem graph",
+    )
+    save.add_argument("cluster", help="cluster to save")
+    save.add_argument(
+        "--subsystem", help="cluster to save (defaults to containment)", default="containment"
+    )
+    save.add_argument("--out", help="output file name")
 
     # This does just the user space subsystem match
     satisfy = subparsers.add_parser(
         "satisfy",
         formatter_class=argparse.RawTextHelpFormatter,
         description="determine clusters that satisfy a jobspec based on user subsystems",
     )
-    for cmd in [satisfy]:
-        cmd.add_argument("jobspec", help="jobspec yaml or json file")
+    satisfy.add_argument("jobspec", help="jobspec yaml or json file")
+    for cmd in [satisfy, save]:
+        cmd.add_argument(
+            "--solver",
+            help="subsystem solved backend",
+            default=defaults.solver_backend_default,
+            choices=defaults.solver_backends,
+        )
 
     extractors = generate.add_subparsers(
         title="generate",
@@ -131,6 +148,8 @@ def help(return_code=0):
         from .generate_subsystem import main
     elif args.command == "satisfy":
         from .satisfy import main
+    elif args.command == "save":
+        from .save import main
     else:
         help(1)
     global registry

examples/fractale/software-nested-slot.yaml

@@ -3,7 +3,7 @@
 import sys
 
 from fractale.store import FractaleStore
-from fractale.subsystem import get_subsystem_registry
+from fractale.subsystem import get_subsystem_solver
 
 
 def main(args, extra, **kwargs):
@@ -12,6 +12,6 @@ def main(args, extra, **kwargs):
     This is a fairly simple (flat) check.
     """
     store = FractaleStore(args.config_dir)
-    registry = get_subsystem_registry(store.clusters_root)
-    is_satisfied = registry.satisfied(args.jobspec)
+    solver = get_subsystem_solver(store.clusters_root, args.solver)
+    is_satisfied = solver.satisfied(args.jobspec)
     sys.exit(0 if is_satisfied else -1)

fractale/cli/__init__.py

@@ -0,0 +1,18 @@
+#!/usr/bin/env python
+
+import sys
+
+from fractale.store import FractaleStore
+from fractale.subsystem import get_subsystem_solver
+
+
+def main(args, extra, **kwargs):
+    """
+    Save a cluster and subsystem graph.
+    """
+    store = FractaleStore(args.config_dir)
+    solver = get_subsystem_solver(store.clusters_root, args.solver)
+    outfile = args.out
+    if not outfile:
+        outfile = f"cluster-{args.cluster}-{args.subsystem}-{args.solver}.svg"
+    solver.save(args.cluster, args.subsystem, outfile)

fractale/cli/satisfy.py

@@ -1,3 +1,5 @@
 fractale_dir = ".fractale"
 valid_settings = {"sharedfs", "stage"}
 sharedfs = True
+solver_backends = ["database", "graph"]
+solver_backend_default = "graph"