Calculate num ops (#434)

* 1119

* 1120

* 1120.2

* model_path

* remove unnecessary files and pre-committed

* remove unnecessary files and pre-committed

* 1121 remove unnecessary files

* modify rev version

* modify rev version

* modify rev version

* accuracy issues targeted

* test script and modify feature

* return set[str]

* add logfile for test

* filter can get the number of kernels in naive_graph_decomposer

* post extract process feature

* remove unnecessary code blocks and variables

* modify the way of counting kernels used

* modify the way of counting kernels used

* modify script, rename files and variables

* add failure protection and log output when removing directories

* add a script to check fusability of a given model

* add a script to check if a given model is fully fusable

* add a script to check if a given model is fully fusable

* a script to check if a given model is fully fusable

* add a script to check if a given model is fully fusionable

* add a script to find fully fusionable subgraph

* find the biggest fully fusionable subgraph

* get fusible subgraph test

* modify get fully fusible subgraph

* improve fully_fusible_subgraph_extractor.py efficiency

* backup code

* Improve efficiency of test/fully_fusible_subgraph_extractor_test.sh

* check_graph_module_parsable

* Calculate num ops

---------

Co-authored-by: roll-away <[email protected]>

Author: lixinqi

graph_net/tools/check_graph_module_parsable.sh

@@ -11,8 +11,8 @@ config_json_str=$(cat <<EOF
     "handler_path": "$GRAPH_NET_ROOT/torch/check_graph_module_parsable.py",
     "handler_class_name": "CheckGraphModuleParsable",
     "handler_config": {
+        "resume": false,
         "model_path_prefix": "$GRAPH_NET_ROOT/../",
-        "resume": true,
         "limits_handled_models": 999999,
         "output_dir": "/tmp/check_graph_module_parsable"
     }

graph_net/torch/check_graph_module_parsable.py

@@ -31,11 +31,16 @@ def __call__(self, rel_model_path):
         model_path = os.path.join(self.config["model_path_prefix"], rel_model_path)
         if self.config["resume"] and self._is_model_path_handled(rel_model_path):
             return
-        parse_immutable_model_path_into_sole_graph_module(model_path)
+        gm = parse_immutable_model_path_into_sole_graph_module(model_path)
         output_dir = Path(self.config["output_dir"]) / rel_model_path
         output_dir.mkdir(parents=True, exist_ok=True)
+        (output_dir / "num_ops.txt").write_text(str(self._get_num_ops(gm)))
         self._inc_num_handled_models()
 
+    def _get_num_ops(self, gm):
+        ignored = {"placeholder", "output"}
+        return len([node for node in gm.graph.nodes if node.op not in ignored])
+
     def _is_model_path_handled(self, rel_model_path):
         return (Path(self.config["output_dir"]) / rel_model_path).exists()