Use pickled data as snapshot file

* Add 'pickle' output module and use it as one of default outputs
* Refactor output file selection into 'select_output_file'. This
  function can be used by all output modules (future improvement)
* Add 'load_pickled_data' to 'utils.read'
* Read pickled or YAML data in 'load_snapshot' function

Minor changes:

* Change the default value of the 'args.snapshot' parameter
* Change the parser default value of '--snapshot' parameter
* Change the help message in 'netlab create' command
* Add 'snapshot' argument to 'netlab down' to be able to shut down
  a lab from the YAML snapshot file
* Add 'netlab.snapshot.pickle' as another file to clean up when the
  lab is shut down
* Fix exception reporting to use 'str(ex)' (just to be on the safe
  side)

Off-topic:

* Create a large topology in platform-integration tests to test the
  performance improvements

Finally:

* Updated the documentation to refer to the pickle snapshot file

Implements #2635

Author: ipspace

docs/dev/config/deploy.md

@@ -193,7 +193,7 @@ The filename search patterns can use Jinja2 expressions that rely on [Ansible va
 | `package:` | **networklab** package (system file) |
 | `topology:` | Directory containing the current lab topology |
 
-The directory search patterns are evaluated during the [data transformation process](../transform.md), and the absolute paths are stored in the Ansible inventory (in the **all** group) and the `netlab.snapshot.yml` file. You can use the **netlab inspect defaults.paths** command to display the transformed values (**netlab inspect** command is available after starting the lab or executing the **netlab create** command).
+The directory search patterns are evaluated during the [data transformation process](../transform.md), and the absolute paths are stored in the Ansible inventory (in the **all** group) and the `netlab.snapshot.pickle` file. You can use the **netlab inspect defaults.paths** command to display the transformed values (**netlab inspect** command is available after starting the lab or executing the **netlab create** command).
 
 For example, the default custom configuration template directory  search path (**defaults.paths.custom.dirs**) contains these entries:
 
@@ -206,7 +206,7 @@ custom:                         # Custom configuration templates
   - "package:extra"
 ```
 
-The evaluated search path that is stored in `netlab.snapshot.yml` might contain these values (the topology directory is equal to the current directory, **netlab** is executed from local Git directory):
+The evaluated search path that is stored in `netlab.snapshot.pickle` might contain these values (the topology directory is equal to the current directory, **netlab** is executed from local Git directory):
 
 ```
 $ netlab inspect defaults.paths.custom.dirs

docs/netlab/connect.md

@@ -35,7 +35,7 @@ The rest of the arguments are passed to SSH or docker exec command
 
 ## Collecting Device Data
 
-**netlab connect** uses the lab snapshot file (default: `netlab.snapshot.yml`) to read device- and node information. You can overwrite the default snapshot file with the `--snapshot` command line parameter.
+**netlab connect** uses the lab snapshot file (default: `netlab.snapshot.pickle`) to read device- and node information. You can overwrite the default snapshot file with the `--snapshot` command line parameter.
 
 **netlab connect** command uses the following device data. Most of that data is derived from the device **group_vars**, although you can override it on [node-](node-ansible-data) or [custom group](groups-object-data) level; use the **[`netlab inspect --node _name_`](inspect.md)** command to inspect it.
 

docs/netlab/create.md

@@ -22,16 +22,17 @@ You can influence the data model transformation with optional [configuration mod
 
 **netlab create** uses transformed node- and link-level data structures to create:
 
-* Snapshot of the transformed topology in the **netlab.snapshot.yml** file. This file is used by **netlab down** command to find the virtualization provider and link (bridge) names.
+* Pickled transformed topology data in the **netlab.snapshot.pickle** file. This file is used by many other **netlab** commands to get the information about the currently running lab topology.
+* Snapshot of the transformed topology in the **netlab.snapshot.yml** file. You can use this file to get the information about the current lab topology in your own scripts.
 * **Vagrantfile** supporting *[libvirt](lab-libvirt)* environment
 * **clab.yml** file used by *containerlab*.
 * Ansible inventory[^1], either as a single-file data structure, or as a minimal inventory file with data stored primarily in **host_vars** and **group_vars**
 * Various graphs in *graphviz* DOT format
 * YAML or JSON representation of transformed lab topology or parts of the transformed data model
-* Configuration file for *graphite* visualization tool
+* Configuration files for [external tools](extools.md)
 
 ```{warning}
-**‌netlab create** command refuses to create provider configuration files, Ansible inventory, or `netlab.snapshot.yml` file if it finds `netlab.lock` file in the current directory. 
+**‌netlab create** command refuses to create provider configuration files, Ansible inventory, or `netlab.snapshot.pickle` file if it finds `netlab.lock` file in the current directory. 
 
 `netlab.lock` file is created by the **‌netlab up** command to ensure subsequent **‌netlab create** commands don't overwrite the provider configuration files. It is automatically removed after a successful completion of **‌netlab down** command.
 ```
@@ -71,6 +72,7 @@ optional arguments:
 
 output files created when no output is specified:
 
+  * Pickled transformed data in netlab.snapshot.pickle
   * Transformed topology snapshot in netlab.snapshot.yml
   * Virtualization provider file with provider-specific filename
     (Vagrantfile or clab.yml)

docs/netlab/down.md

@@ -27,7 +27,7 @@ options:
 Notes:
 
 * **netlab down** needs transformed topology data to find the virtualization provider and link (bridge) names.
-* **netlab down** reads the transformed topology from `netlab.snapshot.yml` file created by **netlab up** or **netlab create**.
+* **netlab down** reads the transformed topology from `netlab.snapshot.pickle` file created by **netlab up** or **netlab create**.
 * With the `--instance` flag, you can shut down a lab instance running in a different directory. Use the `netlab status --all` command to display all running instances.
 * Use the `--cleanup` flag to delete all Ansible-, Vagrant- or containerlab-related configuration files.
 * Use the `--force` flag with the `--cleanup` flag if you want to clean up the directory even when the virtualization provider fails during the shutdown process.

docs/netlab/graph.md

@@ -1,7 +1,7 @@
 (netlab-graph)=
 # Creating Physical or BGP Topology Graphs
 
-**netlab graph** command creates a graph description in [Graphviz](https://graphviz.org/) or [D2](https://d2lang.com/) format from the transformed lab topology data (usually stored in `netlab.snapshot.yml`) created by **netlab create** command. It's replicating the functionality of **netlab create -o graph:_name_** command with a more convenient user interface. 
+**netlab graph** command creates a graph description in [Graphviz](https://graphviz.org/) or [D2](https://d2lang.com/) format from the transformed lab topology data (usually stored in `netlab.snapshot.pickle`) created by **netlab create** command. It's replicating the functionality of **netlab create -o graph:_name_** command with a more convenient user interface. 
 
 ```{note}
 You will have to install [Graphviz](https://graphviz.org/download/) or [D2](https://d2lang.com/tour/install) and use them to create the actual graphs in PNG/JPEG/PDF format.

docs/netlab/initial.md

@@ -26,7 +26,7 @@ Jinja2 templates are used together with **_device_\_config** Ansible modules to
 [^init]: For example, Juniper vMX requires an evaluation license to be applied after the device boots.
 
 ```{tip}
-* The **netlab initial** command reads the transformed lab data from the `netlab.snapshot.yml` file created by the **netlab up** command.
+* The **netlab initial** command reads the transformed lab data from the `netlab.snapshot.pickle` file created by the **netlab up** command.
 * When run with the **-v** parameter, the command displays device configurations before deploying them.
 ```
 

docs/netlab/inspect.md

@@ -1,7 +1,7 @@
 (netlab-inspect)=
 # Inspect Data Structures in Transformed Lab Topology
 
-**netlab inspect** prints data structures in transformed lab topology (usually stored in `netlab.snapshot.yml`) created by the **netlab create** command. You can display data in YAML or JSON format and select a subset of data from the transformed topology or an individual node.
+**netlab inspect** prints data structures in transformed lab topology (usually stored in `netlab.snapshot.pickle`) created by the **netlab create** command. You can display data in YAML or JSON format and select a subset of data from the transformed topology or an individual node.
 
 When selecting data from an individual node, _netlab_ adds group variables to node data, effectively displaying what you would see in the Ansible inventory.
 

docs/netlab/report.md

@@ -1,7 +1,7 @@
 (netlab-report)=
 # Creating a Report
 
-The **netlab report** command creates a report from the transformed lab topology data (usually stored in `netlab.snapshot.yml`) created by the **netlab create** command. It replicates the functionality of the **netlab create -o report:_name_** command with a more convenient user interface.
+The **netlab report** command creates a report from the transformed lab topology data (usually stored in `netlab.snapshot.pickle`) created by the **netlab create** or **netlab up** command. It replicates the functionality of the **netlab create -o report:*name*** command with a more convenient user interface.
 
 ## Usage
 

docs/netlab/restart.md

@@ -1,6 +1,6 @@
 # Restart Virtual Lab
 
-**netlab restart** executes **[netlab down](down.md)** followed by **[netlab up](up.md)** to restart your lab from the transformed lab topology stored in `netlab.snapshot.yml` snapshot file.
+**netlab restart** executes **[netlab down](down.md)** followed by **[netlab up](up.md)** to restart your lab from the transformed lab topology stored in `netlab.snapshot.pickle` snapshot file.
 
 You can use **netlab restart** to restart the existing lab (use `--snapshot` keyword) or recreate the lab configuration files if you changed the lab topology.
 

docs/netlab/status.md

@@ -115,7 +115,7 @@ The **netlab status cleanup _instance_** command shuts down the specified lab in
 $ netlab status --cleanup
 Cleanup will remove lab instance "default" in /home/user/net101/tools/X. Are you sure? [y/n]y
 Shutting down lab default in /home/user/net101/tools/X
-Read transformed lab topology from snapshot file netlab.snapshot.yml
+Read transformed lab topology from snapshot file netlab.snapshot.pickle
 
 ┌──────────────────────────────────────────────────────────────────────────────────┐
 │ CHECKING virtualization provider installation                                    │
@@ -147,4 +147,5 @@ INFO[0001] Removing containerlab host entries from /etc/hosts file
 ... removing directory tree group_vars
 ... removing directory tree host_vars
 ... removing netlab.snapshot.yml
+... removing netlab.snapshot.pickle
 ```