Add 'debug at startup' capability

This is a proof-of-concept (for Cisco IOS) of a capability that
enables debugging at the very beginning of initial device configuration
to ensure no relevant events are lost.

It introduces a new node attribute (debug), an extra flag to
'must_be_list' function that can split lines of a string value when
netlab expects a list of values, and a sample initial config template.

Also, I added a whole document explaning how one could do debugging
based on my lovely recent experience with Cisco IOS and Aruba CX
(more about those coming soon from the usual soapbox).

Author: ipspace

docs/node/debug.md

@@ -0,0 +1,54 @@
+(node-debug)=
+# Debugging Network Devices
+
+It's usually trivial to enable debugging on devices running in a virtual lab:
+
+1. Log in to the affected devices
+2. Enable debugging
+3. Monitor debugging outputs in log files or an SSH session
+
+However, you might want to enable specific debugging printouts every time you start a lab topology. That's pretty easy to do on most devices that allow you to execute regular commands (for example, the **debug** command) within the configuration mode:
+
+1. Create a [custom configuration template](custom-config) that enables debugging, for example:
+
+   ```
+   do debug ipv6 routing
+   do debug bgp ipv6 unicast import events
+   do debug bgp ipv6 unicast import updates
+   do debug bgp ipv6 unicast updates out
+   ```
+
+2. Add the name of the custom configuration template to a node **config** list.
+
+```{tip}
+This approach allows you to automate debugging in a [multi-vendor environment](custom-config-multivendor)
+```
+
+Unfortunately (for debugging purposes), _netlab_ executes the custom configuration templates after configuring the network devices, which means you might lose the interesting part of the debugging information: what happens when the control-plane protocols start.
+
+You can enable debugging *before* configuring network devices with this simple trick:
+
+1. Execute **netlab up --no-config**
+2. If needed, execute **netlab initial --ready** to ensure the network devices are fully operational
+3. Log into the network devices and enable debugging, or use a custom configuration template and execute it with **netlab config _debugging_template_** (this approach yet again gives you multi-vendor capabilities).
+4. Start device configuration with **netlab initial**
+
+Finally, you might be worried that the symptoms you're experiencing depend on the time after the device boots, so you want to enable debugging as soon as possible. In that case, you can use the node **debug** attribute (on devices for which we implemented it), which is a list of device-specific debugging parameters that will be executed at the very beginning of the initial device configuration.
+
+**Caveats:**
+
+* The contents of the **debug** attribute are device-specific. *netlab* currently does not have multi-vendor **debug** capabilites. The values of the **debug** attribute must be relevant to the underlying network device, or you'll get configuration errors during the initial configuration
+* The initial device configuration template supplies the mandatory prefix (for example, **do debug**). You only have to list the debugging conditions, for example:
+
+```
+nodes:
+  dut:
+    module: [ bgp, ospf, routing ]
+    bgp.as: 65000
+    device: iol
+    debug:
+    - ip routing
+    - ipv6 routing
+    - bgp ipv4 unicast updates
+    - bgp ipv6 unicast updates
+```

docs/node-roles.md docs/node/roles.mddocs/node-roles.md renamed to docs/node/roles.md

@@ -67,6 +67,7 @@ nodes:
 * **config** -- extra [configuration templates](custom-config) applied to this device.
 * **cpu** -- virtual CPU cores allocated to the VM lab device. It does not apply to container-based devices.
 * **device** -- device type (see [supported platforms](platforms.md)). [Default device type](default-device-type) is specified in **defaults.device**.
+* **debug** -- device-specific [debugging settings](node-debug) enabled at the very beginning of the initial device configuration.
 * **group** -- list of [groups](topo-groups) this node belongs to.
 * **id** -- static node identifier[^id] (see below)
 * **image** or **box** -- specifies the Vagrant box or Docker container used by the lab device. Default images for individual device types are defined in system defaults and can be changed with **defaults.devices...** settings ([more details](default-device-image)).
@@ -321,5 +322,6 @@ $ netlab inspect --node all id
 .. toctree::
    :maxdepth: 1
 
-   node-roles.md
+   node/roles.md
+   node/debug.md
 ```

docs/nodes.md

@@ -31,6 +31,7 @@ Once you want to know more, check out these lab topology tutorials:
 * [Mix Containers and VMs in the Same Lab Topology](https://blog.ipspace.net/2023/02/netlab-vm-containers/)
 * [Using VLAN and VRF Links](https://blog.ipspace.net/2023/04/netlab-vrf-vlan-links/)
 * [](example-bridge)
+* [](node-debug)
 
 ## User Interface Tutorials
 

docs/tutorials.md

@@ -1,6 +1,14 @@
 hostname {{ inventory_hostname }}
 !
 no ip domain lookup
+logging buffered 256000
+{% for dbg in debug|default([]) %}
+{%   if loop.first %}
+!
+! Enable debugging
+{%   endif %}
+do debug {{ dbg }}
+{% endfor %}
 !
 lldp run
 !

netsim/ansible/templates/initial/ios.j2

@@ -438,15 +438,23 @@ def register_type(tname: str, validator: typing.Callable) -> None:
 """
 
 @type_test(false_value=[],empty_value=[])
-def must_be_list(value: typing.Any, make_list: bool = False) -> dict:
+def must_be_list(value: typing.Any, make_list: bool = False, split_lines: bool = False) -> dict:
 
   def transform_to_list(value: typing.Any) -> list:
+    if isinstance(value,str) and split_lines:
+      return value.rstrip().split("\n")
+
     return [ value ]
 
   if isinstance(value,list):                            # A list is what we want to have ;)
     return { '_valid': True }
 
-  if isinstance(value,(str,int,float,bool)):            # Handle scalar-to-list transformations with a callback function
+  # Handle scalar-to-list transformations with a callback function
+  # Please note that the 'split_lines' value used in that callback function
+  # is tied to the argument value passed to this function (it's a magic
+  # transfer of hidden variables)
+  #
+  if isinstance(value,(str,int,float,bool)):
     return { '_valid': True, '_transform': transform_to_list }
 
   if make_list:                                         # Optional: force any other value to become a list

netsim/data/types.py

@@ -91,6 +91,7 @@ node:
   box: str
   id: { type: int, min_value: 1, max_value: 150 }
   config: list
+  debug: { type: list, split_lines: True }
   group: list
   role: { type: str, valid_values: [ router, host, bridge, gateway ] }
   mgmt: