Use 'files' plugin to implement inline configuration templates (#2580)

This change adds support for inline custom configuration templates in
node-, group- and validation test definitions. It allows the user to
specify small additions to device configurations directly in the node-
or group definition, or to define small changes to device configurations
directly in the validation tests.

Author: ipspace

docs/custom-config-templates.md

@@ -1,7 +1,7 @@
 (custom-config)=
 # Custom Configuration Templates
 
-You can build complex labs with functionality that is [not yet part of *netlab*](netlab-customize) with the custom configuration templates that can be deployed with **[netlab config](netlab-config)**, **[netlab initial](netlab-initial)** or **[netlab up](netlab-up)** commands. The custom configuration templates could be stored in the lab topology directory, the user's defaults directory, or within the _netlab_ package. See [](dev-find-custom) for more details.
+You can build complex labs with functionality that is [not yet part of *netlab*](netlab-customize) with the custom configuration templates that can be deployed with **[netlab config](netlab-config)**, **[netlab initial](netlab-initial)**, or **[netlab up](netlab-up)** commands. The custom configuration templates could be stored in the lab topology directory, the user's defaults directory, or within the _netlab_ package. See [](dev-find-custom) for more details. You can also use the **[files](plugin-files)** plugin to [store the configuration templates in the lab topology](plugin-files-configlets).
 
 For a one-off deployment of custom configuration templates, use the **netlab config** command. To make the deployment of custom configuration template(s) part of a regular lab initialization process[^CC], use **config** group- or node attribute that can specify either a single template or a list of templates.
 
@@ -53,6 +53,8 @@ Node **config** attributes are merged with the group **‌config** attributes. [
 _netlab_ sorts custom configuration templates in the order specified in groups and nodes to speed up their deployment. Specifying `custom: [ a,b ]` on one node and `custom: [ b,a ]` on another will result in a sorting loop and a fatal error.
 ```
 
+Finally, the **files** plugin [allows you to use **config.inline** node— or group parameter](plugin-files-node-config) and store the configuration changes directly in the lab topology.
+
 (custom-config-groups)=
 ## Custom Configuration Templates in Hierarchical Groups
 
@@ -125,6 +127,10 @@ The following configuration templates would be applied to individual nodes in th
 | e    | g2a, g2b (from g2 via g3), g3 (from g3), -g1 is ignored, e (from e) |
 | f    | none (it's not a member of any group)           |
 
+```{tip}
+With the **files** plugin, you can use the [**config.inline** group parameter](plugin-files-node-config) to apply a configuration change to a group of lab devices.
+```
+
 (custom-config-multivendor)=
 ## Multi-Vendor and Device-Specific Templates
 

docs/plugins/files.md

@@ -8,6 +8,15 @@ The plugin defines two new topology attributes:
 * **files** -- a dictionary or list of extra files to create in the lab directory
 * **configlets** -- a dictionary of custom configuration templates
 
+It also extends the meaning of the node/group **config** attribute and adds the **config.inline** attribute to validation tests.
+
+```eval_rst
+.. contents:: Table of Contents
+   :depth: 2
+   :local:
+   :backlinks: none
+```
+
 (plugin-files-create)=
 ## Creating Extra Files
 
@@ -52,6 +61,7 @@ The files specified in the **files** attribute are created just before the outpu
 * The files specified in the **‌files** list will be removed during the `netlab down --cleanup` process, even when they existed before the `netlab up` was executed.
 ```
 
+(plugin-files-configlets)=
 ## Creating Custom Configuration Templates
 
 The **configlets** dictionary can be used to create custom configuration templates. Its contents are automatically converted to [elements of the **files** list](plugin-files-create), but provide a more intuitive way of specifying the templates.
@@ -60,8 +70,14 @@ For example, the following **configlets** dictionary creates **ifup.j2** and **i
 
 ```
 configlets:
-  ifup: ip link set eth1 up
-  ifdown: ip link set eth1 down
+  ifup: |
+    ip link set eth1 up
+  ifdown: |
+    ip link set eth1 down
+```
+
+```{tip}
+Always use the YAML *literal ‌block scalar header* (`|`) for the configlet content; otherwise, the whole content will be folded into a single line.
 ```
 
 You can use a more structured **configlets** dictionary to create custom configuration templates for multiple nodes, devices, or providers. For example, the following dictionary creates **ifup/eos.j2** and **ifup/frr.j2** files that can then be used as `ifup` custom configuration templates on Arista EOS or FRR:
@@ -110,3 +126,74 @@ configlets:
         interface Management1
           description Management interface
 ```
+
+(plugin-files-node-config)=
+## Inline Node/Group Configuration Templates
+
+The **files** plugin allows you to specify the contents of custom configuration templates directly in the node- or group **config** attribute.
+
+If you enabled the **files** plugin, and the content of the **config** attribute is a dictionary, the **files** plugin copies the dictionary into the **configlets** dictionary and replaces it with a list of custom configuration templates (the dictionary keys).
+
+If you don't care about the configuration template names, use the **config.inline** template. **files** plugin will auto-generate a template name and use it to apply extra configuration to the node (or group of nodes) where the **config.inline** template is defined.
+
+For example, the following lab topology creates a template `_n_x1.j2` that is then used to add a route map and extra BGP configuration to X1:
+
+```
+plugin: [ files ]
+module: [ bgp ]
+
+nodes:
+  dut:
+    bgp.as: 65000
+
+  x1:
+    id: 10
+    bgp.as: 65010
+    bgp.originate: 172.0.42.0/24
+    config.inline: |
+      route-map setcomm permit 10
+      set community 65000:1 additive
+      set extcommunity bandwidth 100
+      set large-community 65000:0:1 additive
+      exit
+      !
+      router bgp {{ bgp.as }}
+      !
+      address-family ipv4 unicast
+      {% for n in bgp.neighbors %}
+        neighbor {{ n.ipv4 }} route-map setcomm out
+      {% endfor %}
+
+  x2:
+    id: 11
+    bgp.as: 65011
+```
+
+```{tip}
+Always use the YAML *literal ‌block scalar header* (`|`) for the custom configuration content; otherwise, the whole content will be folded into a single line.
+```
+
+(plugin-files-validation-config)=
+## Inline Configuration Changes in Validation Tests
+
+Once the **files** plugin is activated, the validation tests can use the **config.inline** attribute to specify the changes that should be made to the device configuration directly in the validation test, for example:
+
+```
+validate:
+...
+  bgp_lbd:
+    description: Shut down the loopback interface
+    config:
+      inline:
+        frr: |
+          #!/bin/bash
+          ip link set lo down
+    pass: BGP prefix is no longer announced
+    nodes: [ xf ]
+```
+
+```{tip}
+Always use the YAML *literal ‌block scalar header* (`|`) for the configuration content; otherwise, the whole content will be folded into a single line.
+```
+
+

docs/topology/validate.md

@@ -40,9 +40,10 @@ You can also set these test string attributes to prettify the test results:
 
 The **show**, **exec**, and **valid** parameters can be strings or dictionaries. If you're building a lab that will be used with a single platform, specify them as strings; if you want to execute tests on different platforms, specify a dictionary of commands and Python validation snippets. The values of these parameters can be Jinja2 expressions (see [](validate-multi-platform) for more details).
 
-The **config** parameter can be a string (the template to deploy) or a dictionary with two parameters:
+The **config** parameter can be a string (the template to deploy) or a dictionary with these parameters:
 
-* **template**: the template to deploy
+* **template**: the template to deploy, or
+* **inline**: the [configuration change that has to be applied](plugin-files-validation-config) (needs **[files](plugin-files)** plugin to work)
 * **variable**: a dictionary of variable values that will be passed to the Ansible playbook as external variables. You can use these variables to influence the functionality of the configuration template ([example](validate-config))
 
 **Notes:**

netsim/defaults/attributes.yml

@@ -92,7 +92,12 @@ node:
   device: device
   box: str
   id: { type: int, min_value: 1, max_value: 150 }
-  config: list
+  config:
+    type: dict
+    _subtype:
+      type: error
+      _err_msg: Enable "files" plugin to use the inline node/group configuration template(s)
+    _alt_types: [ list ]
   group: list
   role: { type: str, valid_values: [ router, host, bridge, gateway ] }
   mgmt:
@@ -183,6 +188,9 @@ _v_entry:                   # Validation entry
   config:
     template: str
     variable: dict
+    inline:
+      type: error
+      _err_msg: Enable "files" plugin to use the "inline" configuration template
     _alt_types: [ str ]
   valid: _v_option
   suzieq:

netsim/extra/files/plugin.py

@@ -62,18 +62,73 @@ def build_files(cfg: Box, path: str) -> None:       # Recursively transform conf
 
   topology.pop('configlets',None)                     # We no longer need this
 
+"""
+Change 'validation._test_.config.inline' validation elements into templates
+"""
+def inline_validation_templates(topology: Box) -> bool:
+  found_inline = False
+  if not isinstance(topology.validate,Box):           # We're in pre-validation phase, so we can't trust anything
+    return False
+  for t_name,test in topology.validate.items():       # Iterate over tests
+    i_value = test.get('config.inline',None)          # Does this test have 'config.inline' element?
+    if i_value is None:                               # Nope, we're good
+      continue
+    v_template = f'_v_{t_name}'                       # Configlet name is derived from test name
+    topology.configlets[v_template] = i_value         # Save the template value, the next phase will take it from here
+    test.config.pop('inline',None)                    # Pop the 'inline' element from test config attribute
+    test.config.template = v_template                 # ... and replace it with a configlet template
+    found_inline = True
+
+  return found_inline
+
+"""
+Process 'config.inline' node- or group objects
+"""
+def process_inline_config(topology: Box, o_type: str) -> None:
+  if o_type not in topology:                          # The object type is not in current topology
+    return                                            # --> nothing to do
+  if not isinstance(topology[o_type],Box):            # ... or it's not a Box. Let someone else deal with that
+    return
+
+  for o_name,o_data in topology[o_type].items():      # We got something, iterate over it
+    if not isinstance(o_data,Box):                    # The data is not a Box, nothing to do, move on
+      continue
+    if not 'config' in o_data:                        # The object does not have a 'config' element, move on
+      continue
+    if not isinstance(o_data.config,Box):             # Or maybe it's a traditional 'config' element
+      continue
+    c_templates = []                                  # The fun part starts ;)
+    c_inline = '_' + o_type[0] + '_' + o_name         # Build the name for 'inline' template
+    for c_n,c_v in o_data.config.items():             # Now change the config keys/values into templates
+      if c_n == 'inline':                             # Change 'inline' into object-specific template name
+        c_n = c_inline
+      if c_n in topology.configlets:
+        log.error(
+          f'{o_type} {o_name} tries to define config template {c_n} that already exists in configlets',
+          category=log.IncorrectAttr,
+          module='files')                             # Overlap in naming
+      else:
+        topology.configlets[c_n] = c_v                # Add node- or group-defined configlet
+      c_templates.append(c_n)                         # ... and remember what templates the node/group uses
+
+    o_data.config = c_templates                       # Finally, make node/group config element a list of templates
+
 """
 Restructure the 'files' and 'configlets'
 """
 def init(topology: Box) -> None:
   output_hook = False
+  process_inline_config(topology,'nodes')
+  process_inline_config(topology,'groups')
+  if 'validate' in topology:
+    if inline_validation_templates(topology):
+      output_hook = True
   if 'files' in topology:
     restructure_files(topology)
     output_hook = True
   if 'configlets' in topology:
     restructure_configlets(topology)
     output_hook = True
-
   if output_hook:
     append_to_list(topology.defaults.netlab.create,'output','files')
 

tests/errors/group-invalid-node-config.log

@@ -1,3 +1,4 @@
-IncorrectType in nodes: attribute 'nodes.a.config' must be a scalar or a list, found dictionary
-... use 'netlab show attributes node' to display valid attributes
+IncorrectAttr in nodes: Incorrect node attribute 'x' in nodes.a.config
+... Enable "files" plugin to use the inline node/group configuration template(s)
+IncorrectAttr in nodes: Incorrect node attribute 'y' in nodes.a.config
 Fatal error in netlab: Cannot proceed beyond this point due to errors, exiting

tests/integration/ospf/ospfv2/21-default-policy.yml

@@ -6,6 +6,7 @@ message: |
 
 defaults.sources.extra: [ ../../wait_times.yml, ../../warnings.yml ]
 module: [ ospf ]
+plugin: [ files ]
 
 groups:
   probes:
@@ -62,8 +63,10 @@ validate:
   bgp_lbd:
     description: Shut down the loopback interface
     config:
-      template: bgp_loopback
-      variable.lb_state: down
+      inline:
+        frr: |
+          #!/bin/bash
+          ip link set lo down
     pass: BGP prefix is no longer announced
     nodes: [ xf ]
   df_c:
@@ -75,7 +78,9 @@ validate:
   bgp_lbe:
     description: Enable the loopback interface
     config:
-      template: bgp_loopback
-      variable.lb_state: up
+      inline:
+        frr: |
+          #!/bin/bash
+          ip link set lo up
     pass: BGP prefix should be announced
     nodes: [ xf ]

tests/integration/ospf/ospfv2/bgp_loopback.j2

@@ -8,6 +8,8 @@ message: |
 
 defaults.sources.extra: [ ../wait_times.yml, ../warnings.yml ]
 
+plugin: [ files ]
+
 groups:
   probes:
     members: [ x1, x2 ]
@@ -25,11 +27,25 @@ nodes:
   dut:
     bgp.as: 65000
     module: [ vrf, bgp ]
+
   x1:
     id: 10
     bgp.as: 65010
     bgp.originate: 172.0.42.0/24
-    config: [ frr-community ]
+    config.inline: |
+      route-map setcomm permit 10
+      set community 65000:1 additive
+      set extcommunity bandwidth 100
+      set large-community 65000:0:1 additive
+      exit
+      !
+      router bgp {{ bgp.as }}
+      !
+      address-family ipv4 unicast
+      {% for n in bgp.neighbors %}
+        neighbor {{ n.ipv4 }} route-map setcomm out
+      {% endfor %}
+
   x2:
     id: 11
     bgp.as: 65011