Merge pull request #14 from cisco-open/dev-2.0.10

Dev 2.0.10

Author: tzarski0

.github/workflows/test.yml

@@ -39,6 +39,7 @@ jobs:
           - '3.9'
           - '3.10'
           - '3.11'
+          - '3.12'
     steps:
       - name: Checkout
         uses: actions/checkout@v4

CHANGELOG.md

@@ -1,3 +1,12 @@
+# Catalyst SD-WAN Lab 2.0.10 [May 10, 2024]
+
+- Added support for Python 3.12
+- In setup task, added functionality to covert refplat ISO SD-WAN images to proper format so script can use them
+- In setup task, added --list parameter to list all SD-WAN software images available on CML server
+- Fixed problem where setup task might fail with "Cannot modify read-only node definition"
+- Fixed problem where add task could fail with undescriptive error "IndexError: list index out of range"
+- Updated README with installation guide for Windows
+
 # Catalyst SD-WAN Lab 2.0.9 [Apr 26, 2024]
 
 #### Fixes:

README.md

@@ -1,16 +1,23 @@
 [![Tests](https://github.com/cisco-open/sdwan-lab-deployment-tool/actions/workflows/test.yml/badge.svg)](https://github.com/netascode/iac-validate/actions/workflows/test.yml)
-![Python Support](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11-informational "Python Support: 3.9, 3.10, 3.11")
+![Python Support](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-informational "Python Support: 3.9, 3.10, 3.11, 3.12")
 
 # Catalyst SD-WAN Lab Deployment Tool for Cisco Modeling Labs
 
 This tool automates [Cisco Catalyst SD-WAN](https://www.cisco.com/site/us/en/solutions/networking/sdwan/index.html) lab deployment inside [Cisco Modeling Labs (CML)](https://www.cisco.com/c/en/us/products/cloud-systems-management/modeling-labs/index.html).
 
+The tool will help you automate your CML lab deployments with SD-WAN Manager, Controllers and Validators and up to 20 SD-WAN edges. You can build as pods as your CML platform can host. Please refer to the [Limitations and scale](#limitations-and-scale) for details.
+
+
+
 ## Getting Started
 
 ### Prerequisites
-Catalyst SD-WAN Lab tool requires Python 3.8 or newer. This can be verified by pasting the following to a terminal window:
+Catalyst SD-WAN Lab Deployment Tool requires Linux or macOS system. 
+To run is on Windows, please use [Linux on Windows with WSL](/README.md#appendix---wsl-installation) or set up Linux VM/container.
 
-    % python3 -c "import sys;assert sys.version_info>(3,8)" && echo "ALL GOOD"
+Catalyst SD-WAN Lab Deployment Tool requires Python 3.9 or newer. This can be verified by pasting the following to a terminal window:
+
+    % python3 -c "import sys;assert sys.version_info>(3,9)" && echo "ALL GOOD"
 
 If 'ALL GOOD' is printed it means Python requirements are met. If not, download and install the latest 3.x version at Python.org (https://www.python.org/downloads/).
 
@@ -54,7 +61,7 @@ You can also use the following shortcut to run any lab task:
 
 Notes:
 - The virtual environment is deactivated by typing 'deactivate' at the command prompt.
-- Before running SD-WAN Lab again, make sure to activate the virtual environment back again (source venv/bin/activate).
+- Before running Catalyst SD-WAN Lab Deployment Tool again, make sure to activate the virtual environment back again (source venv/bin/activate).
 
 ## Usage
 Simmilar to [Sastre](https://github.com/CiscoDevNet/sastre), the command line is structured as a set of base parameters, the task specification followed by task-specific parameters:
@@ -302,6 +309,37 @@ This task has several task-specific parameters.
         --lab <lab_name>  Lab name
         --force           Delete the lab without asking for confirmation. Note the all lab data will be lost!
 
+## Limitations and scale
+The tool supports the following scale per CML lab:
+
+- 1 SD-wan Manager instance (Cluster is not yet supported)
+- 8 SD-WAN Validators (Documented support from CCO)
+- 12 SD-WAN Controllers (Documented support from CCO)
+- 20 SD-WAN Edges
+- 10 SD-Routing edges
+
+
+
+## Appendix - WSL Installation
+
+To install WSL on your Windows VM or Physical machine. Ensure that the HW Virutalization is enabled in the BIOS or VM Defintion.
+
+If its on Windows server you may need to run this command to allow the WSL to function properly
+
+Open PowerShell as Administrator and run:
+
+`Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux`
+
+Install WSL with default distribution (Ubuntu)
+Open PowerShell and run
+
+ `wsl --install`
+
+Once the installation is finished and you have restarted Windows you are able to continue the installation of this tool as described in the [installation section](README.md#installing) of this document.
+
+You can read more about [Linux on Windows with WSL here](https://learn.microsoft.com/en-us/windows/wsl/install).
+
+
 ## Authors
 Tomasz Zarski (tzarski@cisco.com)
 
@@ -311,4 +349,4 @@ BSD-3-Clause
 
 ## Acknowledgments
 - Marcelo Reis and [Sastre](https://github.com/CiscoDevNet/sastre)
-- Inigo Alonso
+- Inigo Alonso

catalyst_sdwan_lab/__main__.py

@@ -101,14 +101,12 @@ def main() -> None:
         "setup", help="Setup on-prem CML to use Catalyst SD-WAN Lab automation."
     )
     setup_parser.add_argument(
-        "--migrate",
+        "--list",
         action="store_const",
-        dest="migrate",
+        dest="list",
         const=True,
         default=False,
-        help="Migrate node and image definitions from SD-WAN Lab v1.x to v2.x. "
-        "This task should be run once if CML server was using "
-        "SD-WAN LAb Tool v1.x in the past.",
+        help="After running setup task, list the available SD-WAN software per node type.",
     )
 
     deploy_parser = task_subparsers.add_parser(
@@ -522,7 +520,7 @@ def main() -> None:
     )
     verify_cml_version(cml)
     if cli_args.task == "setup":
-        setup.main(cml, cli_args.loglevel, cli_args.migrate)
+        setup.main(cml, cli_args.loglevel, cli_args.list)
     elif cli_args.task == "deploy":
         deploy.main(
             cml,

catalyst_sdwan_lab/tasks/add.py

@@ -357,6 +357,12 @@ def main(
                 )
             ]
 
+            if number_of_devices > len(free_uuids):
+                exit(
+                    f"Cannot onboard {number_of_devices} WAN Edges as there are only "
+                    f"{len(free_uuids)} unused UUIDs available."
+                )
+
             new_nodes_nums = []
             for i in range(1, number_of_devices + 1):
                 new_nodes_nums.append(f"{biggest_num + i}")
@@ -602,6 +608,12 @@ def main(
                 )
             ]
 
+            if number_of_devices > len(free_uuids):
+                exit(
+                    f"Cannot onboard {number_of_devices} WAN Edges as there are only "
+                    f"{len(free_uuids)} unused UUIDs available."
+                )
+
             uuid_to_token = {
                 device.uuid: device.serial_number
                 for device in device_list.filter(

catalyst_sdwan_lab/tasks/setup.py

@@ -11,7 +11,6 @@
 from os.path import join
 from typing import List, Union
 
-from httpx import HTTPStatusError
 from ruamel.yaml import YAML
 from virl2_client import ClientLibrary
 
@@ -26,14 +25,14 @@
 def upload_image_and_create_definition(
     log: Logger,
     cml: ClientLibrary,
-    existing_image_definitions: List[str],
+    existing_image_definitions_ids: List[str],
     node_type: str,
     software_version: str,
     node_label: str,
     software_images_dir: str,
     filename: str,
 ) -> None:
-    if f"{node_type}-{software_version}" in existing_image_definitions:
+    if f"{node_type}-{software_version}" in existing_image_definitions_ids:
         log.info(
             f"Skipping {filename} as {node_type}-{software_version} image definition already exists."
         )
@@ -49,7 +48,7 @@ def upload_image_and_create_definition(
         cml.definitions.upload_image_definition(body=json.dumps(image_def))
 
 
-def main(cml: ClientLibrary, loglevel: Union[int, str], migrate: bool) -> None:
+def main(cml: ClientLibrary, loglevel: Union[int, str], list: bool) -> None:
     # Setup logging
     log = setup_logging(loglevel)
 
@@ -85,6 +84,13 @@ def main(cml: ClientLibrary, loglevel: Union[int, str], migrate: bool) -> None:
                     )
                 else:
                     # If dicts are not the same, then we need to update the node definition
+                    # Before update, check if node definition is read only
+                    if current_node_definition["general"]["read_only"]:
+                        # Remove read-only flag
+                        cml.definitions.set_node_definition_read_only(
+                            current_node_definition["id"], False
+                        )
+
                     log.info(
                         f'[UPDATE] Updating node {new_node_definition["id"]} with '
                         f'{new_node_definition["sim"]["linux_native"]["cpus"]} CPUs and '
@@ -102,22 +108,46 @@ def main(cml: ClientLibrary, loglevel: Union[int, str], migrate: bool) -> None:
                 # If node is not yet created, we need to create it
                 log.info(f'[CREATE] Creating node {new_node_definition["id"]}...')
                 cml.definitions.upload_node_definition(new_node_definition, json=True)
+
+    # Refresh node definitions
+    node_definitions = cml.definitions.node_definitions()
     track_progress(log, "Verifying software images...")
     # Get the list of all image definitions already created in CML.
     # This is to avoid image duplication during upload.
-    existing_image_definitions = [
-        image_definition["id"]
-        for image_definition in cml.definitions.image_definitions()
+    existing_image_definitions = cml.definitions.image_definitions()
+    existing_image_definitions_ids = [
+        image_definition["id"] for image_definition in existing_image_definitions
     ]
+    for image_definition in existing_image_definitions:
+        match = re.match(
+            r"^cat-sdwan-(edge|controller|validator|manager)-([\w\-]+)$",
+            image_definition["id"],
+        )
+        if match:
+            # Migrate image from using - in software version to using .
+            # For example from cat-sdwan-manager-20-13-1 to cat-sdwan-manager-20.13.1
+            # Before update, check if node definition is read only
+            if image_definition["read_only"]:
+                # Remove read-only flag
+                cml.definitions.set_image_definition_read_only(
+                    image_definition["id"], False
+                )
+            cml.definitions.remove_image_definition(image_definition["id"])
+            # Set new ID and disk folder
+            image_definition["id"] = (
+                f"cat-sdwan-{match.group(1)}-{match.group(2).replace('-', '.')}"
+            )
+            image_definition["disk_subfolder"] = image_definition["id"]
+            cml.definitions.upload_image_definition(image_definition)
+
     log.info(f"Looking for new software images in {os.getcwd()}...")
     software_type_to_node_type_mapping = {
         "edge": "cat-sdwan-validator",
         "bond": "cat-sdwan-validator",
         "smart": "cat-sdwan-controller",
         "vmanage": "cat-sdwan-manager",
     }
-    # Refresh node definitions
-    node_definitions = cml.definitions.node_definitions()
+
     # Check for any software that is present in software_images folder.
     for filename in os.listdir(SOFTWARE_IMAGES_DIR):
         if software_parser := re.match(r"viptela-(\w+)-([\d.]+)-", filename):
@@ -132,7 +162,7 @@ def main(cml: ClientLibrary, loglevel: Union[int, str], migrate: bool) -> None:
             upload_image_and_create_definition(
                 log,
                 cml,
-                existing_image_definitions,
+                existing_image_definitions_ids,
                 node_type,
                 software_version,
                 node_label,
@@ -154,7 +184,7 @@ def main(cml: ClientLibrary, loglevel: Union[int, str], migrate: bool) -> None:
             upload_image_and_create_definition(
                 log,
                 cml,
-                existing_image_definitions,
+                existing_image_definitions_ids,
                 node_type,
                 software_version,
                 node_label,
@@ -165,71 +195,22 @@ def main(cml: ClientLibrary, loglevel: Union[int, str], migrate: bool) -> None:
         else:
             log.debug(f"Skipping file {filename} (not a valid image).")
 
-    if migrate:
-        node_definition_map = {
-            "vmanage": "cat-sdwan-manager",
-            "vsmart": "cat-sdwan-controller",
-            "vedge": "cat-sdwan-validator",
-            "cedge": "cat-sdwan-edge",
-        }
-        node_label_map = {
-            "vmanage": "Catalyst SD-WAN Manager",
-            "vsmart": "Catalyst SD-WAN Controller",
-            "vedge": "Catalyst SD-WAN Validator",
-            "cedge": "Catalyst SD-WAN Edge",
-        }
-        track_progress(log, "Running migration of node/image definitions...")
-        # Migrate SD-WAN Lab Tool v1.x image definitions to v2.x
-        all_migrated = True
-        for image_definition in cml.definitions.image_definitions():
-            current_id = image_definition["id"]
-            current_node_definition = image_definition["node_definition_id"]
-            if current_node_definition in ["vmanage", "vsmart", "vedge", "cedge"]:
-                try:
-                    software_version = image_definition["id"].split("-")[1]
-                    new_image_definition = {
-                        "node_definition_id": node_definition_map[
-                            current_node_definition
-                        ],
-                        "id": f"{node_definition_map[current_node_definition]}-{software_version}",
-                        "read_only": False,
-                        "label": f"{node_label_map[current_node_definition]} {software_version}",
-                        "disk_image": image_definition["disk_image"],
-                    }
-
-                    cml.definitions.remove_image_definition(current_id)
-                    cml.definitions.upload_image_definition(new_image_definition)
-                    log.info(
-                        f"Migrated image definition {current_id} to "
-                        f"{node_definition_map[current_node_definition]}-{software_version}"
-                    )
-
-                except HTTPStatusError as e:
-                    all_migrated = False
-                    error = e.response.json()
-                    if error["code"] == 400 and error["description"].startswith(
-                        "Image Definition in use:"
-                    ):
-                        log.info(
-                            f"Cannot migrate image definition {current_id} as it is currently in use."
-                        )
-
-        for node_definition in cml.definitions.node_definitions():
-            node_id = node_definition["id"]
-            if node_id in ["vmanage", "vsmart", "vedge", "cedge"]:
-                if cml.definitions.image_definitions_for_node_definition(node_id):
-                    log.info(
-                        f"Cannot delete node definition {node_id} as it is currently in use."
-                    )
-                    all_migrated = False
-                else:
-                    cml.definitions.remove_node_definition(node_id)
-                    log.info(f"Node definition {node_id} successfully removed.")
-
-        if not all_migrated:
-            print(
-                "\rSome node/image definitions could not be migrated as they are in use. "
-                "Please remove the labs using the old definitions are rerun the setup with --migrate option."
-            )
-
     track_progress(log, "Setup task done\n")
+
+    if list:
+        print("Available Software Versions:")
+        for node_definition_id in [
+            "cat-sdwan-manager",
+            "cat-sdwan-controller",
+            "cat-sdwan-validator",
+            "cat-sdwan-edge",
+        ]:
+            available_software_versions = []
+            # List available SD-WAN software
+            for (
+                image_definition
+            ) in cml.definitions.image_definitions_for_node_definition(
+                node_definition_id
+            ):
+                available_software_versions.append(image_definition["id"].split("-")[3])
+            print(f"- {node_definition_id}: {available_software_versions}")