Add version change for restore task (#92)

* update gitignore

* add version change for restore task (fix #38)

* update readme

* Increase sd-wan manager cpus to 10

* update gitignore

* update delete task log

* update gitignore

* add pytest test

* update tests

* update contribution guidelines

* update contributing guidelines

Author: tzarski0

.gitignore

@@ -195,4 +195,6 @@ cython_debug/
 # End of https://www.toptal.com/developers/gitignore/api/macos,python
 
 catalyst_sdwan_lab/data/certs/rootCA.key
-vsdwan.sh
+vsdwan.sh
+lab.env
+backup/

CHANGELOG.md

@@ -4,6 +4,8 @@
 - Fix issue where add task might fail for SD-WAN Manager 20.18 and higher
 - In setup task, change behaviour when --list is specified, print software version and exit
 - Update validator cloud-init to support 20.18 release
+- In restore task, add option to change the software version of control components and SD-WAN/SD-Routing edges during restore
+- Increase SD-WAN Manager CPU count from 8 to 10
 
 # Catalyst SD-WAN Lab 2.0.15 [Feb 25, 2025]
 

CONTRIBUTING.md

@@ -1,11 +1,62 @@
 # How to Contribute
 
-Thanks for your interest in contributing to Catalyst SD-WAN Lab! Please take a moment to review this document **before submitting a pull request**:
-  * Want to add something from yourself? [Make a PR](https://github.com/cisco-open/sdwan-lab-deployment-tool/pulls).
-    * To make a PR - fork the repository, make your changes and make the pull request. Now just wait for the review and feedback from our developers.
-    * Make clear PR description and include doc strings in your code to make it easily understandable.
-    * Always write a clear log message for your commits.
-    * Before submitting PR, verify the existings SD-WAN Lab tasks works without any issues.
-    * For CML interactions, use official [CML SDK](https://github.com/CiscoDevNet/virl2-client)
-    * For SD-WAN interactions, use official [Catalyst WAN SDK](https://github.com/CiscoDevNet/catalystwan)
-  * Bugs or feature requests? [Report it here](https://github.com/cisco-open/sdwan-lab-deployment-tool/issues) - remember to provide as much information as you can.
+Thank you for your interest in contributing to Catalyst SD-WAN Lab! Please take a moment to review this document **before submitting a pull request**:
+
+- Want to add something yourself? [Make a PR](https://github.com/cisco-open/sdwan-lab-deployment-tool/pulls).
+  - To make a PR, fork the repository, make your changes, and submit the pull request. Then, wait for review and feedback from our developers.
+  - Write a clear PR description and include docstrings in your code to make it easily understandable.
+  - Always write clear log messages for your commits.
+  - Before submitting a PR, verify that the existing SD-WAN Lab tasks work without any issues (see [Testing Before Pull Requests](#testing-before-pull-requests)).
+  - For CML interactions, use the official [CML SDK](https://github.com/CiscoDevNet/virl2-client).
+  - For SD-WAN interactions, use the official [Catalyst WAN SDK](https://github.com/CiscoDevNet/catalystwan).
+- Found a bug or have a feature request? [Report it here](https://github.com/cisco-open/sdwan-lab-deployment-tool/issues) and provide as much information as possible.
+
+## Testing Before Pull Requests
+
+Before opening a pull request, please ensure all tests pass:
+
+1. Create a virtual environment:
+
+   ```
+   python3 -m venv venv
+   ```
+
+2. Activate the virtual environment:
+
+   ```
+   source venv/bin/activate
+   ```
+
+   The prompt will update with the virtual environment name (`venv`), indicating that it is active.
+
+3. Upgrade initial virtual environment packages:
+
+   ```
+   pip install --upgrade pip setuptools pytest
+   ```
+
+4. Install the tool from your local repository:
+
+   ```
+   pip install --upgrade .
+   ```
+
+   Alternatively, you can install the tool from your branch, for example:
+
+   ```
+   pip install git+https://github.com/cisco-open/sdwan-lab-deployment-tool.git@restore_version_change
+   ```
+
+5. Load the environment variables used to deploy a test lab:
+
+   ```
+   source lab.env
+   ```
+
+6. Run tests:
+
+   ```
+   pytest -s test_csdwan.py
+   ```
+
+   This will run all tasks and display real-time output. Make sure to resolve any test failures before submitting your PR. If needed, you can modify the control components and edge versions in the test_csdwan.py file.

README.md

@@ -303,26 +303,43 @@ This task restores Catalyst SD-WAN lab from a backup. This task will:
 3. Restore SD-WAN Manager templates, policies and configuration groups using [Sastre](https://github.com/CiscoDevNet/sastre).
 4. Verify if there are any WAN Edges in the topology (SD-WAN and SD-Routing). If yes, then generate the new OTP and automatically reonboard them to SD-WAN Manager.
 
-This task has several task-specific parameters, including working directory from where backup is restored.
+This task has several task-specific parameters, including working directory from where backup is restored. You can also overwrite the software version for control components and SD-WAN/SD-Routing Edges (note that specifying version lower than the one in the backup is not supported).
 
       sdwan-lab restore -h
 
        Usage: sdwan-lab restore [OPTIONS]
 
-      ╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-      │ --manager             <manager-ip>        SD-WAN Manager IP address, can also be defined via MANAGER_IP environment                                                                        │
-      │ --muser               <manager-user>      SD-WAN Manager username, can also be defined via MANAGER_USER environment variable.                                                              │
-      │ --mpassword           <manager-password>  SD-WAN Manager password, can also be defined via MANAGER_PASSWORD environment variable.                                                          │
-      │ --mmask               <manager-mask>      Subnet mask for given SD-WAN Manager IP (e.g. /24), can also be defined via MANAGER_MASK environment variable.                                   │
-      │ --mgateway            <manager-gateway>   Gateway IP for given SD-WAN Manager IP, can also be defined via MANAGER_GATEWAY environment variable.                                            │
-      │ --lab                 <lab_name>          CML Lab name, can also be defined via LAB_NAME environment variable.                                                                             │
-      │ --workdir             <directory>         Restore source folder                                                                                                                            │
-      │ --deleteexisting                          If there is already lab running with same name and using same SD-WAN Manager IP, delete this lab before restoring. Note the all running lab data │
-      │                                           will be lost!                                                                                                                                    │
-      │ --retry                                   If for some reason your script lost connectivity during SD-WAN Manager boot, you can add --retry to continue onboarding the lab that is already  │
-      │                                           in CML                                                                                                                                           │
-      │ --help            -h                      Show this message and exit.                                                                                                                      │
-      ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+      ╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────╮
+      │ --manager             <manager-ip>        SD-WAN Manager IP address, can also be defined via      │
+      │                                           MANAGER_IP environment                                  │
+      │ --muser               <manager-user>      SD-WAN Manager username, can also be defined via        │
+      │                                           MANAGER_USER environment variable.                      │
+      │ --mpassword           <manager-password>  SD-WAN Manager password, can also be defined via        │
+      │                                           MANAGER_PASSWORD environment variable.                  │
+      │ --mmask               <manager-mask>      Subnet mask for given SD-WAN Manager IP (e.g. /24), can │
+      │                                           also be defined via MANAGER_MASK environment variable.  │
+      │ --mgateway            <manager-gateway>   Gateway IP for given SD-WAN Manager IP, can also be     │
+      │                                           defined via MANAGER_GATEWAY environment variable.       │
+      │ --lab                 <lab_name>          CML Lab name, can also be defined via LAB_NAME          │
+      │                                           environment variable.                                   │
+      │ --workdir             <directory>         Restore source folder                                   │
+      │ --deleteexisting                          If there is already lab running with same name and      │
+      │                                           using same SD-WAN Manager IP, delete this lab before    │
+      │                                           restoring. Note the all running lab data will be lost!  │
+      │ --retry                                   If for some reason your script lost connectivity during │
+      │                                           SD-WAN Manager boot, you can add --retry to continue    │
+      │                                           onboarding the lab that is already in CML               │
+      │ --contr_version       <contr_version>     Change the controller version when restoring the lab.   │
+      │ --edge_version        <edge_version>      Change the SD-WAN edge version when restoring the lab.  │
+      │ --help            -h                      Show this message and exit.                             │
+      ╰───────────────────────────────────────────────────────────────────────────────────────────────────╯
+
+Below you will find few examples of restore task:
+
+    sdwan-lab restore --workdir backup
+    sdwan-lab restore --workdir backup --deleteexisting
+    sdwan-lab restore --workdir backup --contr_version 20.16.1
+    sdwan-lab restore --workdir backup --contr_version 20.16.1 --edge_version 17.16.01a
 
 ### Delete Task
 
@@ -391,10 +408,11 @@ Once the installation is finished and you have restarted Windows you are able to
 You can read more about [Linux on Windows with WSL here](https://learn.microsoft.com/en-us/windows/wsl/install).
 
 ## FAQ
+
 Q1: My devices' consoles have stopped working after I created my own configuration groups. How do I recover console access?
 
 A1: Always include the `platform console serial` command in an CLI add-on feature parcel to ensure your consoles work from the CML UI. Note that after adding this command, a WAN Edge reboot is required.
- 
+
 Q2: Can I SSH to my Manager instance directly?
 
 A2: Yes, you can if you are using an external IP. However, if you are using PATty, you cannot, as we only map the HTTPS port.

catalyst_sdwan_lab/cli.py

@@ -430,6 +430,20 @@ def cli_backup(
     "during SD-WAN Manager boot, you can add --retry to continue "
     "onboarding the lab that is already in CML",
 )
+@click.option(
+    "--contr_version",
+    metavar="<contr_version>",
+    default=None,
+    help="Change the controller version when restoring the lab. Note the specified "
+    "version cannot be older than the one used in the backup.",
+)
+@click.option(
+    "--edge_version",
+    metavar="<edge_version>",
+    default=None,
+    help="Change the SD-WAN/SD-Routing edge version when restoring the lab. Note the specified "
+    "version cannot be older than the one used in the backup.",
+)
 @click.pass_context
 def cli_restore(
     ctx: click.Context,
@@ -442,6 +456,8 @@ def cli_restore(
     workdir: str,
     deleteexisting: bool,
     retry: bool,
+    contr_version: str,
+    edge_version: str,
 ) -> None:
     cml_config = ctx.obj["CML_CONFIG"]
     cml_ip = cml_config.url
@@ -468,6 +484,8 @@ def cli_restore(
         deleteexisting,
         retry,
         loglevel,
+        contr_version,
+        edge_version,
     )
 
 

catalyst_sdwan_lab/data/cml_nodes_definition/cat-sdwan-manager.yaml

@@ -70,7 +70,7 @@ boot:
     - ': System Ready'
 sim:
   linux_native:
-    cpus: 8
+    cpus: 10
     disk_driver: ide
     driver: server
     libvirt_domain_driver: kvm

catalyst_sdwan_lab/tasks/delete.py

@@ -50,7 +50,7 @@ def main(
         lab.wipe()
         log.info("Removing the lab...")
         lab.remove()
-        track_progress(log, "Delete task done")
+        track_progress(log, "Delete task done\n")
 
     else:
         exit("Could not find a lab with specified name.")

catalyst_sdwan_lab/tasks/restore.py

@@ -50,6 +50,8 @@ def main(
     deleteexisitng: bool,
     retry: bool,
     loglevel: Union[int, str],
+    contr_version: str,
+    edge_version: str,
 ) -> None:
     # Time the script execution
     begin_time = datetime.datetime.now()
@@ -71,14 +73,69 @@ def main(
 
     cml_topology_dict = yaml.load(cml_topology)
 
-    # Verify if requested software version is defined in CML
     track_progress(log, "Preparing for restore...")
     log.info("Checking software version...")
+    if contr_version:
+        # Check if the software version requested by user is compatible with the backup
+        # In particular, check if the requested version is not older than the backup version
+        backup_version = next(
+            node["image_definition"].split("-")[-1]
+            for node in cml_topology_dict["nodes"]
+            if node["node_definition"] == "cat-sdwan-manager"
+        )
+        backup_version_split = backup_version.split(".")
+        contr_version_split = contr_version.split(".")
+        if int(contr_version_split[0]) < int(backup_version_split[0]) or (
+            int(contr_version_split[0]) == int(backup_version_split[0])
+            and int(contr_version_split[1]) < int(backup_version_split[1])
+        ):
+            exit(
+                f"\nDowngrade of image version from {backup_version} to {contr_version} is not supported."
+            )
+    if edge_version:
+        # Check if the software version requested by user is compatible with the backup
+        # In particular, check if the requested version is not older than the backup version
+        backup_version = next(
+            node["image_definition"].split("-")[-1]
+            for node in cml_topology_dict["nodes"]
+            if node["node_definition"] == "cat-sdwan-edge"
+        )
+        backup_version_split = backup_version.split(".")
+        edge_version_split = edge_version.split(".")
+        if int(edge_version_split[0]) < int(backup_version_split[0]) or (
+            int(edge_version_split[0]) == int(backup_version_split[0])
+            and int(edge_version_split[1]) < int(backup_version_split[1])
+        ):
+            exit(
+                f"\nDowngrade of image version from {backup_version} to {edge_version} is not supported."
+            )
+
+    # Verify if requested software version is defined in CML
     cml_image_definitions = cml.definitions.image_definitions()
     defined_images = set(image["id"] for image in cml_image_definitions)
     missing_images = []
     for node in cml_topology_dict["nodes"]:
-        if node["image_definition"] and node["image_definition"] not in defined_images:
+        if (
+            node["node_definition"]
+            in ["cat-sdwan-controller", "cat-sdwan-manager", "cat-sdwan-validator"]
+            and contr_version
+        ):
+            if (
+                f'{node["node_definition"]}-{contr_version}' not in defined_images
+                and f'{node["node_definition"]}-{contr_version}' not in missing_images
+            ):
+                missing_images.append(f'{node["node_definition"]}-{contr_version}')
+        elif node["node_definition"] == "cat-sdwan-edge" and edge_version:
+            if (
+                f'{node["node_definition"]}-{edge_version}' not in defined_images
+                and f'{node["node_definition"]}-{edge_version}' not in missing_images
+            ):
+                missing_images.append(f'{node["node_definition"]}-{edge_version}')
+        elif (
+            node["image_definition"]
+            and node["image_definition"] not in defined_images
+            and node["image_definition"] not in missing_images
+        ):
             missing_images.append(node["image_definition"])
     if missing_images:
         sys.exit(
@@ -178,6 +235,24 @@ def main(
         if patty_used:
             manager_node["tags"] = [f"pat:{manager_port}:443"]
 
+        if contr_version:
+            # Overwrite the image definition for control components
+            for node in cml_topology_dict["nodes"]:
+                if node["node_definition"] in [
+                    "cat-sdwan-controller",
+                    "cat-sdwan-manager",
+                    "cat-sdwan-validator",
+                ]:
+                    node["image_definition"] = (
+                        f'{node["node_definition"]}-{contr_version}'
+                    )
+        if edge_version:
+            # Overwrite the image definition for control components
+            for node in cml_topology_dict["nodes"]:
+                if node["node_definition"] == "cat-sdwan-edge":
+                    node["image_definition"] = (
+                        f'{node["node_definition"]}-{edge_version}'
+                    )
         stream = io.StringIO()
         yaml.dump(cml_topology_dict, stream)
         track_progress(log, "Importing the lab...")

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "catalyst-sdwan-lab"
-version = "2.0.16dev3"
+version = "2.0.16dev4"
 description = "Catalyst SD-WAN Lab Deployment Tool - Automation Tool for managing Cisco Catalyst SD-WAN labs inside Cisco Modeling Labs"
 license = "BSD-3-Clause"
 authors = ["Tomasz Zarski <tzarski@cisco.com>"]

test_csdwan.py

@@ -0,0 +1,21 @@
+import os
+import subprocess
+
+def test_csdwan():
+    commands = [
+        ['csdwan', '--version'],
+        ['csdwan', 'setup'],
+        ['csdwan', 'deploy', '20.16.1'],
+        ['csdwan', 'add', '2', 'edges', '17.16.01a'],
+        ['csdwan', 'add', '1', 'validator', '20.16.1'],
+        ['csdwan', 'add', '1', 'controller', '20.16.1'],
+        ['csdwan', 'backup', '--workdir', 'backup'],
+        ['csdwan', 'delete', '--force'],
+        ['csdwan', 'restore', '--workdir', 'backup'],
+        ['csdwan', 'delete', '--force'],
+    ]
+    print()
+    for cmd in commands:
+        print(f"==== {' '.join(cmd)} ====")
+        result = subprocess.run(cmd, env=os.environ)
+        assert result.returncode == 0, f"Command {' '.join(cmd)} failed"