diff --git a/index.rst b/index.rst index e775fce..e150c77 100644 --- a/index.rst +++ b/index.rst @@ -24,6 +24,7 @@ onramp/roc onramp/devel onramp/blueprints + onramp/testing .. toctree:: :maxdepth: 3 @@ -56,17 +57,6 @@ operations/metering operations/monitor -.. toctree:: - :maxdepth: 3 - :caption: Test Automation - :hidden: - :glob: - - testing/about_system_tests - testing/integration_tests - testing/aether-roc-tests - - .. toctree:: :maxdepth: 2 :caption: Releases diff --git a/intro.rst b/intro.rst index 5d11a6b..960ccda 100644 --- a/intro.rst +++ b/intro.rst @@ -34,9 +34,6 @@ Other Aether guides available on this site include: * :doc:`Runtime Operations `: Learn how to operate Aether's 5G connectivity service. -* :doc:`Test Automation `: Learn how Aether - components are tested. - Note that Aether was originally deployed as a centrally-managed cloud service with a dedicated ops team. The expectation was that organizations would participate in Aether by connecting their edge diff --git a/onramp/testing.rst b/onramp/testing.rst new file mode 100644 index 0000000..231a073 --- /dev/null +++ b/onramp/testing.rst @@ -0,0 +1,35 @@ +Integration Tests +-------------------- + +A set of integration tests validate the released :doc:`Aether +Blueprints `. The tests are managed by Jenkins, +and can be monitored using the following `Dashboard +`__. Log files generated by +recent runs are available on that site. + +Source code for the integration tests can be found in the +`aether-jenkins repo +`__ on +GitHub, where each file in the repo corresponds to a Groovy script +that implements a Jenkins pipeline for one of the blueprints. + +The pipelines are executed daily, with each pipeline parameterized to +run in multiple jobs. The ``${AgentLabel}`` parameter selects the +Ubuntu release being tested (currently ``22.04`` and ``24.04``), with +all jobs running in AWS VMs (currently resourced as +``M7iFlex2xlarge``). Pipelines that exercise two-server +tests—currently ``ueransim.groovy``, ``upf.groovy``, +``srsran.groovy``, and ``gnbsim.groovy``\—invoke the `AWS CLI +`__ to create the second VM. + +All VMs created by Jenkins run images prepared using Packer, with the +current specification templates available in the `aether-packer repo +`__ on GitHub. + +Finally, note that the integration tests run a variety of simulated 5G +workloads, including gNBsim, UERANSIM, OAI 5G RAN (in simulation +mode), and srsRAN (in simulation mode). Of these, gNBsim provides the +most rigorous testing of the Core's control plane, and serves as +Aether's primary validation of that functionality. More information +about gNBsim can be found in the :doc:`Emulated RAN +` section of this Guide. diff --git a/testing/about_system_tests.rst b/testing/about_system_tests.rst deleted file mode 100644 index 7600a02..0000000 --- a/testing/about_system_tests.rst +++ /dev/null @@ -1,27 +0,0 @@ -.. - SPDX-FileCopyrightText: © 2021 Open Networking Foundation - SPDX-License-Identifier: Apache-2.0 - -About -===== - -The Aether Project is transitioning to a new automated testing regime, -consistent with its transition from being a managed service to being -a platform anyone can deploy. Information about the previous framework -can be found in the `Version 2.1 Guide -`__. - -The major changes include: - -* Adopting gNBsim (in place of NG40) as the primary test engine for SD-Core. -* Developing new integration tests for a range of configuration options. -* Dropping support for SD-Fabric tests. - -For detailed information on component-level tests, please see their -respective Guides: - -* :doc:`SD-Core Documentation ` -* :doc:`SD-RAN Documentation ` - -More information about interface testing is include in the :doc:`API -Tests Section ` of this Guide. diff --git a/testing/aether-roc-tests.rst b/testing/aether-roc-tests.rst deleted file mode 100644 index b98f305..0000000 --- a/testing/aether-roc-tests.rst +++ /dev/null @@ -1,220 +0,0 @@ -API Tests -=========== - -The REST API and GUI of Aether are tested with the Robot Framework. -The tests are located in the ``aether-system-tests`` repository on -Gerrit, but they are **not** currently activated on Jenkins. This -section is archived with the goal of reactivating them. - -Development Prerequisites -------------------------- - -To access the ROC from a local system, it is necessary to deploy the -components of µONOS. This can be done with the use of Helm (see -instructions on `this page -`_). - -Additionally, it is necessary to add the SD-RAN chart repo with the -following command: - -.. code-block:: shell - - helm repo add sdran https://charts.aetherproject.org - -Finally, the ROC GUI tests are running on the Firefox browser, so it -is necessary to have the Firefox browser and the Firefox web driver -(``geckodriver``) installed on the system in order to run these tests. - -Running the ROC API Tests -------------------------- - -Follow the steps below to access the ROC API: - -1. Deploy the ``aether-roc-umbrella`` chart from the Aether repo with - the following command: - -.. code-block:: shell - - helm -n micro-onos install aether-roc-umbrella aether/aether-roc-umbrella - -2. Check if all pods are in a Running state: - -.. code-block:: shell - - kubectl -n micro-onos get pods - -3. Once all pods are in a Running state, port-forward aether-roc-api - to port 8181 with the following command: - -.. code-block:: shell - - kubectl -n micro-onos port-forward service/aether-roc-api 8181 & - -4. Port-forward onos-config to port 5150 with the following command: - -.. code-block:: shell - - kubectl -n micro-onos port-forward service/onos-config 5150 & - -Now that we have access to the ROC API, we can proceed with running -the ROC API tests from the ``aether-system-tests`` repository: - -1. Checkout the aether-system-tests repo: - -.. code-block:: shell - - git clone "ssh://$GIT_USER@gerrit.opencord.org:29418/aether-system-tests" - -2. Go to the repo directory: - -.. code-block:: shell - - cd aether-system-tests - -3. Install the requirements and create a virtual environment: - -.. code-block:: shell - - make ast-venv - source ast-venv/bin/activate - -4. Go to the ``roc`` folder and generate the ROC API test framework - and test files: - -.. code-block:: shell - - cd aether-2.1/roc - python libraries/api/codegen/class_generator.py \ - --models=variables/2_0_0_model_list.json \ - --template=libraries/api/codegen/templates/class_template.py.tmpl \ - --common_files_directory=libraries/api/codegen/common \ - --target_directory=libraries/api/ - python tests/api/codegen/tests_generator.py \ - --models=variables/2_0_0_model_list.json \ - --template=tests/api/codegen/templates/tests_template.robot.tmpl \ - --target_directory=tests/api - -5. Go to the directory that contains the test files: - -.. code-block:: shell - - cd tests/api/2_0_0 - -6. Create a folder for the logs and the output files from the tests: - -.. code-block:: shell - - mkdir results - -7. Run any Robot Framework test file from the ``2_0_0`` directory. -Each test file corresponds to one of the Aether 2.0.0 models. - -.. code-block:: shell - - robot -d results .robot - -This will generate test reports and logs in the ``results`` directory. - -Running the ROC GUI Tests -------------------------- - -Test the ROC GUI by installing the ROC with Keycloak enabled. -Currently, only v4 GUI automation tests are supported: - -1. Deploy the ``aether-roc-umbrella`` chart from the Aether repo with the - following command: - -.. code-block:: shell - - helm -n micro-onos install aether-roc-umbrella aether/aether-roc-umbrella \ - --set import.sdcore-adapter.v4.enabled=true \ - --set import.aether-roc-gui.v4.enabled=true \ - --set onos-config.openidc.issuer=https://keycloak-dev.onlab.us/auth/realms/master \ - --set aether-roc-api.openidc.issuer=https://keycloak-dev.onlab.us/auth/realms/master \ - --set aether-roc-gui-v4.openidc.issuer=https://keycloak-dev.onlab.us/auth/realms/master \ - --set prom-label-proxy-acc.config.openidc.issuer=https://keycloak-dev.onlab.us/auth/realms/master \ - --set prom-label-proxy-amp.config.openidc.issuer=https://keycloak-dev.onlab.us/auth/realms/master - -Alternatively, v2 GUI can be deployed with the following command: - -.. code-block:: shell - - helm -n micro-onos install aether-roc-umbrella aether/aether-roc-umbrella \ - --set onos-config.openidc.issuer=https://keycloak-dev.onlab.us/auth/realms/master \ - --set aether-roc-api.openidc.issuer=https://keycloak-dev.onlab.us/auth/realms/master \ - --set aether-roc-gui-v2.openidc.issuer=https://keycloak-dev.onlab.us/auth/realms/master \ - --set prom-label-proxy-acc.config.openidc.issuer=https://keycloak-dev.onlab.us/auth/realms/master \ - --set prom-label-proxy-amp.config.openidc.issuer=https://keycloak-dev.onlab.us/auth/realms/master - -2. Check if all pods are in a Running state: - -.. code-block:: shell - - kubectl -n micro-onos get pods - -3. Once all pods are in a Running state, port-forward to port 8183 to - access the ROC GUI: - -.. code-block:: shell - - kubectl -n micro-onos port-forward service/aether-roc-gui-v4 8183:80 & - -4. Port-forward to port 8181 to access the ROC API (which is necessary - for some test cases): - -.. code-block:: shell - - kubectl -n micro-onos port-forward service/aether-roc-api 8181 & - -Now that we have access to the ROC API and GUI, we can proceed with -running the ROC GUI tests from the ``aether-system-tests`` repository: - -1. Checkout the aether-system-tests repo: - -.. code-block:: shell - - git clone "ssh://$GIT_USER@gerrit.opencord.org:29418/aether-system-tests" - -2. Go to the repo directory: - -.. code-block:: shell - - cd aether-system-tests - -3. Install the requirements and create a virtual environment: - -.. code-block:: shell - - make ast-venv - source ast-venv/bin/activate - -4. Go to the ``roc`` folder and generate the ROC GUI test files: - -.. code-block:: shell - - cd aether-2.1/roc - python tests/gui/codegen/tests_generator.py \ - --models=variables/4_0_0_model_list.json \ - --template=tests/gui/codegen/templates/tests_template.robot.tmpl \ - --target_directory=tests/gui - -5. Go to the directory that contains the test files: - -.. code-block:: shell - - cd tests/gui/4_0_0 - -6. Create a folder for the logs and the output files from the tests: - -.. code-block:: shell - - mkdir results - -7. Run any Robot Framework test file from the ``4_0_0`` directory. Each test - file corresponds to one of the Aether 4.0.0 models. - -.. code-block:: shell - - robot -d results .robot - -This will generate test reports and logs in the ``results`` directory. diff --git a/testing/integration_tests.rst b/testing/integration_tests.rst deleted file mode 100644 index f761b59..0000000 --- a/testing/integration_tests.rst +++ /dev/null @@ -1,26 +0,0 @@ -.. - SPDX-FileCopyrightText: © 2023 Open Networking Foundation - SPDX-License-Identifier: Apache-2.0 - -Integration Tests -=================== - -A set of integration tests validate various configurations of Aether. -The tests are managed by Jenkins, and can be monitored using the -following `Dashboard `__. - -Source code for the integration tests can be found on `GitHub -`__, where each -file in the repo corresponds to a Groovy script that implements a Jenkins pipeline -for one of the :doc:`Aether Blueprints `. - -The pipelines are executed daily, with each pipeline parameterized to -run in multiple jobs. The ``${AgentLabel}`` parameter selects the -Ubuntu release being tested (currently ``20.04`` and ``22.04``), with -all jobs running in AWS VMs (currently resourced as -``M7iFlex2xlarge``). Pipelines that exercise two-server tests (e.g., -``ueransim.groovy``, ``upf.groovy``, ``srsran.groovy``, and -``gnbsim.groovy``) run in VMs that have the `AWS CLI -`__ installed; the CLI is used to -create the second VM. All VMs have Ansible installed, as documented in -the :doc:`OnRamp Guide `.