Merge pull request #28 from stackhpc/rally-tempest

Add section on running Rally and Tempest

Author: oneswig

source/index.rst

@@ -26,6 +26,7 @@ Contents
    operations_and_monitoring
    customising_deployment
    gpus_in_openstack
+   rally_and_tempest
 
 Indices and search
 ==================

source/rally_and_tempest.rst

@@ -0,0 +1,106 @@
+.. include:: vars.rst
+
+==========================================
+Verifying the Cloud with Rally and Tempest
+==========================================
+
+`Rally <https://docs.openstack.org/rally/latest/>`_ is a test framework,
+and `Tempest <https://docs.openstack.org/tempest/latest/>`_ is the
+OpenStack API test suite.  In this guide, Rally is used to run Tempest tests.
+
+Requirements
+------------
+
+OpenStack tests are run from a host with access to the OpenStack APIs
+and external network for instances.
+
+The following software environment is needed:
+
+* OpenStack admin credentials, eg public-openrc.sh from kayobe-config/etc/kolla
+* A virtualenv setup with python-openstackclient installed
+
+.. code-block:: shell
+   :substitutions:
+
+   source venv/bin/activate
+   pip install python-openstackclient
+   source |base_path|/src/|kayobe_config|/etc/kolla/public-openrc.sh
+
+Setup Rally for a new user
+--------------------------
+
+A good directory hierarchy would be `~/rally/shakespeare/tempest-recipes`.
+This is assumed in this guide.
+
+Install Rally into the virtualenv:
+
+.. code-block:: shell
+
+   pip install rally-openstack \
+        --constraint https://releases.openstack.org/constraints/upper/master
+
+Create the Rally test database and configuration file.  For this you will
+need the virtualenv and public-openrc as described above:
+
+.. code-block:: shell
+
+   mkdir -p ~/.rally ~/rally/data
+   echo "[database]" | tee ~/.rally/rally.conf
+   echo "connection=sqlite:///${HOME}/rally/data/rally.db" | tee -a ~/.rally/rally.conf
+   rally db recreate
+   rally verify create-verifier --name default --type tempest
+   rally deployment create --fromenv --name production
+
+Check:
+
+.. code-block:: shell
+
+   rally deployment show
+
+Install Shakespeare
+-------------------
+
+Shakespeare is used for writing Tempest test configuration.
+
+.. code-block:: shell
+
+   cd ~/rally
+   git clone https://github.com/stackhpc/shakespeare.git
+   cd shakespeare
+   pip install -r requirements.txt
+
+Install Tempest Recipe
+----------------------
+
+A custom set of Tempest recipes should be maintained for |project_name|,
+defining key parameters needed and fine-grained control on the test cases
+to run (and which to skip).
+
+In your `shakespeare` directory,
+install the Tempest recipes in your Tempest configuration.
+
+.. code-block:: shell
+   :substitutions:
+
+   git clone |tempest_recipes|
+   ansible-playbook template.yml -e @tempest-recipes/production.yml
+   mkdir -p  ../config/production
+   rally verify configure-verifier --reconfigure --extend ../config/production/production.conf
+
+Rally invocation
+----------------
+
+Invoke the tests (this will take several hours to complete):
+
+.. code-block:: shell
+
+   rally --debug verify start --concurrency 1 --skip-list tempest-recipes/production-skiplist.yml
+
+Report generation
+-----------------
+
+Generate an HTML report of the results:
+
+.. code-block:: shell
+
+   rally verify report --type html --to ~/rally/report-$(date -d "today" +"%Y%m%d%H%M").html

source/vars.rst

@@ -44,5 +44,6 @@
 .. |seed_user| replace:: stack
 .. |support_email| replace:: [email protected]
 .. |support_level| replace:: standard support
+.. |tempest_recipes| replace:: https://github.com/acme-openstack/tempest-recipes.git
 .. |tls_setup| replace:: TLS is implemented using a wildcard certificate available for ``*.acme.example``.
 .. |vault_password_file_path| replace:: ~/vault-password