Add zaza in the charms test-requirements.txt:
git+https://github.com/openstack-charmers/zaza.git#egg=zaza
Add targets to tox.ini should include a target like:
[testenv:func]
basepython = python3
commands =
functest-run-suite --keep-model
[testenv:func-smoke]
basepython = python3
commands =
functest-run-suite --keep-model --smoke
The bundles live in tests/bundles of the built charm, eg:
tests/bundles/xenial.yaml tests/bundles/xenial-ha.yaml tests/bundles/bionic.yaml
The bundle may include overlay templates which are, currently, populated from environment variables. For example the xenial-ha template needs a VIP but the VIP will depend on the setup of the juju provider so will be different between test environments. To accommodate this an overlay is added:
tests/bundles/overlays/xenial-ha.yaml.j2
The overlay is in jinja2 format and the variables correspond to environment variables:
applications:
vault:
options:
vip: '{{ OS_VIP00 }}'
It is also possible to provide overlay templates tailored for specific juju provider types, this can be useful to do any provider specific morphing of a bundle. To use this feature use the following directory layout:
tests/bundles/overlays/xenial.yaml.j2 tests/bundles/overlays/lxd/xenial.yaml.j2
With the above directory layout the overlay template in the lxd sub-directory will be used when tests are executed with juju on a LXD provider and the overlay template in the top level directory will be used for any other provider types.
Bundle templates can be placed in the tests/bundles directory. It is usually preferable to use overlay templates rather than bundle templates. Overlays can be used to neatly capture settings that a user might want to change on a per-cloud basis. However, if a bundle template is required place it in tests/bundles with a j2 extension.
A tests/tests.yaml file that describes the bundles to be run and the tests:
charm_name: vault tests: - zaza.charm_tests.vault.VaultTest configure: - zaza.charm_tests.vault.setup.basic_setup gate_bundles: - base-xenial - base-bionic dev_bundles: - base-xenial-ha smoke_bundles: - base-bionic
When deploying zaza will wait for the deployment to settle and for the charms to display a workload status which indicates that they are ready. Sometimes one or more of the applications being deployed may have a non-standard workload status target state or message. To inform the deployment step what to wait for an optional target_deploy_status stanza can be added:
target_deploy_status:
vault:
workload-status: blocked
workload-status-message: Vault needs to be initialized
ntp:
workload-status-message: Go for it
The setup and tests for a charm should live in zaza, this enables the code to be shared between multiple charms. To add support for a new charm create a directory, named after the charm, inside zaza/charm_tests. Within the new directory define the tests in tests.py and any setup code in setup.py This code can then be referenced in the charms tests.yaml
e.g. to add support for a new congress charm create a new directory in zaza:
mkdir zaza/charm_tests/congress
Add setup code into setup.py:
$ cat zaza/charm_tests/congress/setup.py
def basic_setup():
congress_client(run_special_setup)
Add test code into tests.py:
class CongressTest(unittest.TestCase):
def test_policy_create(self):
policy = congress.create_policy()
self.assertTrue(policy)
These now need to be referenced in the congress charms tests.yaml. Additional setup is needed to run a useful congress tests, so congress' tests.yaml might look like:
charm_name: congress configure: - zaza.charm_tests.nova.setup.flavor_setup - zaza.charm_tests.nova.setup.image_setup - zaza.charm_tests.neutron.setup.create_tenant_networks - zaza.charm_tests.neutron.setup.create_ext_networks - zaza.charm_tests.congress.setup.basic_setup tests: - zaza.charm_tests.keystone.KeystoneBasicTest - zaza.charm_tests.congress.CongressTest gate_bundles: - base-xenial - base-bionic dev_bundles: - base-xenial-ha
In order to allow early testing of new Ubuntu series the juju deploy command has a --force option. This allows deployment of charms that don't specify the series being used, or for a new series that juju doesn't support yet.
Force deploying can be achieved in two ways, either on the command line, or via an tests_options.force_deploy entry in the tests.yaml file.
For the command line a --force param is provided:
functest-run-suite --keep-model --dev --force functest-deploy <other options> --force
In the tests.yaml the option is added as a list item:
charm_name: keystone smoke_bundles: - focal-ussuri
...
- tests_options:
- force_deploy:
- focal-ussuri
In the above case, focal-ussuri will be deployed using the --force parameter. i.e. the tests_options.force_deploy['focal-ussuri'] option applies to the focal-ussuri bundle whether it appears in any of the bundle sections.
By default, after deploying a bundle zaza waits for all applications to reach
the states specified in target_deploy_status (or active/idle if not
configured). This wait can be skipped on a per-bundle basis using the
tests_options.no_wait_deploy option.
This is useful when a charm requires post-deploy configuration before it can reach its target state, or when you want to proceed immediately to the configure step without waiting.
In the tests.yaml the option is added as a list item:
charm_name: vault
gate_bundles:
- focal-ussuri
target_deploy_status:
vault:
workload-status: blocked
workload-status-message: Vault needs to be initialized
tests_options:
no_wait_deploy:
- focal-ussuri
In the above case, zaza will deploy the focal-ussuri bundle and immediately
proceed to the configure step without waiting for vault to enter the blocked
state. This is equivalent to passing --no-wait directly to
functest-deploy.
Individual configuration steps or the library helper they use may define configuration step options that you may use to tweak behaviour from tests.yaml.
An example is the Neutron basic_overcloud_network configure job which for compatibility with existing scenario tests makes use of juju_wait when configuring the deployed cloud.
This does not work well if the model you are testing has applications with non-standard workload status messaging.
To replace the use of juju_wait with Zaza's configurable wait code:
charm_name: neutron-openvswitch smoke_bundles: - focal-ussuri-dvr-snat-migrate-ovn
...
- configure_options:
- configure_gateway_ext_port_use_juju_wait: false