Skip to content

Commit d9af6b6

Browse files
andy-maierandy-maier
committed
Support for function test with YAML files
Details: * Added an implementation for running function tests where the testcases are defined in YAML files: - Added a directory tests/function with a pytest file conftest.py that collects the YAML test files and runs the testcases they define. - Added a JSON schema file tests/function/schemas/test_file.schema.yaml for validating the YAML test files. * Using the new function test concept, added a first YAML test file test_user.yaml with a first few testcases for HMC users. * Added and changed make targets: - Added target 'unittest' to run just the existing unit tests. - Added target 'functiontest' to run just the new function tests. - Changed target 'test' to run both unittest and functiontest. * Split up the GutHub Actions test steps for 'make test' into two steps to run unittest and functiontest separately. Signed-off-by: Andreas Maier <[email protected]>
1 parent acc52ee commit d9af6b6

File tree

7 files changed

+1419
-46
lines changed

7 files changed

+1419
-46
lines changed

‎.github/workflows/test.yml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -236,7 +236,14 @@ jobs:
236236
RUN_TYPE: ${{ steps.set-run-type.outputs.result }}
237237
# TESTCASES: test_cim_obj.py
238238
run: |
239-
make test
239+
make unittest
240+
- name: Run function test
241+
env:
242+
PACKAGE_LEVEL: ${{ matrix.package_level }}
243+
RUN_TYPE: ${{ steps.set-run-type.outputs.result }}
244+
# TESTCASES: test_cim_obj.py
245+
run: |
246+
make functiontest
240247
- name: Run end2end_mocked test
241248
env:
242249
PACKAGE_LEVEL: ${{ matrix.package_level }}

‎Makefile‎

Lines changed: 65 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,14 @@ test_unit_py_files := \
199199
$(wildcard $(test_dir)/unit/*/*.py) \
200200
$(wildcard $(test_dir)/unit/*/*/*.py) \
201201

202+
test_function_py_files := \
203+
$(wildcard $(test_dir)/function/*.py) \
204+
$(wildcard $(test_dir)/function/*/*.py) \
205+
206+
test_function_yaml_files := \
207+
$(wildcard $(test_dir)/function/*.yaml) \
208+
$(wildcard $(test_dir)/function/*/*.yaml) \
209+
202210
test_end2end_py_files := \
203211
$(wildcard $(test_dir)/end2end/*.py) \
204212
$(wildcard $(test_dir)/end2end/*/*.py) \
@@ -238,6 +246,7 @@ bandit_rc_file := .bandit.toml
238246
check_py_files := \
239247
$(filter-out $(vendor_py_files), $(package_py_files)) \
240248
$(test_unit_py_files) \
249+
$(test_function_py_files) \
241250
$(test_end2end_py_files) \
242251
$(test_common_py_files) \
243252
$(example_py_files) \
@@ -277,55 +286,57 @@ help:
277286
@echo "Package version will be: $(package_version)"
278287
@echo ""
279288
@echo "Make targets:"
280-
@echo " install - Install package in active Python environment (non-editable)"
281-
@echo " develop - Prepare the development environment by installing prerequisites"
282-
@echo " check_reqs - Perform missing dependency checks"
283-
@echo " check - Run Flake8 on sources"
284-
@echo " ruff - Run Ruff (an alternate lint tool) on sources"
285-
@echo " pylint - Run PyLint on sources"
286-
@echo " safety - Run safety for install and all"
287-
@echo " bandit - Run bandit checker"
288-
@echo " test - Run unit tests (adds to coverage results)"
289-
@echo " end2end_mocked - Run end2end tests against example mock environments (adds to coverage results)"
290-
@echo " installtest - Run install tests"
291-
@echo " build - Build the distribution files in: $(dist_dir)"
292-
@echo " builddoc - Build documentation in: $(doc_build_dir)"
293-
@echo " all - Do all of the above"
294-
@echo " end2end - Run end2end tests (adds to coverage results)"
295-
@echo " end2end_show - Show HMCs defined for end2end tests"
296-
@echo " end2end_check - Check access to all HMCs defined in your HMC inventory file for end2end tests"
297-
@echo " authors - Generate AUTHORS.md file from git log"
298-
@echo " uninstall - Uninstall package from active Python environment"
299-
@echo " release_branch - Create a release branch when releasing a version (requires VERSION and optionally BRANCH to be set)"
300-
@echo " release_publish - Publish to PyPI when releasing a version (requires VERSION and optionally BRANCH to be set)"
301-
@echo " start_branch - Create a start branch when starting a new version (requires VERSION and optionally BRANCH to be set)"
302-
@echo " start_tag - Create a start tag when starting a new version (requires VERSION and optionally BRANCH to be set)"
303-
@echo " clean - Remove any temporary files"
304-
@echo " clobber - Remove any build products"
305-
@echo " platform - Display the information about the platform as seen by make"
306-
@echo " debuginfo - Display the debug information for the package"
307-
@echo " env - Display the environment as seen by make"
289+
@echo " install - Install package in active Python environment (non-editable)"
290+
@echo " develop - Prepare the development environment by installing prerequisites"
291+
@echo " check_reqs - Perform missing dependency checks"
292+
@echo " check - Run Flake8 on sources"
293+
@echo " ruff - Run Ruff (an alternate lint tool) on sources"
294+
@echo " pylint - Run PyLint on sources"
295+
@echo " safety - Run safety for install and all"
296+
@echo " bandit - Run bandit checker"
297+
@echo " unittest - Run unit tests (adds to coverage results)"
298+
@echo " functiontest - Run function tests (adds to coverage results)"
299+
@echo " test - Run unit and function tests (adds to coverage results)"
300+
@echo " end2end_mocked - Run end2end tests against example mock environments (adds to coverage results)"
301+
@echo " installtest - Run install tests"
302+
@echo " build - Build the distribution files in: $(dist_dir)"
303+
@echo " builddoc - Build documentation in: $(doc_build_dir)"
304+
@echo " all - Do all of the above"
305+
@echo " end2end - Run end2end tests (adds to coverage results)"
306+
@echo " end2end_show - Show HMCs defined for end2end tests"
307+
@echo " end2end_check - Check access to all HMCs defined in your HMC inventory file for end2end tests"
308+
@echo " authors - Generate AUTHORS.md file from git log"
309+
@echo " uninstall - Uninstall package from active Python environment"
310+
@echo " release_branch - Create a release branch when releasing a version (requires VERSION and optionally BRANCH to be set)"
311+
@echo " release_publish - Publish to PyPI when releasing a version (requires VERSION and optionally BRANCH to be set)"
312+
@echo " start_branch - Create a start branch when starting a new version (requires VERSION and optionally BRANCH to be set)"
313+
@echo " start_tag - Create a start tag when starting a new version (requires VERSION and optionally BRANCH to be set)"
314+
@echo " clean - Remove any temporary files"
315+
@echo " clobber - Remove any build products"
316+
@echo " platform - Display the information about the platform as seen by make"
317+
@echo " debuginfo - Display the debug information for the package"
318+
@echo " env - Display the environment as seen by make"
308319
@echo ""
309320
@echo "Environment variables:"
310-
@echo " TESTCASES=... - Testcase filter for pytest -k"
311-
@echo " TESTOPTS=... - Options for pytest"
312-
@echo " TESTHMC=... - HMC group or host name in HMC inventory file to be used in end2end tests. Default: $(default_testhmc)"
321+
@echo " TESTCASES=... - Testcase filter for pytest -k"
322+
@echo " TESTOPTS=... - Options for pytest"
323+
@echo " TESTHMC=... - HMC group or host name in HMC inventory file to be used in end2end tests. Default: $(default_testhmc)"
313324
@echo " TESTINVENTORY=... - Path name of HMC inventory file used in end2end tests. Default: $(default_testinventory)"
314-
@echo " TESTVAULT=... - Path name of HMC vault file used in end2end tests. Default: $(default_testvault)"
325+
@echo " TESTVAULT=... - Path name of HMC vault file used in end2end tests. Default: $(default_testvault)"
315326
@echo " TESTRESOURCES=... - The resources to test with in end2end tests, as follows:"
316-
@echo " random - one random choice from the complete list of resources (default)"
317-
@echo " all - the complete list of resources"
318-
@echo " <pattern> - the resources with names matching the regexp pattern"
319-
@echo " TESTLOGFILE=... - Enable logging in end2end tests to that file. Default: no logging"
320-
@echo " PACKAGE_LEVEL - Package level to be used for installing dependent Python"
327+
@echo " random - one random choice from the complete list of resources (default)"
328+
@echo " all - the complete list of resources"
329+
@echo " <pattern> - the resources with names matching the regexp pattern"
330+
@echo " TESTLOGFILE=... - Enable logging in end2end tests to that file. Default: no logging"
331+
@echo " PACKAGE_LEVEL - Package level to be used for installing dependent Python"
321332
@echo " packages in 'install' and 'develop' targets:"
322-
@echo " latest - Latest package versions available on Pypi"
323-
@echo " minimum - Minimum versions as defined in minimum-constraints*.txt"
333+
@echo " latest - Latest package versions available on Pypi"
334+
@echo " minimum - Minimum versions as defined in minimum-constraints*.txt"
324335
@echo " Optional, defaults to 'latest'."
325-
@echo " PYTHON_CMD=... - Name of python command. Default: python"
326-
@echo " PIP_CMD=... - Name of pip command. Default: pip"
327-
@echo " VERSION=... - M.N.U version to be released or started"
328-
@echo " BRANCH=... - Name of branch to be released or started (default is derived from VERSION)"
336+
@echo " PYTHON_CMD=... - Name of python command. Default: python"
337+
@echo " PIP_CMD=... - Name of pip command. Default: pip"
338+
@echo " VERSION=... - M.N.U version to be released or started"
339+
@echo " BRANCH=... - Name of branch to be released or started (default is derived from VERSION)"
329340

330341
.PHONY: platform
331342
platform:
@@ -679,12 +690,21 @@ else
679690
endif
680691
echo "done" >$@
681692

682-
.PHONY: test
683-
test: $(done_dir)/develop_$(pymn)_$(PACKAGE_LEVEL).done $(package_py_files) $(test_unit_py_files) $(test_common_py_files) $(coverage_config_file)
693+
.PHONY: unittest
694+
unittest: $(done_dir)/develop_$(pymn)_$(PACKAGE_LEVEL).done $(package_py_files) $(test_unit_py_files) $(test_common_py_files) $(coverage_config_file)
684695
bash -c "PYTHONPATH=. coverage run --append -m pytest --color=yes $(pytest_no_log_opt) -s $(pytest_opts) $(test_dir)/unit"
685696
coverage html
686697
@echo "Makefile: $@ done."
687698

699+
.PHONY: functiontest
700+
functiontest: $(done_dir)/develop_$(pymn)_$(PACKAGE_LEVEL).done $(package_py_files) $(test_function_py_files) $(test_function_yaml_files) $(test_common_py_files) $(coverage_config_file)
701+
bash -c "PYTHONPATH=. coverage run --append -m pytest --color=yes $(pytest_no_log_opt) -s $(pytest_opts) $(test_dir)/function"
702+
coverage html
703+
@echo "Makefile: $@ done."
704+
705+
.PHONY: test
706+
test: unittest functiontest
707+
688708
.PHONY: installtest
689709
installtest: $(bdist_file) $(sdist_file) $(test_dir)/installtest/test_install.sh
690710
ifeq ($(PLATFORM),Windows_native)

‎tests/function/README.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Function tests for the zhmcclient
2+
3+
This directory contains YAML files that define testcases for testing the
4+
zhmcclient by simulating the HMC interactions. This allows to test all layers
5+
of the zhmcclient library from its public API down to the "requests" package
6+
providing the transport layer for communicating with the HMC. Specifically,
7+
the "requests-mock" Python package is used for that.
8+
This concept allows end2end-like testing without actually using an HMC.
9+
10+
Each testcase in a YAML test file defines:
11+
12+
* one zhmcclient API method that will be called or one zhmcclient API property
13+
that will be accessed
14+
15+
* zero or more HMC interactions resulting from that method call or property
16+
access. Each HMC interaction defines:
17+
18+
- Expected HMC request payload generated by the zhmcclient code
19+
20+
- HMC response payload to be passed back to the zhmcclient code
21+
22+
* The expected result of the zhmcclient API function, including any exceptions.
23+
24+
There is no setup needed for these tests, and no state of the simulated HMC or
25+
of the zhmcclient session needs to be kept or defined in the testcase. These
26+
tests also do not depend on the zhmcclient mock support.
27+
28+
The behavior is entirely determined by the HMC responses defined in the
29+
testcase. For example, a testcase that performs a "Get XXX Properties"
30+
operation on a non-existing resource is defined by setting the HMC response
31+
payload in the testcase to the corresponding error.
32+
33+
The format of the test files is defined in JSON schema
34+
[schemas/test_file.schema.yaml](schemas/test_file.schema.yaml).
35+
36+
Limitations:
37+
38+
* zhmcclient methods and properties that return zhmcclient objects are not
39+
supported.
40+
* notifications are not supported.

‎tests/function/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)