use pytest-mock to simplify and improve tests, improve test docs (#468)

use pytest-mock to simplify and improve tests, improve test docs

Author: wholmgren

appveyor.yml

@@ -28,7 +28,7 @@ install:
     - cmd: conda info -a
 
     # install depenencies
-    - cmd: conda create -n test_env --yes --quiet python=%PYTHON_VERSION% pip numpy scipy pytables pandas nose pytest pytz ephem numba siphon -c conda-forge
+    - cmd: conda create -n test_env --yes --quiet python=%PYTHON_VERSION% pip numpy scipy pytables pandas nose pytest pytz ephem numba siphon pytest-mock -c conda-forge
     - cmd: activate test_env
     - cmd: python --version
     - cmd: conda list

ci/requirements-py27-min.yml

@@ -4,6 +4,7 @@ dependencies:
     - pytz
     - pytest
     - pytest-cov
+    - pytest-mock
     - nose
     - pip:
         - coveralls

ci/requirements-py27.yml

@@ -14,6 +14,7 @@ dependencies:
     - siphon
     - pytest
     - pytest-cov
+    - pytest-mock
     - nose
     - pip:
         - coveralls

ci/requirements-py34.yml

@@ -14,6 +14,7 @@ dependencies:
     - siphon
     - pytest
     - pytest-cov
+    - pytest-mock
     - nose
     - pip:
         - coveralls

ci/requirements-py35.yml

@@ -14,6 +14,7 @@ dependencies:
     - siphon
     - pytest
     - pytest-cov
+    - pytest-mock
     - nose
     - pip:
         - coveralls

ci/requirements-py36.yml

@@ -14,6 +14,7 @@ dependencies:
     #- siphon
     - pytest
     - pytest-cov
+    - pytest-mock
     - nose
     - pip:
         - coveralls

docs/sphinx/source/contributing.rst

@@ -9,7 +9,7 @@ contribute.
 
 
 Easy ways to contribute
------------------------
+~~~~~~~~~~~~~~~~~~~~~~~
 
 Here are a few ideas for you can contribute, even if you are new to
 pvlib-python, git, or Python:
@@ -33,7 +33,7 @@ pvlib-python, git, or Python:
 
 
 How to contribute new code
---------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Contributors to pvlib-python use GitHub's pull requests to add/modify
 its source code. The GitHub pull request process can be intimidating for
@@ -81,29 +81,142 @@ changes, such as fixing documentation typos.
 
 
 Testing
--------
+~~~~~~~
 
 pvlib's unit tests can easily be run by executing ``py.test`` on the
 pvlib directory:
 
-``py.test pvlib``
+``pytest pvlib``
 
 or, for a single module:
 
-``py.test pvlib/test/test_clearsky.py``
+``pytest pvlib/test/test_clearsky.py``
 
-While copy/paste coding should generally be avoided, it's a great way
-to learn how to write unit tests!
+or, for a single test:
 
-Unit test code should be placed in the corresponding test module in the
-pvlib/test directory.
+``pytest pvlib/test/test_clearsky.py::test_ineichen_nans``
+
+Use the ``--pdb`` flag to debug failures and avoid using ``print``.
+
+New unit test code should be placed in the corresponding test module in
+the pvlib/test directory.
 
 Developers **must** include comprehensive tests for any additions or
 modifications to pvlib.
 
+pvlib-python contains 3 "layers" of code: functions, PVSystem/Location,
+and ModelChain. Contributors will need to add tests that correspond to
+the layer that they modify.
+
+Functions
+---------
+Tests of core pvlib functions should ensure that the function returns
+the desired output for a variety of function inputs. The tests should be
+independent of other pvlib functions (see :issue:`394`). The tests
+should ensure that all reasonable combinations of input types (floats,
+nans, arrays, series, scalars, etc) work as expected. Remember that your
+use case is likely not the only way that this function will be used, and
+your input data may not be generic enough to fully test the function.
+Write tests that cover the full range of validity of the algorithm.
+It is also important to write tests that assert the return value of the
+function or that the function throws an exception when input data is
+beyond the range of algorithm validity.
+
+PVSystem/Location
+-----------------
+The PVSystem and Location classes provide convenience wrappers around
+the core pvlib functions. The tests in test_pvsystem.py and
+test_location.py should ensure that the method calls correctly wrap the
+function calls. Many PVSystem/Location methods pass one or more of their
+object's attributes (e.g. PVSystem.module_parameters, Location.latitude)
+to a function. Tests should ensure that attributes are passed correctly.
+These tests should also ensure that the method returns some reasonable
+data, though the precise values of the data should be covered by
+function-specific tests discussed above.
+
+We prefer to use the ``pytest-mock`` framework to write these tests. The
+test below shows an example of testing the ``PVSystem.ashraeiam``
+method. ``mocker`` is a ``pytest-mock`` object. ``mocker.spy`` adds
+features to the ``pvsystem.ashraeiam`` *function* that keep track of how
+it was called. Then a ``PVSystem`` object is created and the
+``PVSystem.ashraeiam`` *method* is called in the usual way. The
+``PVSystem.ashraeiam`` method is supposed to call the
+``pvsystem.ashraeiam`` function with the angles supplied to the method
+call and the value of ``b`` that we defined in ``module_parameters``.
+The ``pvsystem.ashraeiam.assert_called_once_with`` tests that this does,
+in fact, happen. Finally, we check that the output of the method call is
+reasonable.
+
+.. code-block:: python
+    def test_PVSystem_ashraeiam(mocker):
+        # mocker is a pytest-mock object.
+        # mocker.spy adds code to a function to keep track of how it is called
+        mocker.spy(pvsystem, 'ashraeiam')
+
+        # set up inputs
+        module_parameters = {'b': 0.05}
+        system = pvsystem.PVSystem(module_parameters=module_parameters)
+        thetas = 1
+
+        # call the method
+        iam = system.ashraeiam(thetas)
+
+        # did the method call the function as we expected?
+        # mocker.spy added assert_called_once_with to the function
+        pvsystem.ashraeiam.assert_called_once_with(thetas, b=module_parameters['b'])
+
+        # check that the output is reasonable, but no need to duplicate
+        # the rigorous tests of the function
+        assert iam < 1.
+
+Avoid writing PVSystem/Location tests that depend sensitively on the
+return value of a statement as a substitute for using mock. These tests
+are sensitive to changes in the functions, which is *not* what we want
+to test here, and are difficult to maintain.
+
+ModelChain
+----------
+The tests in test_modelchain.py should ensure that
+``ModelChain.__init__`` correctly configures the ModelChain object to
+eventually run the selected models. A test should ensure that the
+appropriate method is actually called in the course of
+``ModelChain.run_model``. A test should ensure that the model selection
+does have a reasonable effect on the subsequent calculations, though the
+precise values of the data should be covered by the function tests
+discussed above. ``pytest-mock`` can also be used for testing ``ModelChain``.
+
+The example below shows how mock can be used to assert that the correct
+PVSystem method is called through ``ModelChain.run_model``.
+
+.. code-block:: python
+    def test_modelchain_dc_model(mocker):
+        # set up location and system for model chain
+        location = location.Location(32, -111)
+        system = pvsystem.PVSystem(module_parameters=some_sandia_mod_params,
+                                   inverter_parameters=some_cecinverter_params)
+
+        # mocker.spy adds code to the system.sapm method to keep track of how
+        # it is called. use returned mock object m to make assertion later,
+        # but see example above for alternative
+        m = mocker.spy(system, 'sapm')
+
+        # make and run the model chain
+        mc = ModelChain(system, location,
+                        aoi_model='no_loss', spectral_model='no_loss')
+        times = pd.date_range('20160101 1200-0700', periods=2, freq='6H')
+        mc.run_model(times)
+
+        # assertion fails if PVSystem.sapm is not called once
+        # if using returned m, prefer this over m.assert_called_once()
+        # for compatibility with python < 3.6
+        assert m.call_count == 1
+
+        # ensure that dc attribute now exists and is correct type
+        assert isinstance(mc.dc, (pd.Series, pd.DataFrame))
+
 
 This documentation
-------------------
+~~~~~~~~~~~~~~~~~~
 
 If this documentation is unclear, help us improve it! Consider looking
 at the `pandas

docs/sphinx/source/whatsnew/v0.6.0.rst

@@ -24,10 +24,16 @@ Bug fixes
 
 Documentation
 ~~~~~~~~~~~~~
+* Expand testing section with guidelines for functions, PVSystem/Location
+  objects, and ModelChain.
 
 
 Testing
 ~~~~~~~
+* Add pytest-mock dependency
+* Use pytest-mock to ensure that PVSystem methods call corresponding functions
+  correctly. Removes implicit dependence on precise return values of functions
+* Use pytest-mock to ensure that ModelChain DC model is set up correctly.
 
 
 Contributors

pvlib/test/test_modelchain.py

@@ -211,6 +211,23 @@ def test_dc_models(system, cec_dc_snl_ac_system, pvwatts_dc_pvwatts_ac_system,
     assert_series_equal(ac, expected, check_less_precise=2)
 
 
+@requires_scipy
+@pytest.mark.parametrize('dc_model', ['sapm', 'singlediode', 'pvwatts_dc'])
+def test_infer_dc_model(system, cec_dc_snl_ac_system,
+                        pvwatts_dc_pvwatts_ac_system, location, dc_model,
+                        mocker):
+    dc_systems = {'sapm': system, 'singlediode': cec_dc_snl_ac_system,
+                  'pvwatts_dc': pvwatts_dc_pvwatts_ac_system}
+    system = dc_systems[dc_model]
+    m = mocker.spy(system, dc_model)
+    mc = ModelChain(system, location,
+                    aoi_model='no_loss', spectral_model='no_loss')
+    times = pd.date_range('20160101 1200-0700', periods=2, freq='6H')
+    mc.run_model(times)
+    assert m.call_count == 1
+    assert isinstance(mc.dc, (pd.Series, pd.DataFrame))
+
+
 def acdc(mc):
     mc.ac = mc.dc