ENH: Port PVsyst and Desoto parameter estimation from MATLAB (#708)

* ENH: porting pvsyst and single diode param est

from PVLIB_MATLAB by Matt Guttenberg Aug-2016, see #229 and #227

Added ported functions and tests to pvlib

Added ported versions of calc_theta_phi_exact, lambertw, singlediode,
update_io_known_n, update_rsh_fixed_pt and associated tests to pvl-python.
Also added a __init__ file ot the tests folder to describe the file folder
and allow the tests to be called if necessary. PVsyst_parameter_estimation
and singlediode are not complete versions

Changed the tests to properly call the appropriate functions

Finished the singlediode script

Added Test script for singlediode

Added tests for singlediode functions and updated scripts

Updated the lambertw and singlediode scripts to fix some small errors that
were found while testing. Added 38 tests for the singlediode script
functions althought more tests will be added.

Finished singlediode and corresponding tests

Finished adding tests for singlediode that tests all of the
functionalities of the script, including properly raising errors. Updated
the singlediode script to fix any errors found while testing. Added a
statement to the lambertw function to make the calculations slightly
faster in the event a -inf case.

Added a new file for the Schumaker_QSpline script

Finished Schumaker_QSpline script

Fixed Schumaker_QSpline script based on test

Adjusted all index references to be integer values to avoid the warnings
that were displayed. Fixed all logic errors. Fixed the sorting mechanism
for the final array.

Added test file for Schumaker_QSpline script

Added a Schumaker_QSpline test

Added a Schumaker_QSpline test (more to be added) and fixed the
singlediode_tests script to adhere to PEP8 standards

Added a couple more tests for Schumaker_QSpline

Added the est_single_diode_parameter port and fixed typos

Added the est_single_diode_parameter ported script to pvlib. Fixed a typo
in the Schumaker_QSpline script that caused an error under a very
particular circumstance. Added brackets in the Schumaker_QSpline test
script to get rid of syntax errors.

Added tests for the est_single_diode_param script

Delete __init__.py

Changed one test case for lambertw

Fixed some ambiguities in the imports

Updated the imports in many of the regular and test scripts to fix some
ambiguity issues that might have been occurring during testing

Started adding code to the PVsyst_parameter_estimation script

Added Classes to singlediode and PVsyst_Parameter_estimation

Added a class to move on with the code for the PVsyst_paramter_estimation
script and added a class to the singlediode script to wrap up the answers
in a more succinct format. Adjusted the singlediode tests to reflect this
change.

Ported over some of the code for PVsyst_parameter_estimation

Ported over some of the sections from PVsyst_parameter_estimation. Added
numdiff function to the PVsyst_parameter_estimation script. Since the
script uses matplotlib, added the requirement to the setup.

Finished PVsyst_parameter_estimation and adjusted outputs of functions

Finished porting PVsyst_parameter_estimation except for the section with
the robust_fit. Changed the outputs of this function and singlediode from
a class to a ordereddict and changed the inputs for
PVsyst_parameter_estimation from classes to ordereddicts as well.

Changed the setup to include scipy as a required toolbox

Added robust fit algorithm to PVsyst_parameter_estimation

Added the robust fit algorithm where it was needed to determine desired
information. Since this alrogithm used the statsmodels toolbox, the setup
was changed to reflect this addition.

Removed lambertw since scipy already had a working copy. Script Changes

Removed lambertw and associated test script since scipy.special already
had a working lambertw that has been tested and verified. Changed
associated scripts to reflect this change. Some bug fixes for
Pvsyst_paramter_estimation

Removed some coded scripts and updated PVsyst_parameter_estimation

Removed i_from_v, v_from_i and singlediode and used the already coded and
tested versions within pvlib. Updated PVsyst_parameter_estimation based on
some bug fixes.

More bug fixes

Fixed additional typos and other errors that have prevented the code from
completing. Now that the code runs to completion, additional checks will
be run to determine whether the converged values are appropriate.

Updated calc_theta_phi_exact to handle an edge case

modified calc_theta_phi_exact to handle the edge case where nnsvth was
equal to 0.

Brought back lambertw script for use in update_rsh_fixed_pt

Reintroduced lambertw script for use in update_rsh_fixed_pt since the
lambertw script was having problems for cases where rsh was extremely high
in calc_theta_phi_exact

A couple mroe PVsyst_parameter_estimation bug fixes

Fixed import errors and removed unused lines

Fixed an import error in update_io_known_n and removed v_from_i from the
tests since the code is using the code that already exists and has been
tested in pvsystem. Removed an unused import in update_rsh_fixed_pt

Fixed typos / errors

Fixed an error in PVsyst_parameter_estimation in the numdiff function
where the summations were not occuring properly. Fixed a typo in pvsystem
where a parameter was accidentally changed.

Final bug fixes for PVsyst_parameter_estimation

Fixed a typo in one of the plots

Style Update

Modified all written scripts to adhere to the flake8 and pep8-naming
guidelines. Updated tests as suggested by Will

Documentation Update

Updated script documentation for each of the written scripts to fit better
with the pvlib documentation

Added Documents from Cliff

* change function name from est_single_diode_params

- to estimate_parameters() since it was the same as the module
- update imports and calls everywhere
- other modules still have functions shadowing them, like
pvsyst_parametre_estimation()

* use int indices, make docstring, test still fails

* isolate iv fitting and parameter generation

* TST: schumaker test is passing! Yoohoo!

* TST: fix imports and path to test data

* fix Io test sortof, some stickler

* fix tests, merge ivtools

- move ivtools.py to ivtools/__init__.py, all tests PASS
- fix Schmaker_QSpline again, boolean indexing in Numpy is different
 than MATLAB, size must be the same, so get just the first (n-1) indices
 first
- fix est_single_diode_param.py which uses np.tile, only works with
 integers, not floats, so remove periods from 1. to make it an int
- rename calc_theta_phi_exact and lambertw to be prefixed with test so that
 pytest actually runs them, both PASS
- start fixing test_PVsyst_parameter_estimation, FAIL

* fixing test_PVsyst_parameter_estimation

- in calc_theta_phi_exactly, replace if-then with
 np.where(nnsvth is zero, use inf, or calculate)
- restrict arrays to dtype=float where they're created from arrays, this
 prevents arrays of arrays with dtype=object, which FAILS with
 np.isnan(doesn't work on dtype=object, only float)
 TODO: fix this!
- in lambertw b/c any(must be an iterable) is called on z which is
 scalar when called from PVsyst_parameter_estimation then need to make z
at least a 1d array, even tho this causes problems later, oh well
 TODO: fix this!
- in pvsyst_parameter_estimation, replace enumerate with zip(),
 and more importantly in rectify_IV_curve() remove any nan's which were
 added to the ivcurves
- the old tests are still PASSING, but test-pvsyst is a WIP

* include statsmodels and mpl in ci/conda-envs

* stickler fixes, long lines, code tweaks

- ignore w503,w504 break before/after binary operator
- add logging to pvsyst_parameter_estimation
- add latex $math$ to graphics output
- replace repeated len(f) with n

* rename update io and rsh tests so they run

- use concise indexing notation in numdiff

* concise indexing in Schumaker QSpline

* add test for numdiff

- add test data from original MATLAB, based on pvsyst demo
- more concise indices in numdiff

* move pvsyst parameter estimation tests

- to pvlib.test

* use maximum not max in estRsh

- add verbosity to least_squares
- limit to 100 ivcurves

* fudge tolerance for pvsyst_parameter_estimation for fun

* restrict pvsyst estimation test to first 3000 points

- the last 500 points seem to cause a problem that I need more time to pin down
- instead of testing against the matlab generated pvsyst coefficients, which
 seem to be different from `pvlib.pvsystem.calcparams` works, also test the
 actual iv curve points (isc, voc, imp, vmp, pmp) generate from the coefficients in
 pvlib with the original data
- set tolerances to match the test data

* rename pvsyst parameter estimation tests folder

- suffix with _tests instead of prefix
- remove matplotlib from pvsyst_parameter_estimation, I save all of the
 snippets so maybe they can go in the tests folder?
- removes graphic & convergeparamsfig args from check_converge and the
 graphic arg from pvsyst_parameter_estimation funcs
- safely import scipy and statsmodels inside pvsyst_parameter_estimation so
 bare linux tests can run on CI, add comment at top
- also remove mpl from setup optional requires
- add pvsyst_parameter_estimation to api docs
- raise runtime error if doesn't converge instead of the oflag
- remove oflag arg from pvsyst_param_est everywhere
- add decorator for requires_statsmodels in conftest and apply to
 test_pvsyst_parameter_estimation
- rename Schmaker_QSpline and est_single_diode_param tests

Signed-off-by: Mark Mikofski <[email protected]>

* move test_numdiff inside test_pvsyst_parameter_estimation.py

* add requires_scipy to update_io_know_n tests

- remove TODO's to delete old code replaced by np.where in
 calc_theta_phi_exact
- stickler fixes:
  * overindented lines and continutation characters in calc_theta
  * long line in docstring of estimate_single_diode
  * over indentation in pvsyst_param_est
  * extra line after function check_converge before fun_rsh
- also add TODO to change convergence relative ratios

* pycodestyle passing, rm xtra sp, fix indent

* shuffle functions into different modules. Rename to fit_sdX_ pattern

* fix import names

* fix import in ivtools

* refactor deux: rm old files

- remove PVsyst_parameter_estimation.py with 7 methods, and the module
attribute `const_default`
    1. numdiff -> utility.numdiff
    2. rectify_iv_curve -> utility.rectify_iv_curve
    3. estrsh -> fit_sdm._rsh_pvsyst
    4. filter_params -> fit_sdm._filter_params
    5. check_converge -> fit_sdm._check_converge
    6. const_default -> utility.constants
    7. fun_rsh -> local to fit_pvsyst_sandia
    8. pvsyst_parameter_estimation -> fit_sdm.fit_pvsyst_sandia
- remove calc_theta_phi_exact with only 1 method which is in fit_sdm.py
now
- ditto for update_io_to_known_n & update_rsh_fixed_pt.py
- ditto remove est_single_diode_param.py
- remove lambertw, using scipy
- remove Schumaker_QSpline in favor do you even like music.
- fix some typos

Signed-off-by: Mark Mikofski <[email protected]>

* remove old pvsyst parma estimation tests

* stickler fixes

* using lambertw requires scipy in tests

- for calc_theta_phi_exact
- and update_rsh_fixed_pt

* align with scipy.intepolate pattern

* refactor fit_sde_cocontent, change argument order

* stickler

* fix argument order in fit_pvsyst_sandia

* fix rectify function, add test

* stickler

* corrections, add rcond=None

* more corrections

* undo change in rs, rsh order

* fix test for rectify_iv_curve, use np.expm1

* exclude curve #2540

* fix expected outcome for test_rectify_

* messed up csv

* exclude curve #2540

* correct rounding for test_rectify_

* move exclude to line 2540 was on 2536

* create exclude mask for individual curves

* use umask

* helper functions, some streamlining

* rename _update_io_known_n

* docstring improvements

* split _calc_theta_phi_exact, overhaul tests

* remove unused lines

* stickler

* remove test_ivtools.py

* renaming, update api.rst

* update name

* improve _update_rsh_fixed_pt

* fix typo

* fix error

* correct convergence in _update_rsh_fixed_pt

* flake8

* docstring for fit_sdm

* docstring for fit_sdm

* docstring for fit_sdm

* docstring for fit_sdm

* docstring for fit_sde, utility

* update api.rst

* add helper to deal with numpy<1.14

* fix workaround for numpy version

* rename modules and functions, move tests to ivtools/test

* move tests from ivtools/test to pvlib/tests/ivtools

* remove old test folder

* remove old pvlib/test folder

* name correction, remove old test files

* fix test, move test_fit_sde_cocontent into test_sde

* move test_schumaker_qspline to test_utility

* docstring work

* docstring work

* test corrections

* fix order in parametrize

* one comma

* corrections

* remove old files, update api.rst

* update names in ivtools\__init__.py

* docstring work, rename outputs from fit_pvsyst_sandia to be consistent with pvlib.pvsystem functions

* docstring work, use cells_in_series throughout

* docstring work, use mu_gamma instead of mugamma

* missed one

* sort of Rs vs R_s vs R_s_ref

* refactor to prepare for adding desoto, cec estimation; remove logging

* add fit_desoto_sandia

* correct argument order, lint

* change to pvlib-consistent key names

* add test for fit_desoto_sandia

* corrections

* more corrections

* use canonical order iph, io, rs, rsh, n

* convert update_rsh test, move to test_sdm.py

* fix update_rsh tests, improve function

* remove old update_rsh test file

* convert tests for _update_io

* remove old tests for _update_io

* convert tests for calc_theta_phi

* remove old test for calc_theta

* take care of float div by 0 test failures

* move sde functions to canonical argument order

* fix bugs

* more raid for bugs

* die bugs die

* maintain order of variables in the text file

* test repair

* minor editing

* pin pysam version to 1.2.1

* pysam to current version, patch over numpy 1.14 change

* module docstring

* remove ivtools.py

* separate sam_estimate_failure into its own function

* fix stickler

* 'mispelling'

* fix error in _calc_I0

* more fixing of that error

* improvements from review

* another fix for mistake in _calc_I0

* docstring edits

* improve coverage

* tpo

* use the correct error Class

* improvements from review

* changes from review: helper functions for test, _numdiff and _schum private, move and rename data files

* whatsnew

* add renamed and moved data files

* use pathlib.Path.open()

* duh

* add return, doing three things at once = poor quality

* add statsmodels for py38, fix dict declarations

* ()

Co-authored-by: Matt Guttenberg <[email protected]>
Co-authored-by: Cliff Hansen <[email protected]>

Author: 3 people

ci/requirements-py35.yml

@@ -23,6 +23,7 @@ dependencies:
     - scipy
     - shapely  # pvfactors dependency
     - siphon  # conda-forge
+    - statsmodels
     - pip:
         - nrel-pysam>=2.0
         - pvfactors==1.4.1

ci/requirements-py36.yml

@@ -25,6 +25,7 @@ dependencies:
     - scipy
     - shapely  # pvfactors dependency
     - siphon  # conda-forge
+    - statsmodels
     - pip:
         - nrel-pysam>=2.0
         - pvfactors==1.4.1

ci/requirements-py37.yml

@@ -25,6 +25,7 @@ dependencies:
     - scipy
     - shapely  # pvfactors dependency
     - siphon  # conda-forge
+    - statsmodels
     - pip:
         - nrel-pysam>=2.0
         - pvfactors==1.4.1

ci/requirements-py38.yml

@@ -25,6 +25,7 @@ dependencies:
     - scipy
     - shapely  # pvfactors dependency
     - siphon  # conda-forge
+    - statsmodels
     - pip:
         - nrel-pysam>=2.0
         - pvfactors==1.4.1

docs/sphinx/source/api.rst

@@ -314,6 +314,7 @@ Pvsyst model
 .. autosummary::
    :toctree: generated/
 
+   pvsystem.calcparams_pvsyst
    temperature.pvsyst_cell
    pvsystem.calcparams_pvsyst
    pvsystem.singlediode
@@ -328,6 +329,33 @@ PVWatts model
    inverter.pvwatts
    pvsystem.pvwatts_losses
 
+Estimating PV model parameters
+------------------------------
+
+Functions for fitting single diode models
+
+.. autosummary::
+   :toctree: generated/
+
+    ivtools.sdm.fit_cec_sam
+    ivtools.sdm.fit_desoto
+    ivtools.sdm.fit_pvsyst_sandia
+    ivtools.sdm.fit_desoto_sandia
+
+Functions for fitting the single diode equation
+
+.. autosummary::
+   :toctree: generated/
+
+    ivtools.sde.fit_sandia_simple
+
+Utilities for working with IV curve data
+
+.. autosummary::
+   :toctree: generated/
+
+    ivtools.utility.rectify_iv_curve
+
 Other
 -----
 

docs/sphinx/source/whatsnew/v0.8.0.rst

@@ -3,6 +3,17 @@
 v0.8.0 (Month day, year)
 -------------------------
 
+API Breaking Changes
+~~~~~~~~~~~~~~~~~~~~
+
+* The ``ivtools.py`` module has been reorganized into sub-modules and functions
+  in ``ivtools.py`` have been renamed (:pull:`708`)
+   - Functions for fitting diode models are now in ``ivtools\sdm.py``
+   - :py:func:`pvlib.ivtools.fit_sdm_cec_sam` is now :py:func:`pvlib.ivtools.sdm.fit_cec_sam`
+   - :py:func:`pvlib.ivtools.fit_sdm_desoto` is now :py:func:`pvlib.ivtools.sdm.fit_desoto`
+   - Functions for fitting the single diode equation are now in ``ivtools\sde.py``
+   - :py:func:`pvlib.ivtools.fit_sde_sandia` is now :py:func:`pvlib.ivtools.sde.fit_sandia_simple`
+
 API Changes with Deprecations
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 * Moved functions related to inverters from ``pvsystem.py`` to ``inverter.py``.
@@ -70,6 +81,8 @@ Enhancements
   (:pull:`1017`)
 * Add :py:func:`pvlib.inverter.fit_sandia` that fits the Sandia inverter model
   to a set of inverter efficiency curves. (:pull:`1011`)
+* Add :py:func:`pvlib.ivtools.sdm.fit_pvsyst_sandia` and :py:func:`pvlib.ivtools.sdm.fit_desoto_sandia`
+  for fitting the Pvsyst and De Soto models to IV curve data (:issue:`227`)(:pull:`708`)
 * Add factory methods :py:meth:`~pvlib.modelchain.ModelChain.with_pvwatts`
   :py:meth:`~pvlib.modelchain.ModelChain.with_sapm` to create ``ModelChain``
   objects configured for the respective modeling paradigms. The