Adjust srml, Add reference obs documentation #442

lboeman · 2020-05-13T23:09:45Z

Closes Fix SRML sites #436 .
I am familiar with the contributing guidelines.
Tests added.
Updates entries to docs/source/api.rst for API changes.
Adds descriptions to appropriate "what's new" file in docs/source/whatsnew for all changes. Includes link to the GitHub Issue with :issue:`num` or this Pull Request with :pull:`num`. Includes contributor name and/or GitHub username (link with :ghuser:`user`).
New code is fully documented. Includes numpydoc compliant docstrings, examples, and comments where necessary.
Maintainer: Appropriate GitHub Labels and Milestone are assigned to the Pull Request and linked Issue.

This PR does a few things:

Removes SRML sites that haven't reported data for a while.
When initializing SRML sites, use a data file from 30 days ago to ensure there's actually a file. Due to the delay, some of the current SRML sites in prod don't have observations.
Add power observations for Ashland OR, Kalapuya HS, Bend OR, and Portland OR PV. I've added a PV suffix to the end of each of their names. I've used the json format that was introduced in add pvdaq reference sites, fetch, config #438, for the PV sites.
Adds a top level "reference observations" page to the documents as suggested in Add reference event observations and forecasts #429. I've tried to update the explanation from that PR. I think it still needs work.

wholmgren

looks good. minor comments...

docs/source/reference-observations.rst

docs/source/whatsnew/1.0.0rc1.rst

wholmgren · 2020-05-15T03:05:26Z

solarforecastarbiter/io/reference_observations/srml.py

 logger = logging.getLogger('reference_data')


+def adjust_site_parameters(site):


looks like nearly a copy of the pvdaq function. should we refactor into common?

Yes. Minimal changes would be to move the meat to common with a sitefile kwarg then just call that from pvdaq/srml with the right sitefile

wholmgren · 2020-05-15T03:16:10Z

There were a handful of warnings related to the new files when building the docs. Check the readthedocs build log or build locally (make -C docs html)

wholmgren · 2020-05-20T02:53:50Z

solarforecastarbiter/io/reference_observations/tests/test_srml.py

+def request_data_test(mocker, exception, test_site, test_data):
+    get_func = mocker.patch(
+        'solarforecastarbiter.io.reference_observations.srml.iotools.'
+        'read_srml_month_from_solardat')
+    get_func.return_value = test_data
+    data = srml.request_data(test_site, 1, 1)
+    assert_frame_equal(data, test_data)
+
+
+@pytest.mark.parametrize('exception', [
+    pd.errors.EmptyDataError,
+    error.URLError,
+])
+def request_data_test_warnings(mocker, exception, test_site):
+    logger = mocker.patch(
+        'solarforecastarbiter.io.reference_observations.srml.iotools.'
+        'logger.warning')
+    get_func = mocker.patch(
+        'solarforecastarbiter.io.reference_observations.srml.iotools.'
+        'read_srml_month_from_solardat')
+    get_func.side_effect = exception()
+    data = srml.request_data(test_site, 1, 1)
+    assert logger.call_count == 5
+    assert data is None


Are these tests actually run? They don't start with test_ so I would assume not. Also codecov says no.

wholmgren

@alorenzo175 probably worthwhile for you to give this a quick review

…egree tilt portland pv site

Co-authored-by: Will Holmgren <[email protected]>

…rml.py docstrings

lboeman · 2020-05-27T20:05:55Z

I factored out the modelling parameter adjustment code into common.py, but I can't figure out what's causing the FileNotFound error in the tests. I've copied the behavior used for the tests in io.fetch that have their own test data. @alorenzo175 any ideas on what's wrong here?

alorenzo175 · 2020-05-27T20:15:40Z

I'm not seeing the test file at all. Did it git ignored and not committed?

alorenzo175 · 2020-05-27T21:16:30Z

solarforecastarbiter/io/reference_observations/pvdaq.py

-            site_out = site.copy()
-            site_out['modeling_parameters'] = site_metadata[
-                'modeling_parameters']
-            site_out['extra_parameters'].update(site_extra_params)


missing from the combined function

alorenzo175 · 2020-05-27T21:20:22Z

solarforecastarbiter/io/reference_observations/tests/test_common.py

+        test_json_site_file,
+        {'extra_parameters': params},
+    )
+    assert 'modeling_parameters' in new_site


make sure extra_parameters are updated

lboeman changed the title ~~Adjust srml~~ Adjust srml, Add reference obs documentation May 13, 2020

lboeman force-pushed the adjust-srml branch 2 times, most recently from 2dc1f9c to 6f63ce5 Compare May 14, 2020 17:41

lboeman requested a review from wholmgren May 14, 2020 17:48

wholmgren added this to the 1.0 release candidate milestone May 14, 2020

wholmgren added enhancement New feature or request IO Issue pertains to data IO bug Something isn't working labels May 14, 2020

wholmgren reviewed May 15, 2020

View reviewed changes

wholmgren reviewed May 20, 2020

View reviewed changes

lboeman force-pushed the adjust-srml branch from 5f264ed to 666d03d Compare May 20, 2020 16:58

wholmgren approved these changes May 27, 2020

View reviewed changes

alorenzo175 modified the milestones: 1.0 release candidate, 1.0 beta 6 May 27, 2020

lboeman and others added 15 commits May 27, 2020 11:44

remove non-reporting sites

9d80c28

temp commit for rebase

4be09a7

add reference obs json files to gitignore

5990695

declare pv srml sites and obs in json

5098c74

add description of reference_observation system

03d5d35

edits, add note about API keys

97b55ce

add tests for new code

c3f5b1d

add missing timezone field in json, differentiate between 15 and 30 d…

8778278

…egree tilt portland pv site

more test coverage

3d49fce

Update docs/source/reference-observations.rst

1d41714

Co-authored-by: Will Holmgren <[email protected]>

Update docs/source/reference-observations.rst

23b59ab

Co-authored-by: Will Holmgren <[email protected]>

Update docs/source/reference-observations.rst

e635af5

Co-authored-by: Will Holmgren <[email protected]>

Update docs/source/whatsnew/1.0.0rc1.rst

709cde3

Co-authored-by: Will Holmgren <[email protected]>

Update docs/source/whatsnew/1.0.0rc1.rst

ccd603b

Co-authored-by: Will Holmgren <[email protected]>

Update docs/source/reference-observations.rst

c1bbd58

Co-authored-by: Will Holmgren <[email protected]>

lboeman added 5 commits May 27, 2020 11:44

update reference obs docs, fix build warnings, reference classes in s…

f4e82cb

…rml.py docstrings

fix mislabeled and broken tests

4e5ce66

flake8

00806ad

factor out json modeling params loading to common

5059a65

fix rebase error

24ea737

lboeman force-pushed the adjust-srml branch from 5eed5dc to 24ea737 Compare May 27, 2020 18:48

use inspect to specify file relative to executing test

dd4c693

lboeman force-pushed the adjust-srml branch from 06d8d9b to dd4c693 Compare May 27, 2020 19:49

lboeman added 2 commits May 27, 2020 13:27

exclude reference obs test data from gitignore

4604136

change name of non-plant test

d024d51

alorenzo175 reviewed May 27, 2020

View reviewed changes

lboeman added 2 commits May 27, 2020 14:41

replace extra_parameters update in parameter update function

647c88a

flake8

ab34a16

alorenzo175 merged commit fb643db into SolarArbiter:master May 27, 2020

lboeman deleted the adjust-srml branch August 16, 2021 19:54

		logger = logging.getLogger('reference_data')


		def adjust_site_parameters(site):

Adjust srml, Add reference obs documentation #442

Adjust srml, Add reference obs documentation #442

Uh oh!

Conversation

lboeman commented May 13, 2020 • edited by wholmgren Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wholmgren left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wholmgren May 15, 2020

Choose a reason for hiding this comment

Uh oh!

alorenzo175 May 27, 2020

Choose a reason for hiding this comment

Uh oh!

wholmgren commented May 15, 2020

Uh oh!

wholmgren May 20, 2020

Choose a reason for hiding this comment

Uh oh!

wholmgren left a comment

Choose a reason for hiding this comment

Uh oh!

lboeman commented May 27, 2020

Uh oh!

alorenzo175 commented May 27, 2020

Uh oh!

alorenzo175 May 27, 2020

Choose a reason for hiding this comment

Uh oh!

alorenzo175 May 27, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lboeman commented May 13, 2020 •

edited by wholmgren

Loading