Skip to content

Commit fb643db

Browse files
lboemanwholmgren
lboeman
and
wholmgren
authored
Adjust srml, Add reference obs documentation (#442)
* remove non-reporting sites * temp commit for rebase * add reference obs json files to gitignore * declare pv srml sites and obs in json * add description of reference_observation system * edits, add note about API keys * add tests for new code * add missing timezone field in json, differentiate between 15 and 30 degree tilt portland pv site * more test coverage * Update docs/source/reference-observations.rst Co-authored-by: Will Holmgren <[email protected]> * Update docs/source/reference-observations.rst Co-authored-by: Will Holmgren <[email protected]> * Update docs/source/reference-observations.rst Co-authored-by: Will Holmgren <[email protected]> * Update docs/source/whatsnew/1.0.0rc1.rst Co-authored-by: Will Holmgren <[email protected]> * Update docs/source/whatsnew/1.0.0rc1.rst Co-authored-by: Will Holmgren <[email protected]> * Update docs/source/reference-observations.rst Co-authored-by: Will Holmgren <[email protected]> * update reference obs docs, fix build warnings, reference classes in srml.py docstrings * fix mislabeled and broken tests * flake8 * factor out json modeling params loading to common * fix rebase error * use inspect to specify file relative to executing test * exclude reference obs test data from gitignore * change name of non-plant test * replace extra_parameters update in parameter update function * flake8 Co-authored-by: Will Holmgren <[email protected]>
1 parent d123e88 commit fb643db

File tree

13 files changed

+993
-36
lines changed

13 files changed

+993
-36
lines changed

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,9 @@ venv.bak/
112112
.vscode/
113113

114114
*.json
115+
!solarforecastarbiter/io/reference_observations/*.json
116+
!solarforecastarbiter/io/reference_observations/tests/data/*.json
117+
115118
*.html
116119
!solarforecastarbiter/reports/templates/*.html
117120
*.md

‎docs/source/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ for more information about the Solar Forecast Arbiter project.
1616

1717
installation
1818
contributing
19+
reference-observations
1920
reference-forecasts
2021
api
2122
cli

‎docs/source/reference-observations.rst‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
.. curentmodule: solarforecastarbiter.io.reference_observations
2+
3+
######################
4+
Reference Observations
5+
######################
6+
7+
Overview
8+
========
9+
10+
The Solar Forecast Arbiter imports reference observation data from multiple
11+
measurement networks. All of the logic for creating the appropriate Solar
12+
Forecast Arbiter sites and observations and updating observations with new data
13+
from the network can be found in the
14+
:py:mod:`solarforecastarbiter.io.reference_observations` subpackage. Code for
15+
retrieving data from the network's APIs are spread between the
16+
:py:mod:`solarforecastarbiter.io.fetch` subpackage, and the
17+
`pvlib python <https://pvlib-python.readthedocs.io/en/stable/index.html>`_ *iotools*
18+
module.
19+
20+
A list of these networks and their Solar Forecast Arbiter modules can be found
21+
in the `Available Network Handlers`_ section. A map of all of the sites
22+
available in the reference dataset can be found on the
23+
`Solar Forecast Arbiter project website <https://solarforecastarbiter.org/referencedata/>`_.
24+
25+
26+
Structure
27+
=========
28+
The :py:mod:`solarforecastarbiter.io.reference_observations` subpackage
29+
contains python modules and data files in JSON and CSV format.
30+
31+
Data Files
32+
----------
33+
* `sfa_reference_sites.csv`
34+
The master list of reference sites. See the comment at the top of this file
35+
for descriptions of its fields. The file contains extra fields that are not
36+
found in the Solar Forecast Arbiter API schema for Sites. These fields are
37+
for use with the source network's API and are stored in the
38+
`extra_parameters` field when the site is created for use in subsequent
39+
updates.
40+
41+
* `<network>_reference_sites.json`
42+
Network-specific files containing site and observation metadata in the Solar
43+
Forecast Arbiter API's JSON format. These are used when the master CSV does
44+
not contain all of the columns needed to accurately define a site or
45+
observation.
46+
47+
Modules
48+
-------
49+
* :py:mod:`solarforecastarbiter.io.reference_observations.reference_data`
50+
This module coordinates the initialization and update process. It also
51+
contains the `NETWORKHANDLER_MAP` dictionary, which maps network names to
52+
the correct `Network Handlers`_. The functions in the module are utilized by
53+
the CLI `referencedata` command.
54+
55+
* :py:mod:`solarforecastarbiter.io.reference_observations.common`
56+
The `common` module contains utility functions for use throughout the
57+
`reference_data` subpackage. It has useful functions for converting
58+
external data into Solar Forecast Arbiter Datamodel objects and
59+
network-agnostic utilities for preparing and posting data to the Solar
60+
Forecast Arbiter API. Most `Network Handlers`_ rely heavily on these
61+
functions.
62+
63+
Network Handlers
64+
****************
65+
Network Handlers are network specific modules that implement a handful of
66+
functions with a common interface. See
67+
:py:mod:`solarforecastarbiter.io.reference_observations.surfrad` for an
68+
example.
69+
70+
The required network handler functions are:
71+
72+
* `initialize_site_observations(api, site)`
73+
Create an observation at the site for each variable available from the
74+
network.
75+
76+
* api: :py:class:`solarforecastarbiter.io.api.APISession`
77+
* site: :py:class:`solarforecastarbiter.datamodel.Site`
78+
79+
80+
* `initialize_site_forecasts(api, site)`
81+
Create a forecast for each observation at the site.
82+
83+
* api: :py:class:`solarforecastarbiter.io.api.APISession`
84+
* site: :py:class:`solarforecastarbiter.datamodel.Site`
85+
86+
87+
* `update_observation_data(api, sites, observations, start, end)`
88+
Retrieve data from the network then format and post it to each observation
89+
at the site.
90+
91+
* api: :py:class:`solarforecastarbiter.io.api.APISession`
92+
* sites: list of :py:class:`solarforecastarbiter.datamodel.Site`
93+
* observations: list of :py:class:`solarforecastarbiter.datamodel.Observation`
94+
* start: datetime
95+
* end: datetime
96+
97+
98+
* (optional) `adjust_site_parameters(site)`
99+
In instances where the master site CSV does not contain enough metadata about
100+
the site, (e.g. when a PV plant requires `modeling_parameters`) this function
101+
may be used to update the site metadata before it is posted to the API.
102+
103+
* site: dict
104+
105+
106+
Available Network Handlers
107+
^^^^^^^^^^^^^^^^^^^^^^^^^^
108+
* SURFRAD: NOAA Surface Radiation Budget Network
109+
https://www.esrl.noaa.gov/gmd/grad/surfrad/
110+
111+
:py:mod:`solarforecastarbiter.io.reference_observations.surfrad`
112+
113+
* SOLRAD: NOAA SOLRAD Network
114+
https://www.esrl.noaa.gov/gmd/grad/solrad/index.html
115+
116+
:py:mod:`solarforecastarbiter.io.reference_observations.solrad`
117+
118+
* CRN: NOAA U.S. Climate Reference Network
119+
https://www.ncdc.noaa.gov/crn/
120+
121+
:py:mod:`solarforecastarbiter.io.reference_observations.crn`
122+
123+
* NREL MIDC: National Renewable Energy Laboratory Measurement and Instrumentation Data Center
124+
https://midcdmz.nrel.gov/
125+
126+
:py:mod:`solarforecastarbiter.io.reference_observations.midc`
127+
128+
* UO SRML: University of Oregon Solar Radiation Monitoring Laboratory
129+
http://solardat.uoregon.edu/
130+
131+
:py:mod:`solarforecastarbiter.io.reference_observations.srml`
132+
133+
* DOE RTC: DOE Regional Test Centers for Solar Technologies\*
134+
https://pv-dashboard.sandia.gov/
135+
136+
:py:mod:`solarforecastarbiter.io.reference_observations.rtc`
137+
138+
* DOE ARM: DOE Atmospheric Radiation Measurement\*
139+
https://www.arm.gov/
140+
141+
:py:mod:`solarforecastarbiter.io.reference_observations.arm`
142+
143+
* NREL PVDAQ: National Renewable Energy Laboratory PV Data Acquisition\*
144+
https://developer.nrel.gov/docs/solar/pvdaq-v3/
145+
146+
:py:mod:`solarforecastarbiter.io.reference_observations.pvdaq`
147+
148+
\* Requesting data from these networks requires a valid api key for their
149+
associated api.

‎docs/source/whatsnew/1.0.0rc1.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,13 +49,17 @@ Enhancements
4949
'DAYTIME INTERPOLATED VALUES'``. (:issue:`124`)
5050
* A public Docker image is available, and Github Actions use this image for
5151
continuous integration tests (:pull:`446`)
52+
* Adjusted University of Oregon Solar Radiation Measurment Laboratory (UO SRML)
53+
to include available PV power data. (:pull:`442`)
5254

5355
Bug fixes
5456
~~~~~~~~~
5557
* Fix incorrect ordering of months and weekdays in metrics plots.
5658
(:issue:`428`) (:pull:`430`)
5759
* Ensure data is sorted from reference data sources before slicing and
5860
posting to the API (:pull:`435`)
61+
* Remove UO SMRL sites from reference dataset that have not reported recent
62+
data. (:issue:`436`) (:pull:`442`)
5963

6064

6165
Contributors

‎solarforecastarbiter/io/reference_observations/common.py‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -629,3 +629,32 @@ def create_forecasts(api, site, variables, templates):
629629
persist_created = create_persistence_forecasts(
630630
api, site, variables, nwp_templates)
631631
return nwp_created + persist_created
632+
633+
634+
def apply_json_site_parameters(json_sitefile, site):
635+
"""Updates site metadata with modeling parameters found in a json file.
636+
637+
Parameters
638+
----------
639+
json_sitefile: str
640+
Absolute path of a json file with a 'sites' key containing a list of
641+
sites in the Solar Forecast Arbiter JSON format.
642+
site: dict
643+
644+
Returns
645+
-------
646+
dict
647+
Copy of inputs plus a new key 'modeling_parameters'.
648+
"""
649+
with open(json_sitefile) as fp:
650+
sites_metadata = json.load(fp)['sites']
651+
site_api_id = str(site['extra_parameters']['network_api_id'])
652+
for site_metadata in sites_metadata:
653+
site_extra_params = json.loads(site_metadata['extra_parameters'])
654+
if str(site_extra_params['network_api_id']) == site_api_id:
655+
site_out = site.copy()
656+
site_out['modeling_parameters'] = site_metadata[
657+
'modeling_parameters']
658+
site_out['extra_parameters'].update(site_extra_params)
659+
return site_out
660+
return site

‎solarforecastarbiter/io/reference_observations/pvdaq.py‎

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -41,17 +41,7 @@ def adjust_site_parameters(site):
4141
--------
4242
solarforecastarbiter.io.reference_observations.site_df_to_dicts
4343
"""
44-
with open(DEFAULT_SITEFILE) as fp:
45-
sites_metadata = json.load(fp)['sites']
46-
site_api_id = int(site['extra_parameters']['network_api_id'])
47-
for site_metadata in sites_metadata:
48-
site_extra_params = json.loads(site_metadata['extra_parameters'])
49-
if site_extra_params['network_api_id'] == site_api_id:
50-
site_out = site.copy()
51-
site_out['modeling_parameters'] = site_metadata[
52-
'modeling_parameters']
53-
site_out['extra_parameters'].update(site_extra_params)
54-
return site_out
44+
return common.apply_json_site_parameters(DEFAULT_SITEFILE, site)
5545

5646

5747
def initialize_site_observations(api, site):

‎solarforecastarbiter/io/reference_observations/sfa_reference_sites.csv‎

Lines changed: 5 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -26,25 +26,22 @@
2626
#
2727
interval_length,name,latitude,longitude,elevation,network_api_id,network_api_abbreviation,timezone,attribution,network
2828
1,Seattle UW WA,47.653999999999996,-122.309,70,94291.0,ST,Etc/GMT+5,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
29-
1,Ashland OR,42.19,-122.7,595,94040.0,AS,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
29+
1,Ashland OR PV,42.19,-122.7,595,94040.0,AS,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
3030
1,UO Solar Awning Eugene OR,44.05,-123.07,150,94255.0,AW,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
31-
1,Kalapuya High School OR,45.0,-120.0,0,94259.0,KA,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
31+
1,Kalapuya High School OR PV,45.0,-120.0,0,94259.0,KA,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
3232
1,Salem OR,44.92100000000001,-123.01799999999999,62,94806.0,SA,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
33-
1,Portland OR PV,45.51,-122.69,70,94808.0,PS,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
33+
1,Portland OR PV 15 deg tilt,45.51,-122.69,70,94808.0(15),PS,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
34+
1,Portland OR PV 30 deg tilt,45.51,-122.69,70,94808.0,PS,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
3435
1,Cheney WA,47.49,-117.589,777,94158.0,CY,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
35-
1,Dillon MT,45.21,-112.64,1590,94145.0,DI,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
3636
1,Eugene OR,44.05,-123.07,150,94255.0,EU,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
3737
1,Burns OR,43.52,-119.02,1265,94170.0,BU,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
3838
5,Aberdeen ID,42.95,-112.78,1433,94174.0,AB,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
3939
1,Silver Lake OR,43.12,-121.06,1355,94249.0,SL,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
40-
1,Bend OR,44.06,-121.31,1124,94807.0,BD,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
41-
15,Moab UT,38.58,-109.54,1000,94102.0,MO,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
42-
15,Green River WY,41.46,-109.44,1000,94101.0,GR,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
40+
1,Bend OR PV,44.06,-121.31,1124,94807.0,BD,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
4341
15,Seattle WA,47.68,-122.25,20,94290.0,SE,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
4442
15,Forest Grove OR,45.55,-123.08,55,94008.0,FG,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
4543
15,Picabo ID,43.31,-114.17,1472,94172.0,PI,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
4644
15,Twin Falls ID,42.55,-114.35,1200,94171.0,TW,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
47-
15,Boise ID,43.62,-116.21,701,94182.0,BO,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
4845
15,Parma ID,43.8,-116.94,678,94173.0,PA,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
4946
15,Christmas Valley OR,43.24,-120.88,1325,94251.0,CH,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML
5047
15,Bend OR,43.99,-121.35,1100,94256.0,BE,Etc/GMT+8,"Peterson, J., and Vignola, F., 2017: Structure of a Comprehensive Solar Radiation Dataset. Proceedings of the ASES National Solar Conference 2017. doi: 10.18086/solar.2017.07.02",UO SRML

0 commit comments

Comments
 (0)