-
Notifications
You must be signed in to change notification settings - Fork 21
Adjust srml, Add reference obs documentation #442
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
9d80c28
remove non-reporting sites
lboeman 4be09a7
temp commit for rebase
lboeman 5990695
add reference obs json files to gitignore
lboeman 5098c74
declare pv srml sites and obs in json
lboeman 03d5d35
add description of reference_observation system
lboeman 97b55ce
edits, add note about API keys
lboeman c3f5b1d
add tests for new code
lboeman 8778278
add missing timezone field in json, differentiate between 15 and 30 d…
lboeman 3d49fce
more test coverage
lboeman 1d41714
Update docs/source/reference-observations.rst
lboeman 23b59ab
Update docs/source/reference-observations.rst
lboeman e635af5
Update docs/source/reference-observations.rst
lboeman 709cde3
Update docs/source/whatsnew/1.0.0rc1.rst
lboeman ccd603b
Update docs/source/whatsnew/1.0.0rc1.rst
lboeman c1bbd58
Update docs/source/reference-observations.rst
lboeman f4e82cb
update reference obs docs, fix build warnings, reference classes in s…
lboeman 4e5ce66
fix mislabeled and broken tests
lboeman 00806ad
flake8
lboeman 5059a65
factor out json modeling params loading to common
lboeman 24ea737
fix rebase error
lboeman dd4c693
use inspect to specify file relative to executing test
lboeman 4604136
exclude reference obs test data from gitignore
lboeman d024d51
change name of non-plant test
lboeman 647c88a
replace extra_parameters update in parameter update function
lboeman ab34a16
flake8
lboeman File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,149 @@ | ||
| .. curentmodule: solarforecastarbiter.io.reference_observations | ||
|
|
||
| ###################### | ||
| Reference Observations | ||
| ###################### | ||
|
|
||
| Overview | ||
| ======== | ||
|
|
||
| The Solar Forecast Arbiter imports reference observation data from multiple | ||
| measurement networks. All of the logic for creating the appropriate Solar | ||
| Forecast Arbiter sites and observations and updating observations with new data | ||
| from the network can be found in the | ||
| :py:mod:`solarforecastarbiter.io.reference_observations` subpackage. Code for | ||
| retrieving data from the network's APIs are spread between the | ||
| :py:mod:`solarforecastarbiter.io.fetch` subpackage, and the | ||
| `pvlib python <https://pvlib-python.readthedocs.io/en/stable/index.html>`_ *iotools* | ||
| module. | ||
|
|
||
| A list of these networks and their Solar Forecast Arbiter modules can be found | ||
| in the `Available Network Handlers`_ section. A map of all of the sites | ||
| available in the reference dataset can be found on the | ||
| `Solar Forecast Arbiter project website <https://solarforecastarbiter.org/referencedata/>`_. | ||
|
|
||
|
|
||
| Structure | ||
| ========= | ||
| The :py:mod:`solarforecastarbiter.io.reference_observations` subpackage | ||
| contains python modules and data files in JSON and CSV format. | ||
|
|
||
| Data Files | ||
| ---------- | ||
| * `sfa_reference_sites.csv` | ||
| The master list of reference sites. See the comment at the top of this file | ||
| for descriptions of its fields. The file contains extra fields that are not | ||
| found in the Solar Forecast Arbiter API schema for Sites. These fields are | ||
| for use with the source network's API and are stored in the | ||
| `extra_parameters` field when the site is created for use in subsequent | ||
| updates. | ||
|
|
||
| * `<network>_reference_sites.json` | ||
| Network-specific files containing site and observation metadata in the Solar | ||
| Forecast Arbiter API's JSON format. These are used when the master CSV does | ||
| not contain all of the columns needed to accurately define a site or | ||
| observation. | ||
|
|
||
| Modules | ||
| ------- | ||
| * :py:mod:`solarforecastarbiter.io.reference_observations.reference_data` | ||
| This module coordinates the initialization and update process. It also | ||
| contains the `NETWORKHANDLER_MAP` dictionary, which maps network names to | ||
| the correct `Network Handlers`_. The functions in the module are utilized by | ||
| the CLI `referencedata` command. | ||
|
|
||
| * :py:mod:`solarforecastarbiter.io.reference_observations.common` | ||
| The `common` module contains utility functions for use throughout the | ||
| `reference_data` subpackage. It has useful functions for converting | ||
| external data into Solar Forecast Arbiter Datamodel objects and | ||
| network-agnostic utilities for preparing and posting data to the Solar | ||
| Forecast Arbiter API. Most `Network Handlers`_ rely heavily on these | ||
| functions. | ||
|
|
||
| Network Handlers | ||
| **************** | ||
| Network Handlers are network specific modules that implement a handful of | ||
| functions with a common interface. See | ||
| :py:mod:`solarforecastarbiter.io.reference_observations.surfrad` for an | ||
| example. | ||
|
|
||
| The required network handler functions are: | ||
|
|
||
| * `initialize_site_observations(api, site)` | ||
| Create an observation at the site for each variable available from the | ||
| network. | ||
|
|
||
| * api: :py:class:`solarforecastarbiter.io.api.APISession` | ||
| * site: :py:class:`solarforecastarbiter.datamodel.Site` | ||
|
|
||
|
|
||
| * `initialize_site_forecasts(api, site)` | ||
| Create a forecast for each observation at the site. | ||
|
|
||
| * api: :py:class:`solarforecastarbiter.io.api.APISession` | ||
| * site: :py:class:`solarforecastarbiter.datamodel.Site` | ||
|
|
||
|
|
||
| * `update_observation_data(api, sites, observations, start, end)` | ||
| Retrieve data from the network then format and post it to each observation | ||
| at the site. | ||
|
|
||
| * api: :py:class:`solarforecastarbiter.io.api.APISession` | ||
| * sites: list of :py:class:`solarforecastarbiter.datamodel.Site` | ||
| * observations: list of :py:class:`solarforecastarbiter.datamodel.Observation` | ||
| * start: datetime | ||
| * end: datetime | ||
|
|
||
|
|
||
| * (optional) `adjust_site_parameters(site)` | ||
| In instances where the master site CSV does not contain enough metadata about | ||
| the site, (e.g. when a PV plant requires `modeling_parameters`) this function | ||
| may be used to update the site metadata before it is posted to the API. | ||
|
|
||
| * site: dict | ||
|
|
||
|
|
||
| Available Network Handlers | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
| * SURFRAD: NOAA Surface Radiation Budget Network | ||
| https://www.esrl.noaa.gov/gmd/grad/surfrad/ | ||
|
|
||
| :py:mod:`solarforecastarbiter.io.reference_observations.surfrad` | ||
|
|
||
| * SOLRAD: NOAA SOLRAD Network | ||
| https://www.esrl.noaa.gov/gmd/grad/solrad/index.html | ||
|
|
||
| :py:mod:`solarforecastarbiter.io.reference_observations.solrad` | ||
|
|
||
| * CRN: NOAA U.S. Climate Reference Network | ||
| https://www.ncdc.noaa.gov/crn/ | ||
|
|
||
| :py:mod:`solarforecastarbiter.io.reference_observations.crn` | ||
|
|
||
| * NREL MIDC: National Renewable Energy Laboratory Measurement and Instrumentation Data Center | ||
| https://midcdmz.nrel.gov/ | ||
|
|
||
| :py:mod:`solarforecastarbiter.io.reference_observations.midc` | ||
|
|
||
| * UO SRML: University of Oregon Solar Radiation Monitoring Laboratory | ||
| http://solardat.uoregon.edu/ | ||
|
|
||
| :py:mod:`solarforecastarbiter.io.reference_observations.srml` | ||
|
|
||
| * DOE RTC: DOE Regional Test Centers for Solar Technologies\* | ||
| https://pv-dashboard.sandia.gov/ | ||
|
|
||
| :py:mod:`solarforecastarbiter.io.reference_observations.rtc` | ||
|
|
||
| * DOE ARM: DOE Atmospheric Radiation Measurement\* | ||
| https://www.arm.gov/ | ||
|
|
||
| :py:mod:`solarforecastarbiter.io.reference_observations.arm` | ||
|
|
||
| * NREL PVDAQ: National Renewable Energy Laboratory PV Data Acquisition\* | ||
| https://developer.nrel.gov/docs/solar/pvdaq-v3/ | ||
|
|
||
| :py:mod:`solarforecastarbiter.io.reference_observations.pvdaq` | ||
|
|
||
| \* Requesting data from these networks requires a valid api key for their | ||
| associated api. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
missing from the combined function