Skip to content

Allow read_surfrad to parse http files #1459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jul 19, 2022
Merged

Allow read_surfrad to parse http files #1459

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
d7deb80
Add cams.get_cams_radiation function
Feb 22, 2021
510f08e
Revert "Add cams.get_cams_radiation function"
Feb 22, 2021
2b395ee
Allow parsing of http files
AdamRJensen May 19, 2022
b297000
Add test for https file
AdamRJensen May 19, 2022
fa8b9e4
Merge branch 'master' into surfrad_http
AdamRJensen May 19, 2022
b313c64
Squashed commit of the following:
AdamRJensen May 19, 2022
3d63ac3
Revert "Squashed commit of the following:"
AdamRJensen May 19, 2022
ec55285
Merge remote-tracking branch 'upstream/master' into surfrad_http
AdamRJensen May 19, 2022
5c01bc4
Update whatsnew
AdamRJensen May 19, 2022
8efab47
Update read_surfrad documentation
AdamRJensen May 25, 2022
3d72d2c
Merge branch 'master' into surfrad_http
AdamRJensen Jul 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/sphinx/source/whatsnew/v0.9.2.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ Deprecations

Enhancements
~~~~~~~~~~~~
* :py:func:`pvlib.iotools.read_surfrad` now also accepts remote files
with https links in addition to files on the SURFRAD FTP server
(:pull:`1459`)


Bug fixes
~~~~~~~~~
Expand Down Expand Up @@ -37,6 +41,7 @@ Requirements

Contributors
~~~~~~~~~~~~
* Adam R. Jensen (:ghuser:`AdamRJensen`)
* Naman Priyadarshi (:ghuser:`Naman-Priyadarshi`)
* Chencheng Luo (:ghuser:`roger-lcc`)
* Prajwal Borkar (:ghuser:`PrajwalBorkar`)
* Prajwal Borkar (:ghuser:`PrajwalBorkar`)
2 changes: 1 addition & 1 deletion pvlib/iotools/surfrad.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ def read_surfrad(filename, map_variables=True):
.. [2] NOAA SURFRAD Data Archive
`SURFRAD Archive <ftp://aftp.cmdl.noaa.gov/data/radiation/surfrad/>`_
"""
if str(filename).startswith('ftp'):
if str(filename).startswith('ftp') or str(filename).startswith('http'):
req = Request(filename)
response = urlopen(req)
file_buffer = io.StringIO(response.read().decode(errors='ignore'))
Expand Down
13 changes: 13 additions & 0 deletions pvlib/tests/iotools/test_surfrad.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
testfile = DATA_DIR / 'surfrad-slv16001.dat'
network_testfile = ('ftp://aftp.cmdl.noaa.gov/data/radiation/surfrad/'
'Alamosa_CO/2016/slv16001.dat')
https_testfile = ('https://gml.noaa.gov/aftp/data/radiation/surfrad/'
'Alamosa_CO/2016/slv16001.dat')


@pytest.mark.remote_data
Expand All @@ -19,6 +21,17 @@ def test_read_surfrad_network():
assert local_data.equals(network_data)


@pytest.mark.remote_data
@pytest.mark.flaky(reruns=RERUNS, reruns_delay=RERUNS_DELAY)
def test_read_surfrad_https():
# Test reading of https files.
# If this test begins failing, SURFRAD's data structure or data
# archive may have changed.
local_data, _ = surfrad.read_surfrad(testfile)
network_data, _ = surfrad.read_surfrad(https_testfile)
assert local_data.equals(network_data)


def test_read_surfrad_columns_no_map():
data, _ = surfrad.read_surfrad(testfile, map_variables=False)
assert 'zen' in data.columns
Expand Down