pvoutput.get_status throws error: NotImplementedError

[JulianBriggs]

Describe the bug
pvoutput.get_status throws error: NotImplementedError: file structure not yet supported

To Reproduce
Steps to reproduce the behavior:
pv_system_id = 39266
date = '2022-01-13'
status = self.pv.get_status(pv_system_id, date=date)
See error
NotImplementedError: file structure not yet supported

Expected behavior
Download pv_system status without error

Additional context
I'm seeing pvoutput error: NotImplementedError: file structure not yet supported
when requesting status of system_id 29291: Requesting system status for 2022-01-13
See appended. DEBUG oputput and stack traces.

I have seen a similar error with another system
pvoutput get_status 209 INFO system_id 39266: Requesting system status for 2022-01-13

2022-01-13 15:36:13,539 pvoutput get_status 209 INFO system_id 39266: Requesting system status for 2022-01-13
2022-01-13 15:36:13,543 connectionpool _new_conn 1001 DEBUG Starting new HTTPS connection (1): pvoutput.org:443
2022-01-13 15:36:14,063 connectionpool _make_request 456 DEBUG https://pvoutput.org:443 "GET /service/r2/getstatus.jsp?d=20220113&h=1&limit=288&ext=0&sid1=39266 HTTP/1.1" 200 None
2022-01-13 15:36:14,064 utils _get_response 81 DEBUG response: status_code=200; headers={'Date': 'Thu, 13 Jan 2022 15:36:13 GMT', 'Set-Cookie': 'JSESSIONID=us9myf32fu8ndpm2h7qwsc5k;Path=/; Secure', 'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'X-Rate-Limit-Remaining': '230', 'X-Rate-Limit-Limit': '300', 'X-Rate-Limit-Reset': '1642089600', 'Content-Type': 'text/plain', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding, User-Agent', 'Connection': 'close', 'Server': 'Jetty(7.6.17.v20150415)', 'Strict-Transport-Security': 'max-age=15768000'}
2022-01-13 15:36:14,064 pvoutput _set_rate_limit_params 928 DEBUG {'rate_limit_remaining': 230, 'rate_limit_total': 300, 'rate_limit_reset_time': Timestamp('2022-01-13 16:00:00+0000', tz='UTC')}
Traceback (most recent call last):
File "/home/ph1jb/git/pvoutput/src/downloader.py", line 240, in
nsystems = MAIN.run(sys.argv[1:])
File "/home/ph1jb/git/pvoutput/src/downloader.py", line 223, in run
nsystems = self.insert_system_statuses(database_handler, date, pv_system_ids)
File "/home/ph1jb/git/pvoutput/src/downloader.py", line 194, in insert_system_statuses
status = self.download_system_status(pv_system_id, date)
File "/home/ph1jb/git/pvoutput/src/downloader.py", line 154, in download_system_status
status = self.pv.get_status(pv_system_id, date=date)
File "/home/ph1jb/git/pvoutput/.venv/src/pvoutput/pvoutput/pvoutput.py", line 257, in get_status
pv_system_status = pd.read_csv(
File "/home/ph1jb/git/pvoutput/.venv/lib/python3.8/site-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/home/ph1jb/git/pvoutput/.venv/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 586, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/ph1jb/git/pvoutput/.venv/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 488, in _read
return parser.read(nrows)
File "/home/ph1jb/git/pvoutput/.venv/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 1047, in read
index, columns, col_dict = self._engine.read(nrows)
File "/home/ph1jb/git/pvoutput/.venv/lib/python3.8/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 261, in read
raise NotImplementedError("file structure not yet supported")
NotImplementedError: file structure not yet supported

2022-01-13 14:34:06,763 pvoutput get_status 209 INFO system_id 29291: Requesting system status for 2022-01-13
2022-01-13 14:34:06,772 connectionpool _new_conn 1001 DEBUG Starting new HTTPS connection (1): pvoutput.org:443
2022-01-13 14:34:07,310 connectionpool _make_request 456 DEBUG https://pvoutput.org:443 "GET /service/r2/getstatus.jsp?d=20220113&h=1&limit=288&ext=0&sid1=29291 HTTP/1.1" 200 None
2022-01-13 14:34:07,312 utils _get_response 81 DEBUG response: status_code=200; headers={'Date': 'Thu, 13 Jan 2022 14:34:07 GMT', 'Set-Cookie': 'JSESSIONID=10fqtsfp2518apqdw1dfmpdbl;Path=/; Secure', 'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'X-Rate-Limit-Remaining': '163', 'X-Rate-Limit-Limit': '300', 'X-Rate-Limit-Reset': '1642086000', 'Content-Type': 'text/plain', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding, User-Agent', 'Connection': 'close', 'Server': 'Jetty(7.6.17.v20150415)', 'Strict-Transport-Security': 'max-age=15768000'}
2022-01-13 14:34:07,312 pvoutput _set_rate_limit_params 928 DEBUG {'rate_limit_remaining': 163, 'rate_limit_total': 300, 'rate_limit_reset_time': Timestamp('2022-01-13 15:00:00+0000', tz='UTC')}
Traceback (most recent call last):
File "/home/ph1jb/git/pvoutput/src/downloader.py", line 262, in
nsystems = MAIN.run(sys.argv[1:])
File "/home/ph1jb/git/pvoutput/src/downloader.py", line 242, in run
nsystems = self.insert_system_statuses(database_handler, date, pv_system_ids)
File "/home/ph1jb/git/pvoutput/src/downloader.py", line 195, in insert_system_statuses
status = self.download_system_status(pv_system_id, date)
File "/home/ph1jb/git/pvoutput/src/downloader.py", line 155, in download_system_status
status = self.pv.get_status(pv_system_id, date=date)
File "/home/ph1jb/git/pvoutput/.venv/src/pvoutput/pvoutput/pvoutput.py", line 257, in get_status
pv_system_status = pd.read_csv(
File "/home/ph1jb/git/pvoutput/.venv/lib/python3.8/site-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/home/ph1jb/git/pvoutput/.venv/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 586, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/ph1jb/git/pvoutput/.venv/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 488, in _read
return parser.read(nrows)
File "/home/ph1jb/git/pvoutput/.venv/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 1047, in read
index, columns, col_dict = self._engine.read(nrows)
File "/home/ph1jb/git/pvoutput/.venv/lib/python3.8/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 261, in read
raise NotImplementedError("file structure not yet supported")
NotImplementedError: file structure not yet supported

[peterdudfield]

It's this specific to this one system_id?

[JulianBriggs]

Hi peter, No. I query for 225 systems in the Balkans and have noticed 2 systems throwing this error. See end of github issue. There may be more. Can you query the same system ids to reproduce the error? Best regards Julian Sent from phone

…

On Fri, 14 Jan 2022, 15:23 Peter Dudfield, ***@***.***> wrote: It's this specific to this one system_id? — Reply to this email directly, view it on GitHub <#84 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APAUAOI2LFVZXXKSD7WWWZ3UWA5WPANCNFSM5L6XG3MA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[peterdudfield]

Sorry I havent looked into this yet.
Does this happen in the UK too?

[JulianBriggs]

Hi Peter, I've not downloaded UK data yet. The error arises from at least one system having the wrong number of fields in its status. Happy to share the code of the lightweight PVOutput downloader I have developed. best regards Julian

…

On Thu, 3 Feb 2022 at 16:16, Peter Dudfield ***@***.***> wrote: Sorry I havent looked into this yet. Does this happen in the UK too? — Reply to this email directly, view it on GitHub <#84 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APAUAOO2UINA464GBI5BEYTUZKS6JANCNFSM5L6XG3MA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Julian Briggs Research Software Developer Sheffield Solar Physics & Astronomy University of Sheffield In line with government advice, I am currently working remotely. My preferred method of contact is email. My normal working hours are 9:30-5 Mon-Thu

[JulianBriggs]

Hi Peter, here's an example warning about the PVOutput system with Wrong number of values in row 2022-02-04T09:30:23.838688674Z 2022-02-04 09:30:23,838 downloaderpvoutput download_insert_readings 416 WARNING Bad status: *Wrong number of values in row* (12), expected (11): ('20220204', '09:10', '700', '0.156', '3459', '0', '0.000', 'NaN', 'NaN', '9.5', '2', '227.9').. *pv_system_id: 39266, date: 2022-02-04*. I trust you can reproduce this and catch the PVOutput error: *Bad status* more anon best Julian On Thu, 3 Feb 2022 at 19:30, Julian Briggs ***@***.***> wrote:

…

Hi Peter, I've not downloaded UK data yet. The error arises from at least one system having the wrong number of fields in its status. Happy to share the code of the lightweight PVOutput downloader I have developed. best regards Julian On Thu, 3 Feb 2022 at 16:16, Peter Dudfield ***@***.***> wrote: > Sorry I havent looked into this yet. > Does this happen in the UK too? > > — > Reply to this email directly, view it on GitHub > <#84 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/APAUAOO2UINA464GBI5BEYTUZKS6JANCNFSM5L6XG3MA> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > -- Julian Briggs Research Software Developer Sheffield Solar Physics & Astronomy University of Sheffield In line with government advice, I am currently working remotely. My preferred method of contact is email. My normal working hours are 9:30-5 Mon-Thu

-- Julian Briggs Research Software Developer Sheffield Solar Physics & Astronomy University of Sheffield In line with government advice, I am currently working remotely. My preferred method of contact is email. My normal working hours are 9:30-5 Mon-Thu