Implement clearsky filter

[mdeceglie]

This should be done in a new module filter.py

[mdeceglie]

Suggested that this is done withpvlib.clearsky.detect_clearsky, where the user supplies the detection parameters. We may also provide a tool for helping users identify appropriate parameters for their data.

[amnag]

Added clearsky_filter branch to rdtools

[amnag]

@danjruth, in the updated filtering module, is it necessary to use tz_convert('utc') in a couple of lines ?

[danjruth]

@amnag It doesn't look like it: the PVLib documentation suggests that it can handle timezone-aware times, and will localize to UTC when given a timezone-naive time.

Should we just get rid of the conversion, or instead check that the timezone is not naive? I'm not sure how the rest of rdtools deals with timezones, but it may be worthwhile to ensure here that the timezone is localized, since that'll affect the clearsky irradiance values used in the filtering.

[amnag]

We can check whether timezone info is available or not, if not we can set it to 'utc' ?
In get_clearsky_poa, this can be implemented as

if (clearsky.index.tzinfo is None):
times = clearsky.index.tz_convert('utc')

In remove_cloudy_times, this can be implemented as
if (clearsky.index.tzinfo is None):
irrad.index = irrad.index.tz_convert('utc')
clearsky = get_clearsky_irrad(system_loc, irrad.index, correct_bin_labelling=correct_bin_labelling)

[danjruth]

That works--I believe the second line should read times = clearsky.index.tz_localize('utc').

If it's not localized, would it be better to raise an error as opposed to assuming it should be in UTC? I could see forgetting to localize data before passing it to remove_cloudy_times, in which case it'd be helpful to have an error message.

[amnag]

@danjruth, you're right, my bad

[amnag]

I made the changes, can you please take a look and if possible test it ? I did test it and it works.

[danjruth]

Looks good. I tested it with the clearsky_test notebook, and it threw the error when I didn't localize the times. Also it worked fine when I converted the times to UTC before passing the data to the filtering functions, provided I changedstart_time and end_time appropriately.

I added the following to detect_cloudy_days to handle the case where there are no datapoints between start_time and end_time for a given day:

# call a day cloudy if there are no valid points to look at between start_time and end_time
        if (num_false+num_true)==0:
            clear_days[date] = False
        else:
            clear_days[date] = float(num_true)/float(num_false+num_true) >= thresh

[amnag]

@danjruth and @mdeceglie, can you please take a look at the updated filtering.py code in the clearsky_filter branch of rdtools? I had to make some changes in get_clearsky_irrad function because a collections.OrderedDict object was assigned to clearsky and this was creating problems with the remove_cloudy_times function.

[wholmgren]

your error might be related to pvlib/pvlib-python#326

I'll make a new release when I get a chance.

[cdeline]

@mdeceglie At this point can we combine the clearsky temperature with the clearsky_filter branch? Are these two branches accomplishing effectively the same workflow?