Combining multiple WRF history files with combine_files ('kernel died')

[tlsw231]

I have around ~3000 WRF history files which I want to combine.

Pangeo has a useful link but the example there has one file with all the temporal snapshots, which is different from my case.

The instructions here seem more relevant and when I try to follow the example there the following code works if I include only a few history files (e.g., files[0:5]) in the append step. The variable pbl is then read in correctly. But If I include all the files the append step fails with a 'kernel died' message. Is there a way to create the wrflist variable without append? In the examples at the link, the entries in wrflist are written out manually but that is not an option with ~3000 files.

# files is just a list of strings with all the WRF history filenames
wrflist=[]
[ wrflist.append(Dataset(i)) for i in files[0:5] ]  # this step works but fails if I include all the filenames
pbl = wrf.combine_files(wrflist,'PBLH',wrf.ALL_TIMES,method='cat')

Any ideas on why the append step is failing?

[keewis]

Is there a reason why you use list.append in a list comprehension? I'd use this instead:

wrflist = [Dataset(path) for path in files[0:5]]

not sure if that will fix your issue, though

[tlsw231]

I tried that and seem to be getting the same 'kernel died unexpectedly' message if I include all the filenames. I am trying this with 384 gb memory.

[dcherian]

Can you try xwrf instead? You'll probably get better advice asking questions there

[tlsw231]

I am having trouble getting it to work even with just one file, xr.open_dataset(files[0],chunks={}).xwrf.postprocess() gives me:

arg must be a string, datetime, list, tuple, 1-d array, or Series

[tlsw231]

If I use .xwrf.postprocess(decode_times=False), the command goes through but then I lose the time information.

[tlsw231]

I confirmed the command below in xwrf/postprocess.py works as it should and returns a datetime64 object.

pd.to_datetime(xr.open_dataset(files[0]).Times.data.astype('str'),format='%Y-%m-%d_%H:%M:%S')

[andersy005]

this is more of an xwrf issue than an xarray one. Do you mind asking on the xwrf discussion forum: https://github.com/ncar-xdev/xwrf/discussions ?

[tlsw231]

Oh shoot, totally forgot about migrating the discussion there. Will do that.

[andersy005]

For anyone interested, the answer resides here: xarray-contrib/xwrf#78 (comment)