sorby does not forget the dimensions

[OttavioM]

Estimated community of xarray,

I would like to ask you what I can do in order to solve this problem that I have using sortby.
If I open a large dataset and then I sort it by a coordinate and then I store it in a new variable and try to load it, it seems that the dataset does not lose the old dimensions.

I open a very large dataset:
da = xr.open_mfdataset('GOW_CFS_Plataforma_1979.01_spc.nc')

>  <xarray.Dataset>
>  Dimensions:       (time: 248, station: 36455, string16: 16, frequency: 32,
>                    direction: 24)
> Coordinates:
>   * time          (time) datetime64[ns] 1979-01-01 ... 1979-01-31T21:00:00
>   * station       (station) int32 1 2 3 4 5 6 ... 36451 36452 36453 36454 36455
>   * string16      (string16) int32 -2147483647 -2147483647 ... -2147483647
>   * frequency     (frequency) float32 0.0373 0.04103 0.04513 ... 0.6509 0.7159
>   * direction     (direction) float32 90.0 75.0 60.0 45.0 ... 135.0 120.0 105.0
> Data variables:
>     station_name  (station, string16) |S1 dask.array<chunksize=(36455, 16), meta=np.ndarray>
>     longitude     (time, station) float32 dask.array<chunksize=(248, 36455), meta=np.ndarray>
>     latitude      (time, station) float32 dask.array<chunksize=(248, 36455), meta=np.ndarray>
>     frequency1    (frequency) float32 dask.array<chunksize=(32,), meta=np.ndarray>
>     frequency2    (frequency) float32 dask.array<chunksize=(32,), meta=np.ndarray>
>     efth          (frequency, direction, time, station) float32 dask.array<chunksize=(32, 24, 248, 36455), meta=np.ndarray>
>     dpt           (time, station) float32 dask.array<chunksize=(248, 36455), meta=np.ndarray>
>     wnd           (time, station) float32 dask.array<chunksize=(248, 36455), meta=np.ndarray>
>     wnddir        (time, station) float32 dask.array<chunksize=(248, 36455), meta=np.ndarray>
>     cur           (time, station) float32 dask.array<chunksize=(248, 36455), meta=np.ndarray>
>     curdir        (time, station) float32 dask.array<chunksize=(248, 36455), meta=np.ndarray>
> Attributes: (12/19)
>     product_name:              ww3.197901_spec.nc
>     area:                      Tropics2
>     data_type:                 OCO spectra 2D
>     format_version:            1.1
>     southernmost_latitude:     -57.2500000°
>     northernmost_latitude:     68.0000000°
>                        ...
>     start_date:                1979-01-01T00:00:00Z
>     stop_date:                 1979-01-31T21:00:00Z
>     field_type:                3-hourly
>     history:                   Fri Feb 10 20:27:18 2017: /home/local/nco/4.6....
>     NCO:                       "4.6.4"
>     nco_openmp_thread_number:  1

If I do: da1 = da.isel(station = 0).copy().load()
and then da1 = da1.sortby('direction') there is no problem.

But if I sort the da dataset before assigning it to a new variable I have a memory issue:
da = da.sortby('direction')
da1 = da.isel(station = 0).copy().load()

i get :

MemoryError: Unable to allocate 25.9 GiB for an array with shape (32, 24, 248, 36455) and data type float32`

Same error if:
da1 = da.isel(station = 0).copy()
da1 = da1.load()

although the new dimensions of da1 are:

> <xarray.Dataset>
> Dimensions:       (time: 248, string16: 16, frequency: 32, direction: 24)
> Coordinates:
>   * time          (time) datetime64[ns] 1979-01-01 ... 1979-01-31T21:00:00
>     station       int32 1
>   * string16      (string16) int32 -2147483647 -2147483647 ... -2147483647
>   * frequency     (frequency) float32 0.0373 0.04103 0.04513 ... 0.6509 0.7159
>   * direction     (direction) float32 0.0 15.0 30.0 45.0 ... 315.0 330.0 345.0

for what I am doing I cannot reverse the process, and before selecting the station and loading it and sorting. The process is necessary to open the dataset, sort and then load.

Thank you very much for your time.

Ottavio

[dcherian]

The process is necessary to open the dataset, sort and then load.

In that case, try specifying chunks in open_mfdataset. Currently the whole file is being loaded into memory:

dask.array<chunksize=(32, 24, 248, 36455)

[OttavioM]

Thank you very much, I have done this:
da = xr.open_mfdataset('GOW_CFS_Plataforma_1979.01_spc.nc',chunks = {'station':10})
da = da.sortby('direction')
da1 = da.isel(station = 100).copy().load()

And everything worked fine.