Skip to content

Modify ensure_aggregatable_across_cases to ensure NetCDF cubes are correctly merged #1199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jfrost-mo merged 15 commits into from
Mar 4, 2025
Merged

Modify ensure_aggregatable_across_cases to ensure NetCDF cubes are correctly merged #1199

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
74c4ff2
Collapse time coord before taking difference
jfrost-mo Feb 24, 2025
ecf8898
Fix to case aggregation of NetCDF files
jfrost-mo Feb 4, 2025
11f97a3
Allow collapse_by_ operators to handle CubeLists
jfrost-mo Feb 25, 2025
47a1181
Extract cube from CubeList for histogram plot
jfrost-mo Feb 28, 2025
01ab79f
Raise exception when difference can't find common time points
jfrost-mo Feb 28, 2025
cf888d8
Slightly refactor collapse_by_validity_time
jfrost-mo Feb 28, 2025
8233b83
Reconstruct missing time coordinate in collapse_by_validity_time
jfrost-mo Feb 28, 2025
62d201c
Skip time coord equalisation when no time coord in difference
jfrost-mo Mar 3, 2025
05546fe
Give a nice error message when collapsing non-overlapping cubes
jfrost-mo Mar 3, 2025
cd2ead1
Additional difference tests
jfrost-mo Mar 3, 2025
2d75e08
Test collapse_by_validity_time error when no overlap
jfrost-mo Mar 3, 2025
4788e04
Test bucketing leaves incompatible cubes separate
jfrost-mo Mar 4, 2025
07e30a1
Test collapsing cubelists
jfrost-mo Mar 4, 2025
a29adb4
Test error when collapsing a CubeList without multi_case
jfrost-mo Mar 4, 2025
f0b63ad
Remove multi_case option from collapse_by_hour_of_day
jfrost-mo Mar 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 72 additions & 33 deletions src/CSET/operators/aggregate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,14 @@

"""Operators to aggregate across either 1 or 2 dimensions."""

import logging

import iris
import iris.analysis
import iris.coord_categorisation
import iris.cube
import iris.exceptions
import iris.util
import isodate

from CSET._common import iter_maybe
Expand Down Expand Up @@ -88,32 +92,29 @@ def time_aggregate(


def ensure_aggregatable_across_cases(
cube: iris.cube.Cube | iris.cube.CubeList,
) -> iris.cube.Cube:
cubes: iris.cube.Cube | iris.cube.CubeList,
) -> iris.cube.CubeList:
"""Ensure a Cube or CubeList can be aggregated across multiple cases.

The cubes are grouped into buckets of compatible cubes, then each bucket is
converted into a single aggregatable cube with ``forecast_period`` and
``forecast_reference_time`` dimension coordinates.

Arguments
---------
cube: iris.cube.Cube | iris.cube.CubeList
If a Cube is provided it is checked to determine if it has the
the necessary dimensional coordinates to be aggregatable.
These necessary coordinates are 'forecast_period' and
'forecast_reference_time'.If a CubeList is provided a Cube is created
by slicing over all time coordinates and the resulting list is merged
to create an aggregatable cube.
cubes: iris.cube.Cube | iris.cube.CubeList
Each cube is checked to determine if it has the the necessary
dimensional coordinates to be aggregatable, being processed if needed.

Returns
-------
cube: iris.cube.Cube
A time aggregatable cube with dimension coordinates including
'forecast_period' and 'forecast_reference_time'.
cubes: iris.cube.CubeList
A CubeList of time aggregatable cubes.

Raises
------
ValueError
If a Cube is provided and it is not aggregatable a ValueError is
raised. The user should then provide a CubeList to be turned into an
aggregatable cube to allow aggregation across multiple cases to occur.
If any of the provided cubes cannot be made aggregatable.

Notes
-----
Expand All @@ -123,27 +124,65 @@ def ensure_aggregatable_across_cases(
to ensure that the full dataset can be loaded as a single cube. This
functionality is particularly useful for percentiles, Q-Q plots, and
histograms.

The necessary dimension coordinates for a cube to be aggregatable are
``forecast_period`` and ``forecast_reference_time``.
"""
# Check to see if a cube is input and if that cube is iterable.
if isinstance(cube, iris.cube.Cube):
if is_time_aggregatable(cube):
return cube
else:

# Group compatible cubes.
class Buckets:
def __init__(self):
self.buckets = []

def add(self, cube: iris.cube.Cube):
"""Add a cube into a bucket.

If the cube is compatible with an existing bucket it is added there.
Otherwise it gets its own bucket.
"""
for bucket in self.buckets:
if bucket[0].is_compatible(cube):
bucket.append(cube)
return
self.buckets.append(iris.cube.CubeList([cube]))

def get_buckets(self) -> list[iris.cube.CubeList]:
return self.buckets

b = Buckets()
for cube in iter_maybe(cubes):
b.add(cube)
buckets = b.get_buckets()

logging.debug("Buckets:\n%s", "\n---\n".join(str(b) for b in buckets))

# Ensure each bucket is a single aggregatable cube.
aggregatable_cubes = iris.cube.CubeList()
for bucket in buckets:
# Single cubes that are already aggregatable won't need processing.
if len(bucket) == 1 and is_time_aggregatable(bucket[0]):
aggregatable_cube = bucket[0]
aggregatable_cubes.append(aggregatable_cube)
continue

# Create an aggregatable cube from the provided CubeList.
to_merge = iris.cube.CubeList()
for cube in bucket:
to_merge.extend(
cube.slices_over(["forecast_period", "forecast_reference_time"])
)
logging.debug("Cubes to merge:\n%s", to_merge)
aggregatable_cube = to_merge.merge_cube()

# Verify cube is now aggregatable.
if not is_time_aggregatable(aggregatable_cube):
raise ValueError(
"Single Cube should have 'forecast_period' and"
"'forecast_reference_time' dimensional coordinates. "
"To make a time aggregatable Cube input a CubeList."
"Cube should have 'forecast_period' and 'forecast_reference_time' dimension coordinates.",
aggregatable_cube,
)
# Create an aggregatable cube from the provided CubeList.
else:
new_cube_list = iris.cube.CubeList()
for sub_cube in cube:
for cube_slice in sub_cube.slices_over(
["forecast_period", "forecast_reference_time"]
):
new_cube_list.append(cube_slice)
new_merged_cube = new_cube_list.merge_cube()
return new_merged_cube
aggregatable_cubes.append(aggregatable_cube)

return aggregatable_cubes


def add_hour_coordinate(
Expand Down
Loading