Ability to "Interleave" or "insert" cubes

[acchamber]

✨ Feature Request

I often encounter situations where I am constructing a cube from smaller cubes (such as hourly .nc files). When I do so sometimes I find there's missing values (common in observational data) that I want to infill. However, you cannot concatenate data into the middle of a cube, even if it doesn't overlap. For example:

I have a cube main with the following dim_coord

[0,1,2,3...7799,7800]

But there's missing values at [56] and [3567]. The cube just goes [... 55, 57 ...] and [... 3566, 3568 ... ].

I have cubes with the [56] and [3567] coord that I want to insert into the cube but I cannot

the current workaround is to define new cubes of main[:55],main[56:], then concatenate the cubelist of the three( main[:55],main[56:], [56]) , but this approach is brittle and does not scale well (when you have a 100 missing values this scales poorly)

Ideally, the solution would also allow for more complex interleaving - eg

cube A [0,3,6,9,12]
Cube B [1,2,4,5,7,8,10,11]

cube c = interleave([Cube A, Cube b])

Additional context

This would only need to function in one dimension - obviously all other dimensions would have to match. Discussed with @bjlittle at surgery.

[mo-DanCubbon]

Thumbsing this up because I've also had to do this multiple times. Once for the above issue of re-inserting missing data, once for reassembling 24 hourly series (1 for each hour of the day) into a single hourly series.

[ESadek-MO]

From @SciTools/peloton:

The people have spoken, and the Iris devs are in agreement with the community.

For our own benefit, we've come to the following tentative conclusions:

• We don't think concatenate is the way forwards here, for a multitude of reasons.
• We wonder about an API based around the users expectations of the resultant cube.
  • If we did this, it would make multi-dimensional interleaving trivial.

[acchamber]

What do you mean by "user expectations of ther resultant cube"? That does sound positive (the use cases I have in mind can handle some transformation in the output from the input) but If I understood better could contribute.

[trexfeathers]

What do you mean by "user expectations of ther resultant cube"? That does sound positive (the use cases I have in mind can handle some transformation in the output from the input) but If I understood better could contribute.

We don't have a precise vision yet. But the last decade has taught us that automatic interpretation often does more harm than good, e.g:

• Merging logic in Iris
• Auto-chunking in Dask and/or Iris
• open_mfdataset() in Xarray

In these cases the software is written with the assumption that it can infer the user's desired output based on the input, e.g. which scalar coordinates should be turned into new dimensions. But these are complicated situations, and sometimes things go wrong - the output is undesirable, or the software crashes - and because the logic is internal to the software, the user is left 'high and dry' without the understanding nor the tools to fix the situation.

Perhaps the ideal world is automatic logic that can be modified when needed, but this can be difficult to get right - the API can get over-complicated with lots of confusing options, and/or the code can become too complicated to maintain. So we're looking for the right compromise of user utility and maintainability; our instinct for in the case of Merging/Concatenating is:

API that allows the user to lay out the shape of the expected output Cube, then Iris does the heavy lifting to make that Cube a reality.

Too early to say exactly what this would look like. Maybe a dict: {dim_name: dim_length}? Maybe an actual Cube 'skeleton' object that the user could build up in several lines?