Upserting along multiple dimensions

[pjireland]

I'm getting started with Zarr/icechunk and was interested in seeing if it would work for my data. My typical processing workflow is to ingest data across multiple devices for a certain period of time. I'd then like to upsert this data onto an existing table in the cloud (that would typically have observations from each device at earlier times, though some rows could overlap between the existing data and the newly collected data). The typical newly collected data look something like this:

Device ID (index)	Timestamp (index)	Signal 1 Value	Signal 2 Value	...	Signal N Value
1	2025-07-16 02:00:00	2.1	1.5	...	200.3
1	2025-07-16 02:00:01	2.3	1.4	...	200.1
...
1	2025-07-16 02:10:00	3.5	0.6	...	199.2
2	2025-07-16 02:00:00	1.9	1.6	...	201.2
2	2025-07-16 02:00:01	0.3	0.4	...	200.8
...
2	2025-07-16 02:10:00	4.5	0.2	...	207.8
M	2025-07-16 02:00:00	1.6	3.2	...	191.4
M	2025-07-16 02:00:01	0.1	0.8	...	187.6
...
M	2025-07-16 02:10:00	4.2	0.4	...	199.1

From what I can tell, appending along a single dimension is supported, but I didn't see a way to upsert or to do so across multiple dimensions.

Is this type of data ingestion workflow possible using Zarr/icechunk? Do you have any recommendations on how I could use these tools to allow it? Thanks!

[rabernat]

Thanks for the question @pjireland! This looks like tabular data to me. What would you consider to be the "dimensions" of this dataset?

[pjireland]

Thanks for the quick response. I’d consider the device ID and the timestamp to be my “dimensions” here.