Feature: support multi-dimensional batch calculation[experimental]

[TonyXiang8787]

This PR experiments an idea of multi-dimensional batch calculation in the C-API without introducing breaking changes. The Python API is also adjusted without breaking changes.

[sonarqubecloud]

Quality Gate passed

Issues
9 New issues
0 Accepted issues

Measures
0 Security Hotspots
88.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[TonyXiang8787]

@mgovers This is one option to pass a MD datasets into the C-API. You need to have additional helper functions to create an array of datasets. We need a new opaque struct.

Another interesting approach would be using chaining. So define a function called dataset_const_chain_another, which store a pointer to the next dimension dataset into this one. Internnaly we can check if the next_dataset pointer is nullptr or not to traverse dimensions. This way is less intuitive, but still used in many other APIs. And you only need one extra function. We don't even need to change the options.

[mgovers]

i prefer this separate ND approach, but how would we use ND-datasets, e.g. a 3-dimensional one instead of only 2-dimensional

[TonyXiang8787]

It is not really a ND, but a list of datasets. The core will make a cross-join ND calculation of combinations in the list.

The questions, do we create separate opaque struct to store an array of datasets? This requires a lot of new functions. Or do we do the chaining, this only needs one new function.

[mgovers]

i do believe under the current implementation, sure, but it also restricts us going forward. i feel like if we are going to support 1D and have a separate object for 2D, then we might as well go for ND. otherwise, we'll fall into the same rabbit hole that C++ has been in (watch any video on std::mdspan and you'll know, they are better at explaining what i mean than i am)

[TonyXiang8787]

The current experiment is about ND, not just 2D.

The ND calculation is created by a list of datasets. Each dataset in the list represents the mutation of that dimension. The core creates a cross-join on the mutations.

[mgovers]

i think this will crash because buffer.indptr[scenario + 1] is called. In python, it's also not possible to call [1, 2, 3][3] (IndexOutOfRange)

[mgovers]

really useful 👍 If we add an additional increment here, we can actually also use this in the dispatching of multithreaded batch calculations

[mgovers]

i feel like this should be able to use arbitrary dimensions

[mgovers]

C++23 has std::mdspan, which would probably solve this

[mgovers]

inconsistent PGM_dataset_create_* vs PGM_destroy_*

[mgovers]

maybe following the conventions of std::mdspan:

Suggested change

	typedef struct PGM_MultiDimensionalDataset PGM_MultiDimensionalDataset;
	typedef struct PGM_MDDataset PGM_MDDataset;

[mgovers]

maybe we can do a multi-indexing approach instead? e.g.

PGM_mddataset_get_data(PGM_Handle* handle, PGM_MDDataset const* mddataset);

PGM_mddataset_get_flat_element(PGM_Handle* handle, PGM_MDDataset const* mddataset, PGM_Idx flattened_index);

PGM_mddataset_get_scenario(PGM_Handle* handle, PGM_MDDataset const* mddataset, PGM_Idx** multi_index, PGM_Idx n_dimensions);

(we can even omit n_dimensions, although it's probably safer to keep it)