Part 2 - Zarr consolidated metadata

[mannreis]

This PR follows up on #3219 :

• Adds libnczarr/zmetadata.h to the code base and to the build systems
• Introduces the metadata field of type struct NCZ_Metadata in NCZ_FILE_INFO_T

[DennisHeimbigner]

I am curious. I am comparing the zmetadata dispatcher operation from zarr-csl-2
and the original one I had from long ago.
The dispatcher API for zarr-csl-2 is basically this:

int (*list_nodes)(struct NCZ_FILE_INFO*, const char * key, NClist *groups, NClist *vars);
int (*list_groups)(struct NCZ_FILE_INFO*, const char * key, NClist *subgrpnames);
int (*list_variables)(struct NCZ_FILE_INFO*, const char * key, NClist *varnames);
int (*fetch_json_content)(struct NCZ_FILE_INFO*, NCZMD_MetadataType, const char *name, NCjson **jobj);
int (*update_json_content)(struct NCZ_FILE_INFO*, NCZMD_MetadataType, const char *name, const NCjson *jobj);
int (*validate_consolidated)(const NCjson *jobj);

The set I originally used was this:

int (*open)(NC_FILE_INFO_T* file);
int (*create)(NC_FILE_INFO_T* file);
int (*close)(NC_FILE_INFO_T* file);
int (*consolidate)(NC_FILE_INFO_T* file);
int (*fetch_json_content)(NC_FILE_INFO_T *, NCZMD_MetadataType, const char* key, NCjson** jobj);
int (*update_json_content)(NC_FILE_INFO_T *, NCZMD_MetadataType, const char *key, const NCjson *jobj);
/* zmap wrappers */
int (*list)(NC_FILE_INFO_T*, const char* prefix, NClist* matches);
int (*listall)(NC_FILE_INFO_T*, const char* prefix, NClist* matches);
int (*exists)(NC_FILE_INFO_T* file, const char* prefix);

The only two that match as such are fetch_json_content and update_json_content.
So I am trying to match the other function semantics.
Can you give me short 1-2 line description of what your other dispatch API functions do?

[mannreis]

Indeed they are a bit different but not that much. The on you wrote implements the metadata layer as sort of a storage layer and it actually is/uses it. However I consider that approach distances itself a bit away to the Zarr concepts.

I added list_nodes, list_groups, and list_variables but in reality only the first is going to be used.
They basically will either check what groups and/or variables exist on the storage for the no consolidaded case, or provide the same information by parsing the zmetadata content for the consolidated case.

Lastly the validade_consolidated will be a noop for the non consolidated case, and will check if the JSON passed is compliant to what we expect. It will be used with selecting the handler in the case that .zmetadata is corrupted and we can fallback to the non consolidated handler

[DennisHeimbigner]

Thanks, that is helpful.

[mannreis]

Tests need to be re-triggered because of the intermittent issue with dap4_test_test_remote.

The merge-base changed after approval.