HTTP Zarr Map / Store

[mannreis]

The libnczarr implementation has a zmap API inspired by the zarr-python MutableMapping design.
It offers 3 implementations, file, zip and s3.

I'm willing to work on adding a general http implementation. It would:

• Only work on/with consolidated datasets.
• Make it possible to access public data hosted on most web servers.
• Optionally support authentication/authorization via .ncrc + libcurl

NOTE: This would already be somewhat possible by creating a zmap_http just like for S3 (zmap_s3sdk) but replacing the NCZMAP_API.search function with a noop.

[DennisHeimbigner]

I was referring to this project:
```https://pad.gwdg.de/DtHGRP38Sw2YQDAAjPuP2Q#````
Is this project still active, are you working on it, and what more
needs to be done to support it from the netcdf side?

[mannreis]

I was referring to this project:
```https://pad.gwdg.de/DtHGRP38Sw2YQDAAjPuP2Q#````

The goal is still that!

Is this project still active

Yes

are you working on it

Yes by contributing to netcdf

and what more needs to be done to support it from the netcdf side?

What I just proposed above.

The server doesn't need to be a specific software it is so general that the client should be able to fetch data on public S3 buckets, public Swift containers, and even using python -m http.server should work.

[DennisHeimbigner]

Ok But there is still some issues that need to be addressed (assuming read only).

1. I am in the process of providing better source server type info so that users do not have so may mode flags.
2. Sometimes case 1 will fail, in which case, the client will have to force the issue by specifying an appropriate mode flag. For this situation, it is desirable that you pick a mode flag to use. Some time ago I proposed using "zoh" (zarr over http) for that mode flag but you did not like that choice, so this issue is still open.

An important note: If I give a URL path to say nc_open, then for historical reasons, if all inferencing fails, then the URL defaults to assuming DAP2 (OPeNDAP protocol). This choice is unlikely to change, which is one reason why a fallback mode flag is needed.
Also, not that in doing inferencing, I will be probing the server to see what kind of response I get back. Have your group thought about the kinds of requests and corresponding responses will be available for this server?

[mannreis]

Assuming read only is good! But also that only consolidated datasets can be used with this zmap

1. I am in the process of providing better source server type info so that users do not have so may mode flags.

That's great, I'm happy to take a look!

2. Sometimes case 1 will fail, in which case, the client will have to force the issue by specifying an appropriate mode flag. For this situation, it is desirable that you pick a mode flag to use. Some time ago I proposed using "zoh" (zarr over http) for that mode flag but you did not like that choice, so this issue is still open.

I don't oppose to that choice as a fallback!
On a good scenario both https://...#mode=zarr,zoh and https://...#mode=zarr could work.
I understand that for legacy https with no mode "protocol" means DAP.
However with zarr + http things could be different because to my understanding, currently there is no to open zarr via DAP right?

Anyway we could save this particular discussion about protocol precedence and user specifed parameters for later, once a PR is drafted.

Have your group thought about the kinds of requests and corresponding responses will be available for this server?

Probing for .zmetadata, and futurely zarr.json should be good/general enough for this usecase. Anything != 5XX and with a parsable json payload should tell us that we can use the consolidated metadata and vanilla HTTP to access the data.