Discussion regarding refactoring to include a larger variety of datasets

[ctessum]

Currently this package includes functions to 1) download, 2) open, and 3) query two different datasets. If we want to expand the package to include a wider variety of data, step 3 is going to be difficult because it is hard to anticipate how users will want to use the data ahead of time.

So I would propose that it make sense to remove step 3 from the scope of the project, at least for the near term until common workflows for querying datasets become apparent.

To make steps 1 and 2 more general, maybe the following setup would be useful:

export open, cache

abstract type GeoDataset end
abstract type NCDataset <: GeoDataset end
abstract type ZarrDataset <: GeoDataset end
abstract type ShapefileDatset <: GeoDataset end

struct GSHHG <: ShapefileDatset
    resolution::Int
    level::Int
end

function Base.open(d::GeoDataset)
   cache(d)
   _open(d)
end

function cache(d::GeoDataset)
    .....
end

# This function would go in an extension for when GeoDataFrames.jl or whatever is loaded.
function _open(d::ShapefileDataset)
    .....
end
... etc

In the example above there are two main functions, open and cache, where cache would check if the file was locally available and download it if not. And open would open the file.

Each specific dataset would be a type including any fields necessary for configuration of specification, e.g. resolution and level as above information about mirrors, authentication, etc.

I think the cache function could be relatively straightforward, with maybe some different branches for http vs s3, whether authentication is required, whether there's a possibility for hash checking, etc. I think the cache function should be exported to facility data being dowloaded on HPC systems where only the login node has an internet connection.

The _open function is trickier because there will potentially be many different file formats, but I think the way forward would be to have extensions for each file format to provide a method for open that type of file.

The user would ultimately run open(GSHHG(2,1)) to get a geodataframe with the GSHHG data.

Let me know what you think.

cc @simone-silvestri

[Alexander-Barth]

I agree that we should avoid to have too many dependencies and extensions are quite well suited for that. If the necessary extensions are not loaded, we could give error message explaining for example that for GSHHG you need Shapefile (currently a regular dependency).

I would suggest that we put everything that is specific to a dataset (like username and password for access) in parameters for the constructor of the dataset (like you did with resolution and level) and the rest of the API would be (as much as possible) independent from specific aspect of the dataset. This makes it easier to swap out data sources.

In the context of ocean models, that are saved often at different temporal resolutions. I guess that it the same for other models. I would propose that a dataset can be a collection of variables (e.g. temperature, velocity...) but all have the same resolution. The questions whether a user wants to have a daily or monthly averaged temperature, must be specified in the constructor of the dataset.

Some dataset are so large that a user would want to download only a subset. Maybe this can be a type wrapping a GeoDataset (in the same way that a SubArray (a view) wraps an AbstractArray):

ds_subet = subset(ds, time_range = (DateTime(2000,1,1),DateTime(2000,12,31))) 

I think it would be good to have a code for a dataset that could serve as an example.
If you have already a concrete idea for a dataset, do not hesitate to make a PR :-)