Test data with updated structure on azure

[Peter9192]

In #38 we discuss a data directory structure proposal, with nicely nested folders. While this is great for organizing the data, it can be annoying during development, because we typically don't want to download a full copy, but just a small subset of the countries. This requires more clicking.

One option would be to also maintain a subset on the storage explorer that can be used for development. So you would have:

• Version_0_4_2
• Version_0_4_2_development_subset

The development subset is just a copy of the other one, but with only 5 countries or so. It could be generated at the same time the full data is generated.

This would make it easy to download the dev data with a single click, without flooding our hard drives.

[Peter9192]

The subset should include

• One EU country and one non-EU country (as they are different in terms of having NCD data)
• One country and one 'other region', e.g. EU or G7
• Big countries that are easy to click on (US, Russia, ...)

[Peter9192]

Now created

[2.3G]  Version_0_4_2_development_subset/
├── [819K]  ne_110m_admin_0_countries.geojson
├── [ 11M]  xr_policyscen.nc
└── [2.2G]  startyear_2021
    ├── [7.6M]  xr_dataread.nc
    └── [2.2G]  GHG_incl
        ├── [975M]  Aggregated_files
        │   ├── [487M]  xr_alloc_2030.nc
        │   └── [487M]  xr_alloc_2040.nc
        └── [1.3G]  Allocations
            ├── [188M]  xr_alloc_BRA.nc
            ├── [188M]  xr_alloc_COD.nc
            ├── [188M]  xr_alloc_DEU.nc
            ├── [188M]  xr_alloc_G7.nc
            ├── [188M]  xr_alloc_NLD.nc
            ├── [188M]  xr_alloc_RUS.nc
            └── [188M]  xr_alloc_USA.nc

Now on Azure storage explorer.

Update: now in #61

[Peter9192]

If we are all happy with this structure, we should also update the full dataset to adhere to the same structure/naming convention

[Peter9192]

Some points for improvement still:

• Could we combine the assumptionset and the startyear? E.g. 2021_GHG_incl?
• Do all filenames have to start with xr?
• Why are aggregated files called alloc? Do we even need alloc in the filenames?
• Could we find a better name for dataread? global_pathways?
• Does it make sense to use more verbose but less cryptic names?

e.g.

Version_0_4_2_development_subset/
├── ne_110m_admin_0_countries.geojson
├── policy_scenarios.nc
└── startyear2021_allgases_landuseincluded
    ├── global_pathways.nc
    ├── aggregations
    │   ├── 2030.nc
    │   └── 2040.nc
    └── allocations
        ├── BRA.nc
        ├── COD.nc
        ├── DEU.nc
        ├── G7.nc
        ├── NLD.nc
        ├── RUS.nc
        └── USA.nc

@MarkMDekker any thoughts on this?