Repo structure

[peterdesmet]

├── README.md
├── LICENCE
├── .gitignore
│
├── data
│   │
│   ├── raw
│   │   ├── modelling_species.csv
│   │   └── (do not include GBIF data in repo)
│   │
│   ├── interim (.gitignore)
│   │   └── occ with assigned coordinates + grid => S3?
│   │
│   └── processed
│       ├── cube_europe.csv
│       ├── cube_belgium.csv
│       └── cube_belgium_baseline.csv
│
└── src
    │
    ├── belgium
    │   ├── download.Rmd: define filters, trigger download
    │   ├── create_db.Rmd: create sqlite, fill with data, filter on issues
    │   ├── assign_grid.Rmd: assign coordinates, assign grid (chunk based)
    │   └── aggregate.Rmd: (filter on taxa), agg for alien, agg for baseline
    │
    └── europe
        ├── download.Rmd
        ├── create_db.Rmd
        ├── assign_grid.Rmd
        └── aggregate.Rmd

[damianooldoni]

we should add notebook for filtering post download... (e.g. absences, coordinate issues) or just adding the filtering at the begin of assign_coord.ipynb`.

[damianooldoni]

@peterdesmet : up to now we were used to put the file with all triggered downloads (gbif_downloads.tsv) in ./data/output. However, ./data/output doesn't exist anymore, changed to the standard datascience cookiecutter folder ./data/processed, which contains:

The final, canonical data sets for modeling.

I don't like to mix the cubes with this file with all GBIF downloads.
So, should we put it in ./data/processed or maybe use the folder ./references, which contains

Data dictionaries, manuals, and all other explanatory materials.

This is also not completely correct but maybe it is a better solution. The pipeline for triggering a download is ready in branch https://github.com/trias-project/occ-processing/tree/add_download_pipeline_belgium. This is the only doubt I have before PR.

[peterdesmet]

I would put in data/interim