Skip to content

Commit a7f78fd

Browse files
badelebadele
committed
feat: use grafana viewer
1 parent 66acabd commit a7f78fd

File tree

369 files changed

+40516
-50730
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

369 files changed

+40516
-50730
lines changed

‎.dockerignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
dataset
2+
db
3+
download

‎.envrc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
use flake .

‎.gitattributes‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
datasets filter=lfs diff=lfs merge=lfs -text

‎.github/workflows/docker-image.yml‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
name: Build and Push Docker Image to Docker Hub
2+
3+
on:
4+
push:
5+
paths:
6+
- 'Dockerfile'
7+
- 'requirements.txt'
8+
env:
9+
JUST_VERSION: "1.23.0"
10+
11+
jobs:
12+
push_to_registry:
13+
name: push docker image to hub
14+
runs-on: ubuntu-latest
15+
steps:
16+
- name: check repository
17+
uses: actions/checkout@v4
18+
19+
- name: Setup `just`
20+
uses: extractions/setup-just@v1
21+
with:
22+
just-version: ${{ env.JUST_VERSION }}
23+
24+
- name: login to docker registry
25+
uses: docker/login-action@v3
26+
with:
27+
username: ${{secrets.DOCKERHUB_USERNAME}}
28+
password: ${{secrets.DOCKERHUB_TOKEN}}
29+
30+
- name: build and push docker image to registry
31+
run: just docker-build docker-push

‎.gitignore‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,13 @@
1+
.direnv
2+
3+
# Grafana
4+
grafana-store
5+
grafana/csv
6+
grafana/pdf
7+
grafana/png
8+
grafana/plugins
9+
grafana/grafana.db
10+
111
# Database
212
db
313
exported

‎.pre-commit-config.yaml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# See https://pre-commit.com for more information
2+
# See https://pre-commit.com/hooks.html for more hooks
3+
repos:
4+
# Commons
5+
- repo: https://github.com/pre-commit/pre-commit-hooks
6+
rev: v4.6.0
7+
hooks:
8+
- id: trailing-whitespace
9+
- id: end-of-file-fixer
10+
# - id: check-added-large-files
11+
- repo: local
12+
hooks:
13+
- id: nixpkgs-fmt
14+
name: nixpkgs-fmt
15+
description: Format nix code with nixpkgs-fmt.
16+
language: system
17+
entry: nixpkgs-fmt
18+
files: \.nix$
19+
stages: [commit]
20+
# - id: "docupdate"
21+
# name: "docupdate"
22+
# description: Update documentation.
23+
# language: system
24+
# entry: just doc-update
25+
# stages: [commit]
26+
# files: ^README\.md$
27+
# - id: "markdownlint"
28+
# name: "markdownlint"
29+
# description: Format markdown code with markdownlint.
30+
# language: system
31+
# entry: markdownlint
32+
# stages: [commit]
33+
# files: \.md$
34+
- id: "deno-fmt"
35+
name: "deno-fmt"
36+
description: Format deno code with deno fmt.
37+
language: system
38+
entry: deno fmt
39+
stages: [commit]
40+
files: \.ts$
41+
- id: "deno-lint"
42+
name: "deno-lint"
43+
description: Lint deno code with deno lint.
44+
language: system
45+
entry: deno lint
46+
stages: [commit]
47+
files: \.ts$

‎Dockerfile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
FROM ubuntu:24.04
2+
3+
ENV DEBIAN_FRONTEND=noninteractive
4+
5+
# Copy the requirement files
6+
COPY requirements.txt requirements.txt
7+
COPY importer/requirements.sh requirements.sh
8+
9+
# Install the required packages
10+
RUN ./requirements.sh
11+
12+
# Remove cached files
13+
RUN rm -rf /var/lib/apt/lists/* && apt-get clean && rm -rf requirements.*

‎README.md‎

Lines changed: 90 additions & 66 deletions
Original file line numberDiff line numberDiff line change
@@ -1,89 +1,113 @@
1-
<!-- ===================================================================== -->
2-
<!-- This file is generated from .tpl/README.rmd -->
3-
<!-- ===================================================================== -->
1+
# world-datas-analysis
42

3+
Miscellaneous worlds data and analysis
54

5+
![grafana](doc/grafana.png)
66

7+
## Requirements
78

9+
Installing prerequisites on various distributions
810

9-
# world-datas-analysis
10-
miscellaneous worlds data and analysis
11+
```bash
12+
./install_requirements.sh
13+
```
1114

12-
## Providers of dataset
15+
Works on :
1316

17+
- Alpine
18+
- Archlinux
19+
- Ubuntu
1420

15-
|Dataset |Description | Avg scope|Nb datasets |Max variables | Nb observations|
16-
|:----------------------------|:-------------------------------------------------------------|---------:|:--------------------------------------------------|:----------------------------------------------------|---------------:|
17-
|[geonames](dataset/geonames) |[Geonames entries](https://download.geonames.org/export/dump) | 4070489|[1](dataset/dataset_geonames.md#geonames-datasets) |[15](dataset/dataset_geonames.md#geonames-variables) | 4823955|
18-
|[owid](dataset/owid) |[Our World In Data](https://ourworldindata.org) | 836|[1472](dataset/dataset_owid.md#owid-datasets) |[1](dataset/dataset_owid.md#owid-variables) | 32609745|
19-
|[vigilo](dataset/vigilo) |[Vigilo observations](https://vigilo.city) | 232|[1](dataset/dataset_vigilo.md#vigilo-datasets) |[9](dataset/dataset_vigilo.md#vigilo-variables) | 31623|
21+
## Usage
2022

21-
## Todo
23+
```
24+
just import # Import data to PostgreSQL (used by grafana)
25+
just chart # open the grafana (admin/admin)
26+
just stop # stop the grafana server
27+
```
2228

23-
| Status | Category | Scope | Description | Sample Report |
24-
|--------|-----------------------------|-------------|-----------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
25-
| [_] | Geonames | Cities | [Geonames](https://download.geonames.org/export/dump/) | [International Covid-19](international/covid-19/README.md) / [French Covid-19](countries/french/covid-19/README.md) |
26-
| [_] | Covid | Countries | [Johns Hopkins University](https://github.com/CSSEGISandData/COVID-19) | [International Covid-19](international/covid-19/README.md) / [French Covid-19](countries/french/covid-19/README.md) |
27-
| [_] | Population | Countries | [United nation](https://population.un.org/wpp/Download/Standard/Population/) | |
28-
| [_] | Population | Cities | [insee](https://www.insee.fr/fr/information/2008354) | |
29-
| [_] | Population | Cities | [insee estimation](https://www.insee.fr/fr/statistiques/1893198) | |
30-
| [_] | Weather | Cities | [European Climate Assessment & Dataset](https://www.ecad.eu/dailydata/predefinedseries.php) | |
31-
| [_] | Weather | Cities | [European Centre for Medium-Range Weather Forecasts](https://confluence.ecmwf.int/display/WEBAPI/Accessing+ECMWF+data+servers+in+batch) | |
32-
| [_] | bike counter | Montpellier | [Montpellier 3M/Velocité](https://compteurs.velocite-montpellier.fr/) | |
33-
| [_] | Rental bike | Montpellier | [Montpellier 3M](https://data.montpellier3m.fr/dataset/courses-des-velos-velomagg-de-montpellier-mediterranee-metropole) | |
34-
| [_] | universitetetioslo | Countries | [CO2 emissions](https://folk.universitetetioslo.no/roberan/GCB2020.shtml) | |
35-
| [_] | NASA | Countries | [Anormal température](https://data.giss.nasa.gov/gistemp/) | |
36-
37-
## Init environment
29+
**Note:** The grafana account is `admin/admin`
3830

39-
```
40-
# Python
41-
python -m venv .venv
42-
source .venv/bin/activate
43-
pip install --only-binary=:all: -r requirements.txt
31+
## Scopes reference
4432

45-
# R
46-
renv::restore()
47-
```
33+
When you add new data to this project, you can sync with reference data by
34+
scope.
4835

49-
## Import and export datas
36+
For example, if you import a new dataset associated with cities, you can link
37+
them with geonames city elements
5038

51-
```
52-
# All in one
53-
./importer/<dataset>/import_and_export.sh
54-
# or step by step
55-
./importer/<dataset>/download.sh
56-
./importer/<dataset>/import.sh
57-
./importer/<dataset>/export.sh
58-
```
59-
## Update this Readme.md page
60-
```
61-
./build_rmarkdown_pages.sh
62-
```
39+
<!-- BEGIN SCOPEREFERENCE -->
6340

41+
| provider | dataset | wda_scope | source | nb_variables | nb_entries |
42+
| -------- | ---------------------- | --------- | -------------------- | -----------: | ---------: |
43+
| geonames | wda_geonames_cities | city | https://geonames.org | 98 | 534217 |
44+
| geonames | wda_geonames_countries | country | https://geonames.org | 20 | 252 |
6445

65-
```
66-
# Use virtualenn python environment
67-
source .venv/bin/activate
46+
<!-- END SCOPEREFERENCE -->
47+
48+
## Providers
49+
50+
<!-- BEGIN PROVIDER -->
6851

69-
# owid
70-
importer/owid/download.sh
71-
importer/owid/import.sh
52+
| provider | description | website | nb_datasets | nb_observations |
53+
| -------- | ----------------------------------------------------- | ------------------- | ----------: | --------------: |
54+
| vigilo | Observations of the collaborative citizen application | https://vigilo.city | 1 | 25548 |
7255

56+
<!-- END PROVIDER -->
7357

74-
# Level 1 (geoname)
75-
importer/geonames/download.sh
76-
importer/geonames/import.sh
58+
## Datasets
7759

78-
# Worldbnk
79-
python importer/200-worldbank/download.py
80-
sqlite3 -bail world-datas-analysis.db < importer/200-worldbank/import.sql
81-
#international/countryexport_worldbank.sh
82-
#python international/country/download_from_ourworldindata.py
60+
<!-- BEGIN DATASET -->
8361

84-
# Vigilo
85-
./importer/vigilo/import_and_export.sh
62+
| provider | real_provider | dataset | wda_scope | wda_scope_ref | description | source | nb_variables | nb_observations | nb_scopes |
63+
| ---------- | ------------- | --------------------------------------- | --------- | ------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------- | -----------: | --------------: | --------: |
64+
| opendata3m | opendata3m | wda_opendata3m_ecocompteur_observations | city | wda_geonames_cities | ecocompteur observations | https://data.montpellier3m.fr/dataset/comptages-velo-et-pieton-issus-des-eco-compteurs/resource/edf3e04f-9409-40fe-be66 | 107 | 54149 | 11 |
65+
| vigilo | vigilo | wda_vigilo_observations | city | wda_geonames_cities | vigilo citizen observations | https://vigilo.city | 111 | 25548 | 173 |
8666

87-
# Summary
88-
sqlite3 -bail world-datas-analysis.db < db_summary.sql
67+
<!-- END DATASET -->
68+
69+
## Todo
70+
71+
| Status | Category | Scope | Description | Sample Report |
72+
| ------ | ------------------ | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
73+
| [x] | Geonames | Cities | [Geonames](https://download.geonames.org/export/dump/) | [International Covid-19](international/covid-19/README.md) / [French Covid-19](countries/french/covid-19/README.md) |
74+
| [x] | bike counter | Montpellier | [Montpellier 3M](https://data.montpellier3m.fr/dataset/comptages-velo-et-pieton-issus-des-eco-compteurs/resource/edf3e04f-9409-40fe-be66) | |
75+
| [x] | vigilo | Montpellier | [Vigilo](https://vigilo.city) | |
76+
| [ ] | Covid | Countries | [Johns Hopkins University](https://github.com/CSSEGISandData/COVID-19) | [International Covid-19](international/covid-19/README.md) / [French Covid-19](countries/french/covid-19/README.md) |
77+
| [ ] | NASA | Countries | [Anormal température](https://data.giss.nasa.gov/gistemp/) | |
78+
| [ ] | Population | Cities | [insee estimation](https://www.insee.fr/fr/statistiques/1893198) | |
79+
| [ ] | Population | Cities | [insee](https://www.insee.fr/fr/information/2008354) | |
80+
| [ ] | Population | Countries | [United nation](https://population.un.org/wpp/Download/Standard/Population/) | |
81+
| [ ] | Rental bike | Montpellier | [Montpellier 3M](https://data.montpellier3m.fr/dataset/courses-des-velos-velomagg-de-montpellier-mediterranee-metropole) | |
82+
| [ ] | Weather | Cities | [European Centre for Medium-Range Weather Forecasts](https://confluence.ecmwf.int/display/WEBAPI/Accessing+ECMWF+data+servers+in+batch) | |
83+
| [ ] | Weather | Cities | [European Climate Assessment & Dataset](https://www.ecad.eu/dailydata/predefinedseries.php) | |
84+
| [ ] | universitetetioslo | Countries | [CO2 emissions](https://folk.universitetetioslo.no/roberan/GCB2020.shtml) | |
85+
86+
## Project commands
87+
88+
<!-- COMMANDS -->
89+
90+
```text
91+
justfile commands:
92+
browse # Browse world datas
93+
chart # Open browser to grafana page
94+
db-reset # Reset duckdb database
95+
doc-update FAKEFILENAME # Update documentation
96+
docker-build # Build the wda docker image
97+
docker-duckdb # Run duckdb cli on docker
98+
docker-push # Push the wda docker image to docker hub
99+
docker-run CMD="" # Run the wda docker image
100+
help # This help
101+
import # Import datasets to sqlite
102+
lint # Lint the project
103+
packages # Show installed packages
104+
precommit-check # precommit check
105+
precommit-install # Setup pre-commit
106+
precommit-update # Update pre-commit
107+
reset # Reset grafana storage
108+
start # Start grafana
109+
stop # Stop grafana
110+
update # Update datasets
89111
```
112+
113+
<!-- /COMMANDS -->

‎build_rmarkdown_pages.R‎

Lines changed: 0 additions & 33 deletions
This file was deleted.

‎build_rmarkdown_pages.sh‎

Lines changed: 0 additions & 10 deletions
This file was deleted.

0 commit comments

Comments
 (0)