Skip to content

Commit da56d51

Browse files
badelebadele
authored
Merge pull request #3 from badele/grafana
Visualize datas from grafana tool
2 parents 66acabd + a465cff commit da56d51

File tree

369 files changed

+40531
-50728
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

369 files changed

+40531
-50728
lines changed

‎.dockerignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
dataset
2+
db
3+
download

‎.envrc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
use flake .

‎.gitattributes‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
datasets filter=lfs diff=lfs merge=lfs -text

‎.github/workflows/docker-image.yml‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
name: Build and Push Docker Image to Docker Hub
2+
3+
on:
4+
push:
5+
paths:
6+
- 'Dockerfile'
7+
- 'requirements.txt'
8+
env:
9+
JUST_VERSION: "1.23.0"
10+
11+
jobs:
12+
push_to_registry:
13+
name: push docker image to hub
14+
runs-on: ubuntu-latest
15+
steps:
16+
- name: check repository
17+
uses: actions/checkout@v4
18+
19+
- name: Setup `just`
20+
uses: extractions/setup-just@v1
21+
with:
22+
just-version: ${{ env.JUST_VERSION }}
23+
24+
- name: login to docker registry
25+
uses: docker/login-action@v3
26+
with:
27+
username: ${{secrets.DOCKERHUB_USERNAME}}
28+
password: ${{secrets.DOCKERHUB_TOKEN}}
29+
30+
- name: build and push docker image to registry
31+
run: just docker-build docker-push

‎.gitignore‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,13 @@
1+
.direnv
2+
3+
# Grafana
4+
grafana-store
5+
grafana/csv
6+
grafana/pdf
7+
grafana/png
8+
grafana/plugins
9+
grafana/grafana.db
10+
111
# Database
212
db
313
exported

‎.pre-commit-config.yaml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# See https://pre-commit.com for more information
2+
# See https://pre-commit.com/hooks.html for more hooks
3+
repos:
4+
# Commons
5+
- repo: https://github.com/pre-commit/pre-commit-hooks
6+
rev: v4.6.0
7+
hooks:
8+
- id: trailing-whitespace
9+
- id: end-of-file-fixer
10+
# - id: check-added-large-files
11+
- repo: local
12+
hooks:
13+
- id: nixpkgs-fmt
14+
name: nixpkgs-fmt
15+
description: Format nix code with nixpkgs-fmt.
16+
language: system
17+
entry: nixpkgs-fmt
18+
files: \.nix$
19+
stages: [commit]
20+
# - id: "docupdate"
21+
# name: "docupdate"
22+
# description: Update documentation.
23+
# language: system
24+
# entry: just doc-update
25+
# stages: [commit]
26+
# files: ^README\.md$
27+
# - id: "markdownlint"
28+
# name: "markdownlint"
29+
# description: Format markdown code with markdownlint.
30+
# language: system
31+
# entry: markdownlint
32+
# stages: [commit]
33+
# files: \.md$
34+
- id: "deno-fmt"
35+
name: "deno-fmt"
36+
description: Format deno code with deno fmt.
37+
language: system
38+
entry: deno fmt
39+
stages: [commit]
40+
files: \.ts$
41+
- id: "deno-lint"
42+
name: "deno-lint"
43+
description: Lint deno code with deno lint.
44+
language: system
45+
entry: deno lint
46+
stages: [commit]
47+
files: \.ts$

‎Dockerfile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
FROM ubuntu:24.04
2+
3+
ENV DEBIAN_FRONTEND=noninteractive
4+
5+
# Copy the requirement files
6+
COPY requirements.txt requirements.txt
7+
COPY importer/requirements.sh requirements.sh
8+
9+
# Install the required packages
10+
RUN ./requirements.sh
11+
12+
# Remove cached files
13+
RUN rm -rf /var/lib/apt/lists/* && apt-get clean && rm -rf requirements.*

‎README.md‎

Lines changed: 106 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -1,89 +1,131 @@
1-
<!-- ===================================================================== -->
2-
<!-- This file is generated from .tpl/README.rmd -->
3-
<!-- ===================================================================== -->
1+
# world-datas-analysis
42

3+
Miscellaneous worlds data and analysis
54

5+
![grafana](doc/grafana.png)
66

7+
## Requirements
78

9+
Installing prerequisites on various distributions
810

9-
# world-datas-analysis
10-
miscellaneous worlds data and analysis
11+
```bash
12+
./install_requirements.sh
13+
```
1114

12-
## Providers of dataset
15+
Works on :
1316

17+
- Alpine
18+
- Archlinux
19+
- Ubuntu
1420

15-
|Dataset |Description | Avg scope|Nb datasets |Max variables | Nb observations|
16-
|:----------------------------|:-------------------------------------------------------------|---------:|:--------------------------------------------------|:----------------------------------------------------|---------------:|
17-
|[geonames](dataset/geonames) |[Geonames entries](https://download.geonames.org/export/dump) | 4070489|[1](dataset/dataset_geonames.md#geonames-datasets) |[15](dataset/dataset_geonames.md#geonames-variables) | 4823955|
18-
|[owid](dataset/owid) |[Our World In Data](https://ourworldindata.org) | 836|[1472](dataset/dataset_owid.md#owid-datasets) |[1](dataset/dataset_owid.md#owid-variables) | 32609745|
19-
|[vigilo](dataset/vigilo) |[Vigilo observations](https://vigilo.city) | 232|[1](dataset/dataset_vigilo.md#vigilo-datasets) |[9](dataset/dataset_vigilo.md#vigilo-variables) | 31623|
21+
## Usage
2022

21-
## Todo
23+
### View on Grafana
2224

23-
| Status | Category | Scope | Description | Sample Report |
24-
|--------|-----------------------------|-------------|-----------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
25-
| [_] | Geonames | Cities | [Geonames](https://download.geonames.org/export/dump/) | [International Covid-19](international/covid-19/README.md) / [French Covid-19](countries/french/covid-19/README.md) |
26-
| [_] | Covid | Countries | [Johns Hopkins University](https://github.com/CSSEGISandData/COVID-19) | [International Covid-19](international/covid-19/README.md) / [French Covid-19](countries/french/covid-19/README.md) |
27-
| [_] | Population | Countries | [United nation](https://population.un.org/wpp/Download/Standard/Population/) | |
28-
| [_] | Population | Cities | [insee](https://www.insee.fr/fr/information/2008354) | |
29-
| [_] | Population | Cities | [insee estimation](https://www.insee.fr/fr/statistiques/1893198) | |
30-
| [_] | Weather | Cities | [European Climate Assessment & Dataset](https://www.ecad.eu/dailydata/predefinedseries.php) | |
31-
| [_] | Weather | Cities | [European Centre for Medium-Range Weather Forecasts](https://confluence.ecmwf.int/display/WEBAPI/Accessing+ECMWF+data+servers+in+batch) | |
32-
| [_] | bike counter | Montpellier | [Montpellier 3M/Velocité](https://compteurs.velocite-montpellier.fr/) | |
33-
| [_] | Rental bike | Montpellier | [Montpellier 3M](https://data.montpellier3m.fr/dataset/courses-des-velos-velomagg-de-montpellier-mediterranee-metropole) | |
34-
| [_] | universitetetioslo | Countries | [CO2 emissions](https://folk.universitetetioslo.no/roberan/GCB2020.shtml) | |
35-
| [_] | NASA | Countries | [Anormal température](https://data.giss.nasa.gov/gistemp/) | |
36-
37-
## Init environment
25+
**INFO:** While waiting for [DuckDB](https://duckdb.org/) support in Grafana and
26+
to speed up data access from Grafana (especially for accessing views), the data
27+
is exported to the [PostgreSQL](https://www.postgresql.org/) database.
3828

3929
```
40-
# Python
41-
python -m venv .venv
42-
source .venv/bin/activate
43-
pip install --only-binary=:all: -r requirements.txt
44-
45-
# R
46-
renv::restore()
30+
just import # Import data to PostgreSQL (used by Grafana)
31+
just chart # Open Grafana (admin/admin)
32+
just stop # Stop the Grafana server
4733
```
4834

49-
## Import and export datas
35+
**Note:** The Grafana account credentials are `admin/admin`.
5036

51-
```
52-
# All in one
53-
./importer/<dataset>/import_and_export.sh
54-
# or step by step
55-
./importer/<dataset>/download.sh
56-
./importer/<dataset>/import.sh
57-
./importer/<dataset>/export.sh
58-
```
59-
## Update this Readme.md page
60-
```
61-
./build_rmarkdown_pages.sh
62-
```
37+
### For developers: Convert data to Parquet format
6338

39+
Before having a dataset viewable in Grafana, you need to download and convert it
40+
to Parquet format. This reduces size and improves performance in DuckDB. Learn
41+
more about [Parquet](https://parquet.apache.org/).
6442

43+
```bash
44+
just download
45+
just update
6546
```
66-
# Use virtualenn python environment
67-
source .venv/bin/activate
6847

69-
# owid
70-
importer/owid/download.sh
71-
importer/owid/import.sh
48+
## Scopes reference
49+
50+
When you add new data to this project, you can sync with reference data by
51+
scope.
52+
53+
For example, if you import a new dataset associated with cities, you can link
54+
them with geonames city elements
7255

56+
<!-- BEGIN SCOPEREFERENCE -->
7357

74-
# Level 1 (geoname)
75-
importer/geonames/download.sh
76-
importer/geonames/import.sh
58+
| provider | dataset | wda_scope | source | nb_variables | nb_entries |
59+
| -------- | ---------------------- | --------- | -------------------- | -----------: | ---------: |
60+
| geonames | wda_geonames_cities | city | https://geonames.org | 98 | 534217 |
61+
| geonames | wda_geonames_countries | country | https://geonames.org | 20 | 252 |
7762

78-
# Worldbnk
79-
python importer/200-worldbank/download.py
80-
sqlite3 -bail world-datas-analysis.db < importer/200-worldbank/import.sql
81-
#international/countryexport_worldbank.sh
82-
#python international/country/download_from_ourworldindata.py
63+
<!-- END SCOPEREFERENCE -->
8364

84-
# Vigilo
85-
./importer/vigilo/import_and_export.sh
65+
## Providers
8666

87-
# Summary
88-
sqlite3 -bail world-datas-analysis.db < db_summary.sql
67+
<!-- BEGIN PROVIDER -->
68+
69+
| provider | description | website | nb_datasets | nb_observations |
70+
| -------- | ----------------------------------------------------- | ------------------- | ----------: | --------------: |
71+
| vigilo | Observations of the collaborative citizen application | https://vigilo.city | 1 | 25548 |
72+
73+
<!-- END PROVIDER -->
74+
75+
## Datasets
76+
77+
<!-- BEGIN DATASET -->
78+
79+
| provider | real_provider | dataset | wda_scope | wda_scope_ref | description | source | nb_variables | nb_observations | nb_scopes |
80+
| ---------- | ------------- | --------------------------------------- | --------- | ------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------- | -----------: | --------------: | --------: |
81+
| opendata3m | opendata3m | wda_opendata3m_ecocompteur_observations | city | wda_geonames_cities | ecocompteur observations | https://data.montpellier3m.fr/dataset/comptages-velo-et-pieton-issus-des-eco-compteurs/resource/edf3e04f-9409-40fe-be66 | 107 | 54149 | 11 |
82+
| vigilo | vigilo | wda_vigilo_observations | city | wda_geonames_cities | vigilo citizen observations | https://vigilo.city | 111 | 25548 | 173 |
83+
84+
<!-- END DATASET -->
85+
86+
## Todo
87+
88+
| Status | Category | Scope | Description |
89+
| ------ | ------------------ | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
90+
| [x] | Geonames | Cities | [Geonames](https://download.geonames.org/export/dump/) |
91+
| [x] | bike counter | Montpellier | [Montpellier 3M](https://data.montpellier3m.fr/dataset/comptages-velo-et-pieton-issus-des-eco-compteurs/resource/edf3e04f-9409-40fe-be66) |
92+
| [x] | vigilo | Montpellier | [Vigilo](https://vigilo.city) |
93+
| [ ] | Enterprise | Countries | [Sirene](https://www.data.gouv.fr/fr/datasets/base-sirene-des-entreprises-et-de-leurs-etablissements-siren-siret/) |
94+
| [ ] | Covid | Countries | [Johns Hopkins University](https://github.com/CSSEGISandData/COVID-19) |
95+
| [ ] | NASA | Countries | [Anormal température](https://data.giss.nasa.gov/gistemp/) |
96+
| [ ] | Population | Cities | [insee estimation](https://www.insee.fr/fr/statistiques/1893198) |
97+
| [ ] | Population | Cities | [insee](https://www.insee.fr/fr/information/2008354) |
98+
| [ ] | Population | Countries | [United nation](https://population.un.org/wpp/Download/Standard/Population/) |
99+
| [ ] | Rental bike | Montpellier | [Montpellier 3M](https://data.montpellier3m.fr/dataset/courses-des-velos-velomagg-de-montpellier-mediterranee-metropole) |
100+
| [ ] | Weather | Cities | [European Centre for Medium-Range Weather Forecasts](https://confluence.ecmwf.int/display/WEBAPI/Accessing+ECMWF+data+servers+in+batch) |
101+
| [ ] | Weather | Cities | [European Climate Assessment & Dataset](https://www.ecad.eu/dailydata/predefinedseries.php) |
102+
| [ ] | universitetetioslo | Countries | [CO2 emissions](https://folk.universitetetioslo.no/roberan/GCB2020.shtml) |
103+
104+
## Project commands
105+
106+
<!-- COMMANDS -->
107+
108+
```text
109+
justfile commands:
110+
browse # Browse world datas
111+
chart # Open browser to grafana page
112+
db-reset # Reset duckdb database
113+
doc-update FAKEFILENAME # Update documentation
114+
docker-build # Build the wda docker image
115+
docker-duckdb # Run duckdb cli on docker
116+
docker-push # Push the wda docker image to docker hub
117+
docker-run CMD="" # Run the wda docker image
118+
help # This help
119+
import # Import datasets to sqlite
120+
lint # Lint the project
121+
packages # Show installed packages
122+
precommit-check # precommit check
123+
precommit-install # Setup pre-commit
124+
precommit-update # Update pre-commit
125+
reset # Reset grafana storage
126+
start # Start grafana
127+
stop # Stop grafana
128+
update # Update datasets
89129
```
130+
131+
<!-- /COMMANDS -->

‎build_rmarkdown_pages.R‎

Lines changed: 0 additions & 33 deletions
This file was deleted.

‎build_rmarkdown_pages.sh‎

Lines changed: 0 additions & 10 deletions
This file was deleted.

0 commit comments

Comments
 (0)