A Krawler based service to download diverse datasets from the public Météo-France API portal
The paquetobs job files enable the scraping of real-time observations from all weather stations of the French network at hourly and/or sub-hourly intervals (every 6 minutes). These data are exposed by the public API Package Observations.
This repository provides with 2 jobs:
This job allows to scape the stations. The main collected properties are:
- id, the id of the station
- name, the name of the station
- location, the location of the station
The station data are stored in compliance with the GeoJSON standard.
| Variable | Description |
|---|---|
PAQUETOBS_TOKEN |
The token to use the paquetobs API |
DEPARTMENTS |
The list of departments used to filter the collected observations, e.g, "11,09,31" |
STATIONS |
The list of station ids used to filter the collected observations, e.g, "09099001,09301001" |
DB_URL |
the database url. By default: mongodb://localhost:27017/meteofrance. |
DEBUG |
Enables debug output. Set it to krawler* to enable full output. By default it is undefined. |
This job allows to scape the observations. The basic collected properties are:
- id, the id of the station
- name, the name of the station
- temperature (°),
- humidity (%),
- wind direction (°),
- wind speed (m/s),
- precipitation (mm/h)
The observation data are stored in compliance with the GeoJSON standard.
| Variable | Description |
|---|---|
PAQUETOBS_TOKEN |
The paquetobs API token. |
FREQUENCY |
The frequency of the observations to collect. It must be horaire or infrahoraire-6m. Default value is horaire. |
LATENCY |
The latency of the observations to collect. It must be horaire or infrahoraire-6m. Default value is horaire. |
TTL |
The retention period in seconds of the data. By default: 7 * 24 * 60 * 60 (~7 days) |
DB_URL |
the database url. By default: mongodb://localhost:27017/meteofrance. |
DEBUG |
Enables debug output. Set it to krawler* to enable full output. By default it is undefined. |
This repository implements a two-stage pipeline for forecast models:
These jobs download forecast data from the public Météo-France API:
Files are retrieved in GRIB2 format.
| Job | Type | Resolution | Coverage |
|---|---|---|---|
arome-france |
AROME | 0.025° | France |
arome-france-high |
AROME | 0.01° | France |
arpege-europe |
ARPEGE | 0.1° | Europe |
arpege-world |
ARPEGE | 0.25° | Global |
| Variable | Description |
|---|---|
DATA_SOURCE |
Source of the data (meteofrance or data-gouv) |
AROME_TOKEN |
API token for AROME |
ARPEGE_TOKEN |
API token for ARPEGE |
OUTPUT_DIR |
Base directory for downloaded files |
WORKERS_LIMIT |
Maximum number of concurrent downloads |
OLDEST_RUN_INTERVAL_MS |
Max allowed run age |
RUN_TIMES |
Override model run times |
PACKAGES |
Override packages |
FORECAST_TIMES |
Override forecast steps |
We personally use Kargo to deploy the service.
Please refer to contribution section for more details.
Licensed under the MIT license.
Copyright (c) 2017-20xx Kalisio
