1.2.0

See wis2box 1.2.0 release notes at https://github.com/World-Meteorological-Organization/wis2box-release/releases/tag/1.2.0

1.1.0

See wis2box 1.1.0 release notes at https://github.com/World-Meteorological-Organization/wis2box-release/releases/tag/1.1.0

1.0.0

See wis2box 1.0.0 release notes at https://github.com/World-Meteorological-Organization/wis2box-release/releases/tag/1.0.0

1.0.0rc1

This is the first Release Candidate of wis2box 1.0.0. This release is considered feature complete and under consideration for the final 1.0.0 release. This release adds improvements to updating wis2box installations, numerous bug fixes and updates for WIS2 compliance in support of stabilizing the codebase.

Users are strongly encouraged to update to this release for WIS2 compliance.

When updating from 1.0b8 users can preserve their configuration for stations and datasets. Updating from older releases will require a clean start. See the migration notes below for more details

Enhancements/updates

• SFTP support for data ingestion
• improve support for custom csv2bufr mapping templates
• new UI customization options to configure logo and color: see https://docs.wis2box.wis.wmo.int/en/latest/reference/configuration.html#web-application
• update to latest bufr2geojson JSON schema changes
• enable image-updates; wis2box-ctl.py update using VERSION.txt
• docs
  • update to CAP publishing
  • update OS recommendations
  • update csv2bufr
• update GRIB/BUFR media types
• add CI for Trivy scanning
• fix station station links in API
• handle API 404 errors
• update ES and MinIO settings
• add environment variable to enable/disable CAP signature checking
• update storage management
• remove data movement to archive bucket
• separate retention periods for API and MinIO using WIS2BOX_STORAGE_DATA_RETENTION_DAYS and WIS2BOX_STORAGE_API_RETENTION_DAYS

Bug fixes

• safeguard empty data mappings

New installations

Please follow the instructions provided at https://docs.wis2box.wis.wmo.int/en/1.0.0rc1/user/getting-started.html

Migrating existing installations

Users running versions earlier than 1.0b8 should prune wis2box volumes and images and re-configure wis2box.

python3 wis2box-ctl.py stop
docker volume prune -a
docker image prune -a
cd ../

Then proceed with instructions at:
https://docs.wis2box.wis.wmo.int/en/1.0.0rc1/user/setup.html

Updating from 1.0b8

Users updating from wis2box-1.0b8 do not need to prune volumes but will be required to perform data migrations to retain previously published data in the API backend.

To update please perform the following steps:

python3 wis2box-ctl.py stop
docker image prune -a
cd ../

Download the new archive, copy your old wis2box.env and restart:

wget https://github.com/wmo-im/wis2box/releases/download/1.0.0rc1/wis2box-setup-1.0.0rc1.zip
unzip wis2box-setup-1.0.0rc1.zip
cd wis2box-1.0.0rc1
cp ../wis2box-1.0b8/wis2box.env .
python3 wis2box-ctl.py start

Upon start you will receive the following prompt:

No docker-compose.images-*.yml files found, creating one
Current version=Undefined, latest version=1.0.0rc1
Would you like to update ? (y/n/exit)

Select y to continue with the download of images and wait until services are started:

Check if you previous wis2box URL shows your existing datasets before proceeding. This may take some time.

To run migrations first login to the wis2box container:

python3 wis2box-ctl.py login

Then run the following command to recreate data collections in the API backend and re-process last 2 days of data:

python3 /app/migrations/1.0b8_to_1.0.0rc1/update_data_collections.py --days-to-backfill 2

Wait for the command to complete.

Finally, republish the metadata for your datasets using the following command:

wis2box metadata discovery republish

After this, you can exit the wis2box-management container with:

exit

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0.0rc1

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b8

This is the eighth 1.0 beta release of wis2box. This release includes a new CAP XML data pipeline, added utilities for WIS2 data download, as well numerous bug fixes and updates for WIS2 compliance.

Users are strongly encouraged to update to this release for WIS2 compliance.

Enhancements/updates

• add CAP XML data pipeline
• add data download functionality (via wis2downloader)
• add support for GTS headers applied in WIS2 Notification message
• adds discovery metadata republish command in CLI (includes automated daily publishing job)
• add station association capability in CLI
• show all system warnings (previously set to errors only)
• dataset driven approach implemented in wis2box data automated ingest and CLI
• improved logging / logfile management
• improved disk usage monitoring and management
• update MinIO container version
• add UI vulnerability testing for wis2box-webapp and wis2box-ui
• updated help links for FM-12 and CSV to BUFR in wis2box-webapp
• add metadata_id (for all data) and security (for recommended data) elements to WIS2 Notifications
• store local copy of WIGOS station metadata lookups
• add new discovery metadata templates in wis2box-webapp (drifters and ship based surface weather observations)
• supporting documentation for configuring email alerts from Grafana dashboard
• improved management of API configuration in wis2box-api

Bug fixes

• fix WCMP2 output themes/concepts formatting
• store local copy of WIGOS station metadata lookups for stability
• ensure wis2box starts successfully on Docker host restart
• add test to ensure WIS2BOX_BROKER_PUBLIC is correctly configured

New installations

Please follow the instructions provided at https://docs.wis2box.wis.wmo.int/en/1.0b8/user/getting-started.html

Migrating existing installations

When updating an existing wis2box instance, please run the following commands from the current release directory:

# stop wis2box
python3 wis2box-ctl.py stop

# delete old Docker images
docker image prune -a

# copy wis2box.env from the old to the new release directory
cp ~/wis2box-1.0b7/wis2box.env ~/wis2box-1.0b8/wis2box.env

wis2box-1.0b8 includes new/added download functionality. To use this functionality, run the following commands:

# set the DOCKER_GID environment variable
echo "DOCKER_GID=$(getent group docker | cut -d: -f3)" >> ~/wis2box-1.0b8/wis2box.env

# and add the "/downloads" directory to the directory defined as your WIS2BOX_HOST_DATADIR and set the permission to the docker group:
export $(grep -v '^#' ~/wis2box-1.0b8/wis2box.env | xargs -d '\n')
mkdir -p ${WIS2BOX_HOST_DATADIR}/downloads
chown -R $(whoami):docker ${WIS2BOX_HOST_DATADIR}/downloads
chmod -R 775 ${WIS2BOX_HOST_DATADIR}/downloads

Go to the new wis2box release directory and start wis2box:

cd ~/wis2box-1.0b8
python3 wis2box-ctl.py start

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b8

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b7

This is the seventh 1.0 beta release of wis2box. This release includes a number of new features, enhancements and bugfixes. This release introduces a "dataset" driven approach to closer align to core WIS2 concepts. A "dataset" in wis2box is a single record containing both the metadata and the data mappings definition. The separate data-mappings.yml will not longer be used.

This release also introduces the "dataset" editor that supports the creation of WCMP2 discovery metadata and data mappings definitions without requiring the user to edit configuration files directly.

The release also includes the a new "universal" data plugin to support the publishing of data that is produced upstream of wi2box.

Users are strongly encouraged to update to this latest release for WIS2 compliance.

Enhancements/updates

• move data mappings to discovery metadata record, removing data-mappings.yml
• added dataset editor to wis2box-webapp to support WCMP2 publishing and data mappings configuration
• added basic authentication to wis2box-webapp
• add new universal data pipeline
• update station metadata columns to be consistent between station list CSV and station editor, using WIGOS metadata tables
• csv2bufr templates now referenced from external repository
• remove API links from WCMP2 records
• numerous documentation updates, including updates for station metadata as well as security recommendations
• reduced loading-times for wis2box-ui

Bug fixes

• fix incorrect originating centre and sub centre incorrect in BUFR generation
• fix links in WCMP2 metadata

New installation

Please follow the instructions provided at https://docs.wis2box.wis.wmo.int/en/latest/user/getting-started.html

Migrating existing installations

Migrating existing installations will require you to recreate your discovery metadata records.

First, login to the wis2box-management container and unpublish your current discovery metadata:

cd wis2box-1.0b6
python3 wis2box-ctl.py login
wis2box metadata discovery unpublish <metadata-id>

Repeat the wis2box metadata discovery unpublish command until there are no items remaining in https://YOUR_HOST/oapi/collections/discovery-metadata/items

Then stop wis2box, and download the new release and extract the archive:

python3 wis2box-ctl.py stop
cd ../
wget https://github.com/wmo-im/wis2box/releases/download/1.0b6/wis2box-setup-1.0b7.zip
unzip wis2box-setup-1.0b7.zip

Copy your old wis2box.env and delete the api-config volume:

cp ~/wis2box-1.0b6/wis2box.env ~/wis2box-1.0b7wis2box.env
docker volume rm wis2box_project_api-config

Optionally, add new entries to wis2box.env for environment variables for WIS2BOX_WEBAPP_USERNAME and WIS2BOX_WEBAPP_PASSWORD. If these are not defined, the default credentials are:

WIS2BOX_WEBAPP_USERNAME=wis2box-user
WIS2BOX_WEBAPP_PASSWORD=${WIS2BOX_STORAGE_PASSWORD}

Then start the wis2box from the new release-folder:

cd ~/wis2box-1.0b7
python3 wis2box-ctl.py start

And run the migration script to update your stations to match the new definitions as follows:

python3 wis2box-ctl.py execute python3 /app/migrations/v1_0b6_to_v1_0b7/update_station_definition_v1.0b7.py

Finally use the dataset editor in wis2box-webapp to recreate your datasets, see https://docs.wis2box.wis.wmo.int/en/1.0b7/user/setup.html#adding-datasets.

Note that you can also recreate your metadata by adding data mappings within your wis2box metadata configurations and use the wis2box dataset publish <file-name> command to publish

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b7

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

wis2box-1.0b7-dev

This is in the WIP-release for the upcoming wis2box-1.0b7 release, as reference by the 'latest' documentation.

1.0b6

This is the sixth 1.0 beta release of wis2box. This release includes a number of new features, enhancements and bugfixes. This release implements the required updates to the WIS2 Topic Hierarchy regarding dataset topics. Users are strongly encouraged to update to this latest release for WIS2 compliance.

Enhancements/updates:

• Implementation of the WIS2 Topic Hierarchy update to merge country and centre-id (see the section below)
• UI: add option/toggle to display station points as clusters #588
• duplicate-data-check: wis2box will not publish the same data twice and instead print an error #600
• updated-data-check: wis2box will detect updates and add a link to indicate data is updated #565
• WNM: provide data inline in notification messages when data size less than 4096 bytes
• Storage guidelines are now 24GB
• Station list: barometer height is no longer calculated if not defined (left null/blank)
• updating of BUFR tools to their latest versions
• collect NGINX logs in loki

Bug fixes:

• BUFR handling:
  • "Passed array is too small" fix (#559)
  • fix TSI matching (#581)
• WCMP2
  • fix update date calculation
  • fix contact information
  • fix metadata message publication
• WTH
  • remove pywcmp in lieu of pywis-topics for topic handling

Choosing a valid centre-id for the pre-operational phase

The WIS2 Topic Hierarchy merges country and centre-id in a new centre-id that uniquely defines a WIS2-node.

If you plan to use wis2box-1.0b6 to run a WIS2-node in the pre-operational phase, please ensure to use your new centre-id starting with your TLD during the configuration process.

If you are unsure how to define the centre-id for your WIS2-node, please contact WMO.

New installation

Please follow the instructions provided at https://docs.wis2box.wis.wmo.int/en/latest/user/getting-started.html

Migrating existing installations

The following steps are required when upgrading from wis2box 1.0b5 to 1.0b6:

# stop current wis2box
cd ~/wis2box-1.0b5 && python3 wis2box-ctl.py stop

# download new release in home-directory
cd ~/ && wget https://github.com/wmo-im/wis2box/releases/download/1.0b6/wis2box-setup-1.0b6.zip

# unzip and go to new directory
unzip wis2box-setup-1.0b6.zip && cd wis2box-1.0b6

IMPORTANT: before starting the new wis2box-release your configuration files need to be updated to use the new WTH. You can use the migration-script to update your configuration files

# run migration script to update configuration
python3 migration/update_configuration.py

The migration-script will ask you to answer a series of questions to replace country-code and old centre-id with your new centre-id, as per the following example:

Enter your 3-letter country-code: ita
3-letter country-code: ita ? (y/n) y
Enter your old centre-id: maaike-test
Old centre-id: maaike-test ? (y/n) y
Enter your new centre-id: it-maaike-test
New centre-id: it-maaike-test ? (y/n) y
Updated file 'metadata-synop.yml'
Updated file 'data-mappings.yml'

Check and verify that your data-mappings.yml file contains the new topic hierarchy. An example can be found below:

data:
    it-maaike-test.data.core.weather.surface-based-observations.synop:
        plugins:
            txt:
                - plugin: wis2box.data.synop2bufr.ObservationDataSYNOP2BUFR
                  notify: true

Continue with the update:

# start wis2box
python3 wis2box-ctl.py start  # wait until the new images are downloaded

# login to wis2box and execute commands
python3 wis2box-ctl.py login

# create new data collection, replace with your filename
wis2box data add-collection /data/wis2box/metadata/discovery/metadata-synop.yml

# publish new discovery metadata, replace with your filename
wis2box metadata discovery publish /data/wis2box/metadata/discovery/metadata-synop.yml

Verify the wis2box UI to ensure that there are 2 entries per dataset (one for old and one for new topic hierarchy):

Continue with the update:

# reindex your data collection items, replace with your old and new discovery metadata ID
wis2box data reindex-collection-items urn:x-wmo:md:ita:maaike-test:surface-based-observations.synop urn:x-wmo:md:it-maaike-test:surface-based-observations.synop

# unpublish your old dataset using the old discovery metadata ID
wis2box metadata discovery unpublish urn:x-wmo:md:ita:maaike-test:surface-based-observations.synop

# associate stations to the new topic hierarchy
wis2box metadata station add-topic origin/a/wis2/it-maaike-test/data/core/weather/surface-based-observations/synop

# exit the wis2box-management container
exit

Finally you can clean the unused Docker images to preserve diskspace:

docker image prune -a

(Or docker image prune without -a depending on your docker version)

As a final step, you will need to update any data ingestion processes to copy to new corresponding folder in the wis2box-incoming bucket

(Old)

(New)

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b6

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b5

This is the fifth 1.0 beta release of wis2box. This release includes a number of new features, enhancements and bugfixes.

Updates

wis2box-webapp (new container) provides an additional web application adding the following functionality:

• new station editor user interface with support to query OSCAR/Surface
• data entry via web-based form for SYNOP ASCII and CSV
• monitoring dashboard to via data notifications and download/inspect BUFR produced by wis2box

Updates to support wis2box configuration:

• new configuration creation script to allow the user to a create initial wis2box.env, data-mappings.yml and template discovery metadata files by answering a series of question on the command line
• environment files default.env and dev.env have been replaced by wis2box.env

Enhancements:

• add barometer height to station metadata management
• enable adding more user to default mosquitto broker
• fix to generate valid WNM (base64 integrity)
• fixes to generate valid WCMP2 (needs flushing of ES discovery-metadata)
• support for Docker Compose v2 (Note is it required or supported?)
• CSV publication (csv2bufr):
  • Improved logging and return of warnings and errors to the result object
  • BUFR templates now managed though the csv2bufr.templates module (note this will be moved to a stand alone module in a future release)
  • addition of minimal template for data from AWS weather stations

Bug fixes:

• fix dataset deletion workflow
• do not remove discovery metadata when running data cleaning
• BUFR publication: BUFR headers are now correctly set in extracted subsets. Previously the headers were not copied, leading to incorrect replication factors and data corruption
• GeoJSON API publication: code table entry description now correctly extracted and set in GeoJSON output. Previously only the first word of the description was extracted

important

• use the station editor in wis2box-webapp and ensure your stations are associated to a topic to ensure you can see the data in the "MAP" and "EXPLORE" option of wis2box-ui
• the cronjob running wis2box metadata station publish-collection in the wis2box-management every 10 minutes is now disabled

Migrating existing installations

The following steps are required when upgrading from wis2box 1.0b4 to 1.0b5:

# stop current wis2box
cd ~/wis2box-1.0b4 && python3 wis2box-ctl.py stop
# prune unused docker images
docker image prune -a
# delete api-config volume
docker volume rm wis2box_project_api-config
# download new release in home-directory
cd && wget https://github.com/wmo-im/wis2box/releases/download/1.0b5/wis2box-setup-1.0b5.zip
# unzip and go to new directory
unzip wis2box-setup-1.0b5.zip && cd wis2box-1.0b5
# run migration script to create wis2box.env
python3 migration/migration_create_wis2box_env.py
# check content of wis2box.env correctly copied your local updates in dev:env
cat wis2box.env
# run migration script to add barometer_height column to station_list.csv:
python3 migration/migration_update_station_list.py
# start wis2box
python3 wis2box-ctl.py start
# login to wis2box and execute commands
python3 wis2box-ctl.py login
# (inside wis2box-management container)
wis2box data add-collection /data/wis2box/surface-weather-observations.yml
wis2box metadata station publish-collection
# new images are downloaded (this may take awhile)

For installations older than 1.0b4, it is recommended to start from a clean/fresh installation.

To start a fresh installation, please clean your docker system after stopping your wis2box:

python3 wis2box-ctl.py stop
docker image prune -a
docker volume prune -a

...then delete the directory containing your current release and download the latest release and following the setup instructions.

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b5

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b4

This is the fourth beta release of wis2box. This release includes a number of bugfixes.

Updates

• discovery metadata publication: fix message publication for discovery metadata (#454)
• metrics: do not display failure descriptions: metrics (#455)
• update to latest stable versions of csv2bufr, synop2bufr and pymetdecoder (#457)
• safeguard station metadata geometry z (#458)

Migrating existing installations

There are no migrations required when upgrading from 1.0b3.

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b4

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int