Skip to content

Commit 11fe5be

Browse files
jgroossjoernu76
jgrooss
and
joernu76
authored
Use aviso triggers to start file conversion (#44)
* decouple download freom convert of ECMWF data * decouple download freom convert of ECMWF data -- adaption of the configuration file * typo in lockfile exit added missing time units * Inclide trigger starting conversion jobs using the aviso system * Fixed test cases. --------- Co-authored-by: Joern Ungermann <[email protected]>
1 parent 47f8c82 commit 11fe5be

File tree

9 files changed

+451
-1
lines changed

9 files changed

+451
-1
lines changed

‎_test/mss/2021-01-29T00:00:00.an.sfc.nc‎

2.77 KB
Binary file not shown.

‎_test/test_cds.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,5 +29,6 @@ def test_get_cds(tmpdir):
2929
assert var in fut.variables
3030
for att in ["units", "standard_name"]:
3131
if att in ref[var].attrs:
32+
print(var, att, ref[var].attrs[att], fut[var].attrs[att])
3233
assert att in fut[var].attrs
3334
assert ref[var].attrs[att] == fut[var].attrs[att]

‎aviso/aviso_readme.txt‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
Use of aviso to trigger job scripts on the European Weather Cloud (EWC)
2+
=======================================================================
3+
4+
5+
* The aviso service runs automatically on EWC machines with the data setup
6+
It can be activated and deactivated by the command
7+
sudo systemctl start aviso.service
8+
sudo systemctl stop aviso.service
9+
The status can be viewd be
10+
sudo systemctl status aviso.service
11+
12+
* For test purposes it my be used to test the behaviour on past trigger
13+
events for a given (past) time range. This can be done by the listen
14+
command within an env, where aviso is installed, like
15+
sudo systemctl stop aviso.service
16+
mamba activate avisoenv
17+
aviso listen --from 2025-03-05T00:00:00.0Z --to 2025-03-05T08:00:00.0Z
18+
19+
* The file config.yaml in the direactory $HOME/.aviso guides the triggering
20+
21+
* if multiple steps are given in a trigger command, like
22+
step: [36,72,144]
23+
the programs called by the trigger are executed sequentially, possibly
24+
not in the time order of the triggers
25+
26+
* the script get_ecmwf_aviso.sh creates ECMWF grib files and converts
27+
them to MSS-convorm NetCDF files. One can speed up the process by
28+
parallel download the grib files using "DOWNLOAD_ONLY=yes" in the file
29+
settings.config and starting convert_all.sh separately.

‎aviso/config.yaml.parallel‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
listeners:
2+
- event: mars
3+
request:
4+
class: od
5+
expver: 1
6+
domain: g
7+
stream: oper
8+
step: 36
9+
triggers:
10+
- type: command
11+
working_dir: $HOME/data-retrieval/aviso
12+
command: $HOME/data-retrieval/bin/get_ecmwf_aviso.sh --date ${request.date} --time ${request.time} --step ${request.step} > mars_${request.time}_${request.step}.out
13+
- event: mars
14+
request:
15+
class: od
16+
expver: 1
17+
domain: g
18+
stream: oper
19+
step: 72
20+
triggers:
21+
- type: command
22+
working_dir: $HOME/aviso
23+
command: $HOME/data-retrieval/bin/get_ecmwf_aviso.sh --date ${request.date} --time ${request.time} --step ${request.step} > mars_${request.time}_${request.step}.out
24+
- event: mars
25+
request:
26+
class: od
27+
expver: 1
28+
domain: g
29+
stream: oper
30+
step: 108
31+
triggers:
32+
- type: command
33+
working_dir: $HOME/aviso
34+
command: $HOME/data-retrieval/bin/get_ecmwf_aviso.sh --date ${request.date} --time ${request.time} --step ${request.step} > mars_${request.time}_${request.step}.out
35+
- event: mars
36+
request:
37+
class: od
38+
expver: 1
39+
domain: g
40+
stream: oper
41+
step: 144
42+
triggers:
43+
- type: command
44+
working_dir: $HOME/aviso
45+
command: $HOME/data-retrieval/bin/get_ecmwf_aviso.sh --date ${request.date} --time ${request.time} --step ${request.step} > mars_${request.time}_${request.step}.out
46+
- event: mars
47+
request:
48+
class: od
49+
expver: 1
50+
domain: g
51+
stream: oper
52+
step: 228
53+
triggers:
54+
- type: command
55+
working_dir: $HOME/aviso
56+
command: $HOME/data-retrieval/bin/get_ecmwf_aviso.sh --date ${request.date} --time ${request.time} --step ${request.step} > mars_${request.time}_${request.step}.out

‎aviso/config.yaml.sequential‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
listeners:
2+
- event: mars
3+
request:
4+
class: od
5+
expver: 1
6+
domain: g
7+
stream: oper
8+
step: [36, 72, 108, 144, 228]
9+
triggers:
10+
- type: command
11+
working_dir: $HOME/data-retrieval/aviso
12+
command: $HOME/data-retrieval/bin/get_ecmwf_aviso.sh --date ${request.date} --time ${request.time} --step ${request.step} > mars_${request.time}_${request.step}.out

‎bin/convert.sh‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,10 @@ rm ${tmpfile}_z
5858
if [[ x$SFC_PARAMETERS != x"" ]]; then
5959
echo converting sfc
6060
cdo -f nc4c -t ecmwf copy grib/${BASE}.sfc.grib $sfcfile
61+
62+
cdo showatts $sfcfile
63+
echo "ncatted"
64+
6165
ncatted -O \
6266
-a standard_name,BLH,o,c,atmosphere_boundary_layer_thickness \
6367
-a standard_name,CI,o,c,sea_ice_area_fraction \
@@ -69,10 +73,23 @@ if [[ x$SFC_PARAMETERS != x"" ]]; then
6973
-a standard_name,SSTK,o,c,sea_surface_temperature \
7074
-a standard_name,U10M,o,c,surface_eastward_wind \
7175
-a standard_name,V10M,o,c,surface_northward_wind \
76+
-a standard_name,TCW,o,c,total_column_water \
77+
-a standard_name,TCWV,o,c,total_column_cloud_liquid_water \
78+
-a standard_name,T2M,o,c,2_meter_temperature \
79+
-a standard_name,D2M,o,c,2_meter_dewpoint \
80+
-a standard_name,ISHF,o,c,instantaneous_surface_sensible_heat_flux\
81+
-a standard_name,IEWS,o,c,instantaneous_x_surface_stress\
82+
-a standard_name,INSS,o,c,instantaneous_y_surface_stress\
7283
-a units,HCC,o,c,dimensionless \
7384
-a units,LCC,o,c,dimensionless \
7485
-a units,MCC,o,c,dimensionless \
86+
-a units,CI,o,c,dimensionless \
87+
-a units,LSM,o,c,dimensionless \
88+
-a units,ASN,o,c,dimensionless \
89+
-a units,SD,o,c,m \
7590
$sfcfile
91+
92+
cdo showatts $sfcfile
7693
fi
7794
# extract lnsp and remove lev dimension.
7895
grib_copy -w shortName=lnsp grib/${BASE}.ml2.grib ${tmpfile}

‎bin/convert_all.sh‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
#!/bin/bash
2+
#Copyright (C) 2025 by Forschungszentrum Juelich GmbH
3+
#Author(s): Jens-Uw Grooss
4+
5+
export MAINDIR=$HOME/data-retrieval
6+
export BINDIR=$MAINDIR/bin
7+
8+
. ${MAINDIR}/settings.default
9+
10+
if [ ! -f ${MAINDIR}/settings.config ]; then
11+
echo Please copy the settings.example to settings.config and configure your setup!
12+
exit 1
13+
fi
14+
15+
. ${MAINDIR}/settings.config
16+
17+
cd $WORKDIR
18+
19+
export YEAR=`date +%Y`
20+
export MONTH=`date +%m`
21+
export DAY=`date +%d`
22+
AMPM=`date +%p`
23+
if [ $AMPM == AM ]
24+
then
25+
export HH=00
26+
else
27+
export HH=12
28+
fi
29+
30+
31+
# script should start at 06h/18h to look for 00 12h forecast
32+
export h_exit=`date --date="+6hours" +%H`
33+
34+
for FCSTEP in $FCSTEPS
35+
do
36+
37+
# Set path, filenames and variables used later in the script
38+
export DATE=${YEAR}-${MONTH}-${DAY}
39+
export YMD=${YEAR}${MONTH}${DAY}
40+
export TIME=${HH}:00:00
41+
export BASE=${DATASET}.${YMD}T${HH}.${FCSTEP}
42+
export init_date=${DATE}T${TIME}
43+
echo BASE: $BASE
44+
export time_units="hours since ${init_date}"
45+
46+
47+
lockfile=grib/${DATASET}.${YMD}T${HH}.${FCSTEP}.ready
48+
echo `date` waiting for lockfile $lockfile
49+
until test -e $lockfile
50+
do
51+
sleep 30
52+
h_now=`date +%H`
53+
if [[ $h_now == $h_exit ]]
54+
then
55+
echo lockfile $lockfile not found by `date`
56+
echo exiting script
57+
exit 1
58+
fi
59+
done
60+
rm $lockfile
61+
62+
# Convert grib to netCDF, set init time
63+
echo `date`: converting ${FCSTEP}h forecast
64+
. $BINDIR/convert.sh
65+
66+
if [ $ECTRANS_ID == "none" ]
67+
then
68+
echo "no ectrans transfer -- move data to " $MSSDIR
69+
if [ x$TRANSFER_MODEL_LEVELS == x"yes" ]; then
70+
mv $mlfile $MSSDIR
71+
fi
72+
if [ -f $tlfile ]; then
73+
mv $tlfile $MSSDIR
74+
fi
75+
if [ -f $plfile ]; then
76+
mv $plfile $MSSDIR
77+
fi
78+
if [ -f $pvfile ]; then
79+
mv $pvfile $MSSDIR
80+
fi
81+
if [ -f $alfile ]; then
82+
mv $alfile $MSSDIR
83+
fi
84+
if [ -f $sfcfile ]; then
85+
mv $sfcfile $MSSDIR
86+
fi
87+
fi
88+
done
89+
echo `date`: converting finished
90+
91+
92+
if [[ x$CLEANUP == x"yes" ]]
93+
then
94+
export CYMD=${CLEANUP_YEAR}${CLEANUP_MONTH}${CLEANUP_DAY}
95+
export CBASE=${DATASET}.${CYMD}T${HH}.${FCSTEP}
96+
echo cleanup $CBASE
97+
98+
# clean up locally
99+
for f in $mlfile $tlfile $plfile $pvfile $alfile $sfcfile grib/${CBASE}*.grib;
100+
do
101+
if [ -f $f ];
102+
then
103+
rm $f
104+
fi
105+
done
106+
if [ $ECTRANS_ID == "none" ]
107+
then
108+
# clean up MSS server dir
109+
for f in $MSSDIR/${CBASE}*.nc
110+
do
111+
if [ -f $f ];
112+
then
113+
rm $f
114+
fi
115+
done
116+
fi
117+
fi
118+
echo `date`: ECMWF data converson finished

0 commit comments

Comments
 (0)