Skip to content

Commit 6f254f2

Browse files
senesisStephane Senesirigoudyg
authored
Add operator plotmap, intake catalogs, fixes for IDRIS and check(), run on Obelix, validate plotmap with C-ESM-EP, allow for data check by ds() (#260)
* PEP008 on environment.py and test_example_data_plot.py * use generate_pdf.py also at IPSL * Remove test condition SkipIf_CondaEnv * Fix some project's root for IDRIS * Fix for IDRIS (atIDRIS diag, path for igcm_out) * Another fix for IDRIS (igcm_cmip6) * Fix concurrency issue in makedirs(tmpdir); don't use pdfjam at TGCC and IDRIS os.makedirs(tmpdir) more robust * Create tests/reference_data/test_data_plot/idris_20230611_V3.0_IPSL2 Only one plot change vs. spirit_conda_20221224 * Fix typo in ea4c992 (makedirs(test_ok=True)) * Improve tools_for_tests (compare_picture_files) compare_picture_files don't anymore stop on first instance of different pictures It also allows to create a html file showing the differences and the two images that differ * PEP008 on environment.py and test_example_data_plot.py * use generate_pdf.py also at IPSL * Remove test condition SkipIf_CondaEnv * Fix concurrency issue in makedirs(tmpdir); don't use pdfjam at TGCC and IDRIS os.makedirs(tmpdir) more robust * Fix typo in ea4c992 (makedirs(test_ok=True)) * Restore utils/remove_keys_with_same_values * Fix on : Restore utils/remove_keys_with_same_values * dataset.glob() also includes facet 'project' in returned dict(s) * dataset.glob() returns intersection of requested and avilable periods * dataset.glob() has argument ensure_period (default is True) * tests.tools_for_tests.build_diffs_html() now based on chtml * dataset.check handles monthly data (and cleaner) * PEP8 * Add facet 'frequency' in project CMIP6 * Add arg 'check' to ds(). Create period.build_date_rexep() * PEP008 * Some flesh in plotmap, and its interface in plot_operators.py * Change plotmap Climaf interface. Stage plotmap example and Aladin sfcWind file * 1st version OK with xarray for LambertII * Add script output format 'show' (no output, value forwarded to script) * Script call parameters are dumped in json format * Discard now extraneous " in script call parameters. Needs testing ! * Discard empty arguments in instanciated script calls * plotmap.py interface looks almost like gplot.ncl * Add data samples : Nemo, Aladin, uas + vas * Fix vector plots, titles. Add gv methods. CCRS from metadata Fix vector plots Fix titles Diagnose CCRS from metadata Add gv methods Default plot_engine is contourf * Fix cscript() call for Ncl scripts, and fix those scripts re. decoding "true" * Fix plotmap.py re. compute_xy and default projection * Stage plotmap_demo.py, python version of the plotmap example notebook * Fix tools_for_tests for comparing figure ensembles * For ensemble members title, fix combining ensemble title and member_label * version of plotmap.py mimicking gplot.ncl, with demo notebook * api.cshow now calls display o Image if running in a notebook * More fixes and improvements for plotmap and related notebooks * GitHub actions : upgrade versions. Update environment for plotmap * GitHub actions : fix setup-miniconda version * Github Actions : fix cache version and update test_environment.yml * Fix ipython in conda_test_env.yml * Create doc for plotmap + some changes on plotmap * Stage plotmapdemo and Getting_started_with_plotmap (html and ipynb) * Polish doc/news * Fix doc/conf.py * Introduce env.robust_pid for alleviating issue with docker at TGCC * Fix ensemble_time_series.py for case with only one time step * Handle IPSL intake catalogs * Fixes and doc (in news) for intake * Add forgotten module intake_search * Add project PMIP3 * Fixes re. intake related changes, and fix tests * Add reference_data/test_index_html/spiritIMv7 for ImageMagick V7 * Fix calling convert (arguments order) * Fix setting a robust PID when running in a container * At TGCC (i.e. in a container), use SLURM_JOBID as PID * Fix on using SLURM_JOBID * Use SLURM_JOBID also at IDRIS * gplot.ncl understands both 'true' and 'True' * store_wildcard_facet_value admits facet filenameVar * Fix intake_search and allow version='latest' for CMIP6 and CMIP5 * Fix intake re. cfreqs. Fix Cordex cfreqs. Fix intake_search. latest in Cordex * Add projects's method for solving ambiguity in facet values With instances for projects igcm_out and cmip5 * plotmap avoids decoding time when not necessary * Introduce a wrapper for 'plot', which may call plotmap Function plot_operators.plot calls plotmap if env.environment.plot_use_plotmap=True (otherwise it calls plot) It transforms some plot arguments to plotmap's equivalent * driver.scripts_ouput_write_mode allows for mode append for last.out * Fixes for plot wrapper and for classes.dataset.prefered_values() * plot wrapper handles cnMissingValFillColor and cnLineLabelsOn * Add driver.cxr(); fix bug in find_files when using fileNameVar Also fix typo in standard_operators (for curves) * chtml.cell allows to choose image filename * Add args for ensemble_time_series_plot : 'year_delta' and 'draw_grid' * Called scripts arguments are not json-coded when declared as {!arg} * Ensure backward compatbility for tests regarding chtml.cell() * Add operator cnkso, which calls ncks with a (possibly composite) operator * Update doc and news re. ensemble_ts_plot args draw_grid and year_delta * Changes to .gitignore * Fix intake_search re. e.g. CORDEX_domain * Fix gplot (NhlSetColor). Fix period.start_with re. fx. * Fix gplot re NhlSetColor. Fix ensemble_ts_plot re legendHandles * ds.resolve() stores files list. Fix operator rescale. Improve plot scripts ensemble_time_series_plot show months names on seasonal cycle plots gplot.ncl knows more latnames and lonnames * gplot.ncl does not change all colormaps when defining a grey * Fix errors with check for some observation datasets * Fix typos and polish doc * PEP8 * Test suite on GitHub uses Ubuntu 24.04 * Test suite on GitHub uses Miniforge3 * Doc build on GitHub uses Ubuntu 24.04 and Miniforge3 * Disable test_html in GitHub test workflow * For doc build on GitHub, update upload-artifact to v4 * GitHub test suite does install Image magick and pdftk * Fix some doc strings * Insert changes for obelix. Simplify find_files * chtml.cell() fixed re. file extension; ensemble_time_series_plot allows for single time * Netcdfbasics.varsOfFile() is flexible about nav_lat* * functions.tsplot() handle argument None * ds() has new args data_check and check_type, with default values in env * Fix cache.cdrop for index entries without corresponding file * Tune plotmap and its interface with a C-ESM-EP run. Fix hurs plot params For hurs bias : avoid defining 24 levels when using a 12 colors colormap * Default behaviour for ds() doesn't include a data check env.environment.data_check turned to False * Fix at CNRM --------- Co-authored-by: Stephane Senesi <ssenesi@obelix4.lsce.ipsl.fr> Co-authored-by: rigoudyg <gaelle.rigoudy@meteo.fr>
1 parent 6d118e3 commit 6f254f2

File tree

140 files changed

+27305
-2313
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

140 files changed

+27305
-2313
lines changed

.github/workflows/build_doc.yml

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -8,24 +8,24 @@ on:
88
jobs:
99
Sole_job_yet:
1010
name: Build doc
11-
runs-on: ubuntu-20.04
11+
runs-on: ubuntu-24.04
1212
defaults:
1313
run:
1414
shell: bash -el {0}
1515
steps:
1616
- name: Check out code
17-
uses: actions/checkout@v3
17+
uses: actions/checkout@v4
1818

19-
- name: Setup Mambaforge
20-
uses: conda-incubator/setup-miniconda@v2
19+
- name: Setup Miniforge3
20+
uses: conda-incubator/setup-miniconda@v3
2121
with:
22-
miniforge-variant: Mambaforge
22+
miniforge-variant: Miniforge3
2323
miniforge-version: latest
2424
activate-environment: climaf_test
2525
use-mamba: true
2626

2727
- name: Cache Conda env
28-
uses: actions/cache@v3
28+
uses: actions/cache@v4
2929
with:
3030
path: ${{ env.CONDA }}/envs
3131
key:
@@ -44,7 +44,7 @@ jobs:
4444
- run: (cd doc && make -k html)
4545

4646
- name: Archive doc
47-
uses: actions/upload-artifact@v3
47+
uses: actions/upload-artifact@v4
4848
with:
4949
name: doc
5050
retention-days: 5

.github/workflows/test_suite.yml

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@ on:
77

88
jobs:
99
Sole_job_yet:
10-
name: Testing with ubuntu 20.04, conda and python 3.11
11-
runs-on: ubuntu-20.04
10+
name: Testing with ubuntu 24.04, conda and python 3.11
11+
runs-on: ubuntu-24.04
1212
strategy:
1313
matrix:
1414
#python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
@@ -18,18 +18,30 @@ jobs:
1818
shell: bash -el {0}
1919
steps:
2020
- name: Check out code
21-
uses: actions/checkout@v3
21+
uses: actions/checkout@v4
2222

23-
- name: Setup Mambaforge
24-
uses: conda-incubator/setup-miniconda@v2
23+
- name: Setup Miniforge3
24+
uses: conda-incubator/setup-miniconda@v3
2525
with:
26-
miniforge-variant: Mambaforge
26+
miniforge-variant: Miniforge3
2727
miniforge-version: latest
2828
activate-environment: climaf_test
2929
use-mamba: true
3030

31+
- name: Mettre à jour les paquets
32+
run: sudo apt-get update
33+
34+
- name: Installer ImageMagick et pdftk
35+
run: |
36+
sudo apt-get install -y imagemagick pdftk
37+
38+
- name: Vérifier les installations
39+
run: |
40+
convert -version
41+
pdftk --version
42+
3143
- name: Cache Conda env
32-
uses: actions/cache@v3
44+
uses: actions/cache@v4
3345
with:
3446
path: ${{ env.CONDA }}/envs
3547
key:
@@ -58,7 +70,7 @@ jobs:
5870
- run: (cd tests && ./launch_tests_with_coverage.sh 1 3 "driver" 0)
5971
- run: (cd tests && ./launch_tests_with_coverage.sh 1 3 "dataloc" 0)
6072
- run: (cd tests && ./launch_tests_with_coverage.sh 1 3 "find_files" 0)
61-
- run: (cd tests && ./launch_tests_with_coverage.sh 1 3 "html" 0)
73+
#- run: (cd tests && ./launch_tests_with_coverage.sh 1 3 "html" 0)
6274
- run: (cd tests && ./launch_tests_with_coverage.sh 1 3 "example_data_retrieval" 0)
6375
- run: (cd tests && ./launch_tests_with_coverage.sh 1 3 "example_index_html" 0)
6476
- run: (cd tests && ./launch_tests_with_coverage.sh 1 3 "mcdo" 0)

.gitignore

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,4 +18,7 @@ examples/romain/
1818
/lmdz/
1919
/doc/_static/
2020
/testing/last.out
21-
NOTES
21+
NOTES
22+
/tests/a.pdf
23+
last.out
24+
/examples/climaf_tmp_*

climaf/ESMValTool_diags.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ def call_evt_script(climaf_name, script, ensembles, *operands, **parameters):
9595
'write_netcdf': True,
9696
'write_plots': True,
9797
'quick_plot': {},
98-
}
98+
}
9999

100100
# Account for dynamical, un-controlled, script call parameters to update settings
101101
settings.update(parameters)

climaf/__init__.py

Lines changed: 131 additions & 121 deletions
Original file line numberDiff line numberDiff line change
@@ -1,121 +1,131 @@
1-
#!/usr/bin/env python3
2-
# -*- coding: utf-8 -*-
3-
"""
4-
Climaf is documented at ReadTheDocs : http://climaf.readthedocs.org/
5-
6-
"""
7-
from __future__ import print_function, division, unicode_literals, absolute_import
8-
9-
import time
10-
import os
11-
12-
# Created : S.Sénési - 2014
13-
14-
__all__ = ["cache", "classes", "dataloc", "driver", "netcdfbasics",
15-
"operators", "period", "standard_operators", "cmacro", "chtml", "functions", "plot",
16-
"projects", "derived_variables", "ESMValTool_diags"]
17-
18-
19-
def tim(string=None):
20-
"""
21-
Utility function : print duration since last call
22-
Init it by a call without arg
23-
"""
24-
if not string or not getattr(tim, 'last'):
25-
tim.last = time.time()
26-
else:
27-
delta = time.time() - tim.last
28-
tim.last = time.time()
29-
# if ("dotiming" in vars() and dotiming) :
30-
print_delta = False
31-
if print_delta:
32-
print("Duration %.1f for step %s" % (delta, string), file=sys.stderr)
33-
34-
35-
already_inited = False
36-
onrtd = os.environ.get('READTHEDOCS', None) in ['True', ]
37-
38-
if not already_inited and not onrtd:
39-
import sys
40-
41-
#
42-
already_inited = True
43-
#
44-
tim()
45-
import atexit
46-
47-
tim("atexit")
48-
#
49-
from env.environment import *
50-
print("CliMAF climaf_version = " + climaf_version, file=sys.stderr)
51-
print("CliMAF install => " + "/".join(__file__.split("/")[:-2]), file=sys.stderr)
52-
53-
tim("softwares")
54-
#
55-
import env.clogging as clogging
56-
import env.site_settings as site_settings
57-
from . import cache
58-
from . import standard_operators
59-
from . import cmacro
60-
from . import operators
61-
62-
tim("imports")
63-
#
64-
# Set default logging levels
65-
clogging.logdir = logdir
66-
clogging.clog(loglevel)
67-
clogging.clog_file(logfilelevel)
68-
tim("loggings")
69-
#
70-
# Decide for cache location
71-
cachedir = default_cache # TODO: For compatibility, delete ones useless
72-
cache.setNewUniqueCache(default_cache, raz=False)
73-
print("Cache directory set to : " + default_cache + " (use $CLIMAF_CACHE if set) ", file=sys.stderr)
74-
tim("set cache")
75-
# Decide for cache location for remote data
76-
print("Cache directory for remote data set to : " + default_remote_cache + " (use $CLIMAF_REMOTE_CACHE if set) ",
77-
file=sys.stderr)
78-
#
79-
# Init dynamic CliMAF operators, and import projects and some funcs in main
80-
tim("execs_projects")
81-
exec("from climaf.classes import ds, eds, cens, fds", sys.modules['__main__'].__dict__)
82-
tim("execs_classes")
83-
exec("from climaf.operators import cscript", sys.modules['__main__'].__dict__)
84-
tim("execs_cscript")
85-
standard_operators.load_standard_operators()
86-
tim("load_ops")
87-
from . import projects
88-
exec("from climaf.projects import %s" % ",".join(projects.__all__), sys.modules['__main__'].__dict__)
89-
#
90-
# Read and execute user config file
91-
conf_file = os.path.expanduser("~/.climaf")
92-
if os.path.isfile(conf_file):
93-
exec(compile(open(conf_file).read(), conf_file, "exec"), sys.modules['__main__'].__dict__)
94-
tim(".climaf")
95-
#
96-
# Load cache scalar values
97-
cache.load_cvalues()
98-
tim("load_cvalues")
99-
#
100-
# Init and load macros
101-
macroFilename = os.environ.get("CLIMAF_MACROS", "~/.climaf.macros")
102-
cmacro.read(macroFilename)
103-
print("Available macros read from %s are : %s" % (macroFilename, repr(list(cmacros))),
104-
file=sys.stderr)
105-
tim("macros")
106-
#
107-
# Load cache index
108-
cache.cload()
109-
tim("cload")
110-
#
111-
atexit.register(cache.csync)
112-
atexit.register(cache.sync_cvalues)
113-
atexit.register(cmacro.write, macroFilename)
114-
tim("atexit")
115-
if cache.stamping:
116-
# Check if exiv2 is installed
117-
#
118-
# graphic_formats = environment.get_variable("climaf_graphic_formats")
119-
if (os.system("type exiv2 >/dev/null 2>&1") != 0) and 'eps' in graphic_formats:
120-
graphic_formats.remove('eps')
121-
print("exiv2 is not installed so you can not use 'eps' output format")
1+
#!/usr/bin/env python3
2+
# -*- coding: utf-8 -*-
3+
"""
4+
Climaf is documented at ReadTheDocs : http://climaf.readthedocs.org/
5+
6+
"""
7+
from __future__ import print_function, division, unicode_literals, absolute_import
8+
9+
import time
10+
import os
11+
12+
# Created : S.Sénési - 2014
13+
14+
__all__ = ["cache", "classes", "dataloc", "driver", "netcdfbasics",
15+
"operators", "period", "standard_operators", "plot_operators",
16+
"cmacro", "chtml", "functions", "plot",
17+
"projects", "derived_variables", "ESMValTool_diags"]
18+
19+
20+
def tim(string=None):
21+
"""
22+
Utility function : print duration since last call
23+
Init it by a call without arg
24+
"""
25+
if not string or not getattr(tim, 'last'):
26+
tim.last = time.time()
27+
else:
28+
delta = time.time() - tim.last
29+
tim.last = time.time()
30+
# if ("dotiming" in vars() and dotiming) :
31+
print_delta = False
32+
if print_delta:
33+
print("Duration %.1f for step %s" %
34+
(delta, string), file=sys.stderr)
35+
36+
37+
already_inited = False
38+
onrtd = os.environ.get('READTHEDOCS', None) in ['True', ]
39+
40+
if not already_inited and not onrtd:
41+
import sys
42+
43+
#
44+
already_inited = True
45+
#
46+
tim()
47+
import atexit
48+
49+
tim("atexit")
50+
#
51+
from env.environment import *
52+
print("CliMAF climaf_version = " + climaf_version, file=sys.stderr)
53+
print("CliMAF install => " +
54+
"/".join(__file__.split("/")[:-2]), file=sys.stderr)
55+
56+
tim("softwares")
57+
#
58+
import env.clogging as clogging
59+
import env.site_settings as site_settings
60+
from . import cache
61+
from . import standard_operators
62+
from . import plot_operators
63+
from . import cmacro
64+
from . import operators
65+
66+
tim("imports")
67+
#
68+
# Set default logging levels
69+
clogging.logdir = logdir
70+
clogging.clog(loglevel)
71+
clogging.clog_file(logfilelevel)
72+
tim("loggings")
73+
#
74+
# Decide for cache location
75+
cachedir = default_cache # TODO: For compatibility, delete ones useless
76+
cache.setNewUniqueCache(default_cache, raz=False)
77+
print("Cache directory set to : " + default_cache +
78+
" (use $CLIMAF_CACHE if set) ", file=sys.stderr)
79+
tim("set cache")
80+
# Decide for cache location for remote data
81+
print("Cache directory for remote data set to : " + default_remote_cache + " (use $CLIMAF_REMOTE_CACHE if set) ",
82+
file=sys.stderr)
83+
#
84+
# Init dynamic CliMAF operators, and import projects and some funcs in main
85+
tim("execs_projects")
86+
exec("from climaf.classes import ds, eds, cens, fds",
87+
sys.modules['__main__'].__dict__)
88+
tim("execs_classes")
89+
exec("from climaf.operators import cscript",
90+
sys.modules['__main__'].__dict__)
91+
tim("execs_cscript")
92+
standard_operators.load_standard_operators()
93+
plot_operators.load_plot_operators()
94+
tim("load_ops")
95+
from . import projects
96+
exec("from climaf.projects import %s" %
97+
",".join(projects.__all__), sys.modules['__main__'].__dict__)
98+
#
99+
# Read and execute user config file
100+
conf_file = os.path.expanduser("~/.climaf")
101+
if os.path.isfile(conf_file):
102+
exec(compile(open(conf_file).read(), conf_file, "exec"),
103+
sys.modules['__main__'].__dict__)
104+
tim(".climaf")
105+
#
106+
# Load cache scalar values
107+
cache.load_cvalues()
108+
tim("load_cvalues")
109+
#
110+
# Init and load macros
111+
macroFilename = os.environ.get("CLIMAF_MACROS", "~/.climaf.macros")
112+
cmacro.read(macroFilename)
113+
print("Available macros read from %s are : %s" % (macroFilename, repr(list(cmacros))),
114+
file=sys.stderr)
115+
tim("macros")
116+
#
117+
# Load cache index
118+
cache.cload()
119+
tim("cload")
120+
#
121+
atexit.register(cache.csync)
122+
atexit.register(cache.sync_cvalues)
123+
atexit.register(cmacro.write, macroFilename)
124+
tim("atexit")
125+
if cache.stamping:
126+
# Check if exiv2 is installed
127+
#
128+
# graphic_formats = environment.get_variable("climaf_graphic_formats")
129+
if (os.system("type exiv2 >/dev/null 2>&1") != 0) and 'eps' in graphic_formats:
130+
graphic_formats.remove('eps')
131+
print("exiv2 is not installed so you can not use 'eps' output format")

0 commit comments

Comments
 (0)