Dataset problem: CLMcom-CCLM4-8-17 has big endian data #2057
Description
Describe the dataset issue
In cordex_fixes.CLMcomCCLM4817.fix_metadata(...) there is a fix for casting big-endian values in coords to np.float64. However, there are also big-endian values in the data, inconsistently between files, which causes error when concatenating. I propose making a similar cast for the data to np.float32 in fix_metadata(...). Example dataset:
- {dataset: CCLM4-8-17, project: CORDEX, domain: EUR-11,
exp: [historical, rcp45], ensemble: r12i1p1,
mip: day, institute: CLMcom, rcm_version: v1,
driver: ICHEC-EC-EARTH, version: v20140515}
Please attach
- The
main_log_debug.txtfile, this can also be found in therundirectory in the output directory
Relevant output:
Traceback (most recent call last):
File "/home/sm_joalo/.conda/envs/esmvaltool-climix/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/sm_joalo/.conda/envs/esmvaltool-climix/lib/python3.10/site-packages/esmvalcore/_task.py", line 795, in _run_task
output_files = task.run()
File "/home/sm_joalo/.conda/envs/esmvaltool-climix/lib/python3.10/site-packages/esmvalcore/_task.py", line 262, in run
self.output_files = self._run(input_files)
File "/home/sm_joalo/.conda/envs/esmvaltool-climix/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 678, in _run
product.cubes # pylint: disable=pointless-statement
File "/home/sm_joalo/.conda/envs/esmvaltool-climix/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 481, in cubes
self._cubes = [
File "/home/sm_joalo/.conda/envs/esmvaltool-climix/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 482, in <listcomp>
ds._load_with_callback(callback) for ds in self.datasets
File "/home/sm_joalo/.conda/envs/esmvaltool-climix/lib/python3.10/site-packages/esmvalcore/dataset.py", line 688, in _load_with_callback
cube = self._load(callback)
File "/home/sm_joalo/.conda/envs/esmvaltool-climix/lib/python3.10/site-packages/esmvalcore/dataset.py", line 765, in _load
result = preprocess(
File "/home/sm_joalo/.conda/envs/esmvaltool-climix/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 375, in preprocess
result.append(_run_preproc_function(function, items, settings,
File "/home/sm_joalo/.conda/envs/esmvaltool-climix/lib/python3.10/site-packages/esmvalcore/preprocessor/__init__.py", line 328, in _run_preproc_function
return function(items, **kwargs)
File "/home/sm_joalo/.conda/envs/esmvaltool-climix/lib/python3.10/site-packages/esmvalcore/preprocessor/_io.py", line 241, in concatenate
result = _by_two_concatenation([result, cube])
File "/home/sm_joalo/.conda/envs/esmvaltool-climix/lib/python3.10/site-packages/esmvalcore/preprocessor/_io.py", line 198, in _by_two_concatenation
_get_concatenation_error(concatenated)
File "/home/sm_joalo/.conda/envs/esmvaltool-climix/lib/python3.10/site-packages/esmvalcore/preprocessor/_io.py", line 217, in _get_concatenation_error
raise ValueError(f'Can not concatenate cubes: {msg}')
ValueError: Can not concatenate cubes: failed to concatenate into a single cube.
Dimension coordinates metadata differ: time != time
Data types differ: float32 != >f4
Cheers 🍺
Metadata
Metadata
Assignees
Type
Projects
Status