Error messages from `read_mdim()` that can neither be caught nor suppressed

[pepijn-devries]

When calling the code below:

Sys.setenv(AWS_S3_ENDPOINT     = "s3.waw3-1.cloudferro.com")
Sys.setenv(AWS_NO_SIGN_REQUEST = "YES")
Sys.setenv(AWS_VIRTUAL_HOSTING = "FALSE")

my_proxy <- stars::read_mdim(
  "/vsis3/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202406/2022/06/glo12_rg_1h-m_20220621-20220621_2D_hcst_R20220629.nc", proxy = TRUE
)

The following error message is printed in the console: "Error:NCpathcvt: stat=-36 (NetCDF: Invalid argument)". I suspect this is the netCDF driver used by GDAL trying (and failing) directory listing at the server. The problem is that I cannot catch these errors with tryCatch(), they can also not be suppressed with suppressWarnings() or suppressMessages().

Note that the code above still seems to work. But I think CRAN policy doesn't accept the use of stderr stream, which I think is the case. Maybe something to look into?

Cheers

[edzer]

I don't see any messages, with:

Sys.setenv(AWS_S3_ENDPOINT     = "s3.waw3-1.cloudferro.com")
Sys.setenv(AWS_NO_SIGN_REQUEST = "YES")
Sys.setenv(AWS_VIRTUAL_HOSTING = "FALSE")

library(stars)
# Loading required package: abind
# Loading required package: sf
# Linking to GEOS 3.12.2, GDAL 3.11.4, PROJ 9.4.1; sf_use_s2() is TRUE
stars::read_mdim(
  "/vsis3/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202406/2022/06/glo12_rg_1h-m_20220621-20220621_2D_hcst_R20220629.nc", proxy = TRUE
)
# stars_proxy object with 5 attributes in 5 file(s):
# $so
# [1] "[...]/glo12_rg_1h-m_20220621-20220621_2D_hcst_R20220629.nc"

# $zos
# [1] "[...]/glo12_rg_1h-m_20220621-20220621_2D_hcst_R20220629.nc"

# $thetao
# [1] "[...]/glo12_rg_1h-m_20220621-20220621_2D_hcst_R20220629.nc"

# $uo
# [1] "[...]/glo12_rg_1h-m_20220621-20220621_2D_hcst_R20220629.nc"

# $vo
# [1] "[...]/glo12_rg_1h-m_20220621-20220621_2D_hcst_R20220629.nc"

# dimension(s):
#           from   to                  offset   delta  refsys
# longitude    1 4320                      NA      NA      NA
# latitude     1 2041                      NA      NA      NA
# depth        1    1               0.494 [m]      NA udunits
# time         1   24 2022-06-21 00:30:00 UTC 1 hours POSIXct
#                                                  values x/y
# longitude  [-180.0417,-179.9583),...,[179.875,179.9584) [x]
# latitude  [-80.04167,-79.95833),...,[89.95834,90.04166) [y]
# depth                                              NULL    
# time                                               NULL    

[pepijn-devries]

Hi @edzer ,

Then, I think this is fixed in later versions of GDAL. I'm currently using 3.11.0. Could this be the case?

[kadyb]

Maybe this is not the GDAL version, but the operating system? Like Edzer, I have GDAL version 3.11.4, but on Windows 10, and I'm seeing these messages.

I don't see this error if I use the HDF5 driver to load this data:

stars::read_mdim(
  "HDF5:/vsis3/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202406/2022/06/glo12_rg_1h-m_20220621-20220621_2D_hcst_R20220629.nc",
  proxy = TRUE
)

[pepijn-devries]

Maybe this is not the GDAL version, but the operating system? Like Edzer, I have GDAL version 3.11.4, but on Windows 10, and I'm seeing these messages.

I'm using Windows as well, so this could be the case. Then maybe this is something that should be addressed. Perhaps some compiler flags are set differently for Windows?

I don't see this error if I use the HDF5 driver to load this data:

stars::read_mdim(
"HDF5:/vsis3/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202406/2022/06/glo12_rg_1h-m_20220621-20220621_2D_hcst_R20220629.nc",
proxy = TRUE
)

Thanks for the tip. I will give this driver a try.

[kadyb]

This may be known limitation of the netCDF driver on Windows/macOS: https://gdal.org/en/stable/drivers/raster/netcdf.html#vsi-virtual-file-system-api-support

[edzer]

Great catch!

[pepijn-devries]

Using the HDF5 driver fixed the issue for me as well.

stars::read_mdim(
"HDF5:/vsis3/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202406/2022/06/glo12_rg_1h-m_20220621-20220621_2D_hcst_R20220629.nc",
proxy = TRUE
)

Quoting the file name like this: "HDF5:\"/vsis3/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202406/2022/06/glo12_rg_1h-m_20220621-20220621_2D_hcst_R20220629.nc\"" didn't work. Any idea why?