RasterioIOError for LitPop

[Adventurer-E]

Describe the bug
A clear and concise description of what the bug is.
I encountered RasterioIOError when using climada.entity.exposures.LitPop.from_countries()

To Reproduce
Steps to reproduce the behavior/error:

1. Follow the tutorial 1_main_climada.ipynb
2. Make sure gpw-v4-population-count-rev11_2020_30_sec_tif.zip is downloaded in the current directory and unzipped (I checked this several times)
3. Run the code below.

Code example:

exp_litpop = LitPop.from_countries(
    ["United States"], res_arcsec=120
)

Note: The tutorial-given code uses "Puerto Rico", which works just fine. Once changing it to "United States" RasterioIOError occurs

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Climada Version: [Version or branch]

System Information (please complete the following information):

• Operating system and version: [e.g. Ubuntu 22.04, macOS 14.3.1, Windows 10]
• Python version: [e.g. 3.10]
  (to obtain this information execute > import sys >print(sys.version))

Additional context
Add any other context about the problem here.

[Adventurer-E]

CPLE_AppDefinedError Traceback (most recent call last)
CPLE_AppDefinedError: TIFFFillStrip:Read error at scanline 4294967295; got 0 bytes, expected 949

The above exception was the direct cause of the following exception:

CPLE_AppDefinedError Traceback (most recent call last)
CPLE_AppDefinedError: TIFFReadEncodedStrip() failed.

The above exception was the direct cause of the following exception:

CPLE_AppDefinedError Traceback (most recent call last)
File rasterio/_io.pyx:969, in rasterio._io.DatasetReaderBase._read()

File rasterio/_io.pyx:199, in rasterio._io.io_multi_band()

File rasterio/_io.pyx:205, in rasterio._io.io_multi_band()

File rasterio/_err.pyx:325, in rasterio._err.StackChecker.exc_wrap_int()

CPLE_AppDefinedError: BlackMarble_2016_A1_geo_gray.tif, band 1: IReadBlock failed at X offset 0, Y offset 15596: TIFFReadEncodedStrip() failed.

The above exception was the direct cause of the following exception:

RasterioIOError Traceback (most recent call last)
Cell In[11], line 1
----> 1 exp_litpop = LitPop.from_countries(
2 ["United States"], res_arcsec=120
3 ) # We'll go lower resolution than default to keep it simple
5 exp_litpop.plot_hexbin(pop_name=True, linewidth=4, buffer=0.1);

File ~/miniforge3/envs/climada_env/lib/python3.11/site-packages/climada/entity/exposures/litpop/litpop.py:204, in LitPop.from_countries(cls, countries, res_arcsec, exponents, fin_mode, total_values, admin1_calc, reference_year, gpw_version, data_dir)
187 litpop_list = [
188 _calc_admin1_one_country(
189 country,
(...) 198 for tot_value, country in zip(total_values, countries)
199 ]
201 else: # else, as default, country is initiated as a whole:
202 # loop over countries: litpop is initiated for each individual polygon
203 # within each country and combined at the end.
--> 204 litpop_list = [
205 cls._from_country(
206 country,
207 res_arcsec=res_arcsec,
208 exponents=exponents,
209 fin_mode=fin_mode,
210 total_value=total_values[idc],
211 reference_year=reference_year,
212 gpw_version=gpw_version,
213 data_dir=data_dir,
214 )
215 for idc, country in enumerate(countries)
216 ]
217 # make lists of countries with Exposure initaited and those ignored:
218 countries_in = [
219 country for lp, country in zip(litpop_list, countries) if lp is not None
220 ]

File ~/miniforge3/envs/climada_env/lib/python3.11/site-packages/climada/entity/exposures/litpop/litpop.py:205, in (.0)
187 litpop_list = [
188 _calc_admin1_one_country(
189 country,
(...) 198 for tot_value, country in zip(total_values, countries)
199 ]
201 else: # else, as default, country is initiated as a whole:
202 # loop over countries: litpop is initiated for each individual polygon
203 # within each country and combined at the end.
204 litpop_list = [
--> 205 cls._from_country(
206 country,
207 res_arcsec=res_arcsec,
208 exponents=exponents,
209 fin_mode=fin_mode,
210 total_value=total_values[idc],
211 reference_year=reference_year,
212 gpw_version=gpw_version,
213 data_dir=data_dir,
214 )
215 for idc, country in enumerate(countries)
216 ]
217 # make lists of countries with Exposure initaited and those ignored:
218 countries_in = [
219 country for lp, country in zip(litpop_list, countries) if lp is not None
220 ]

File ~/miniforge3/envs/climada_env/lib/python3.11/site-packages/climada/entity/exposures/litpop/litpop.py:797, in LitPop._from_country(country, res_arcsec, exponents, fin_mode, total_value, reference_year, gpw_version, data_dir)
788 # for countries with multiple sperated shapes (e.g., islands), data
789 # is initiated for each shape separately and 0 values (e.g. on sea)
790 # removed before combination, to save memory.
791 # loop over single polygons in country shape object:
792 for idx, polygon in enumerate(country_geometry.geoms):
793 # get litpop data for each polygon and combine into GeoDataFrame:
794 (
795 gdf_tmp,
796 meta_tmp,
--> 797 ) = _get_litpop_single_polygon(
798 polygon,
799 reference_year,
800 res_arcsec,
801 data_dir,
802 gpw_version,
803 exponents,
804 verbose=(idx > 0),
805 region_id=iso3n,
806 )
807 if gdf_tmp is None:
808 LOGGER.debug(
809 f"Skipping polygon with index {idx} for" + f" country {iso3a}."
810 )

File ~/miniforge3/envs/climada_env/lib/python3.11/site-packages/climada/entity/exposures/litpop/litpop.py:907, in _get_litpop_single_polygon(polygon, reference_year, res_arcsec, data_dir, gpw_version, exponents, region_id, verbose)
905 total_population = pop.sum()
906 # import nightlight data (2d array) and associated meta data:
--> 907 nlight, meta_nl = nl_util.load_nasa_nl_shape(
908 polygon, reference_year, data_dir=data_dir, dtype=float
909 )
911 # if resolution is the same as for lit (15 arcsec), set grid same as lit:
912 if res_arcsec == 15:

File ~/miniforge3/envs/climada_env/lib/python3.11/site-packages/climada/entity/exposures/litpop/nightlight.py:136, in load_nasa_nl_shape(geometry, year, data_dir, dtype)
133 # loop through required files, load and crop data for each:
134 for idx, i_file in enumerate(req_files):
135 # read cropped data from source file (src) to np.ndarray:
--> 136 out_image, meta_tmp = load_nasa_nl_shape_single_tile(
137 geometry, data_dir / (BM_FILENAMES[i_file] % (year))
138 )
139 # sort indicies to northenr and southern hemisphere:
140 if i_file in [0, 2, 4, 6]: # indicies of northern hemisphere files

File ~/miniforge3/envs/climada_env/lib/python3.11/site-packages/climada/entity/exposures/litpop/nightlight.py:412, in load_nasa_nl_shape_single_tile(geometry, path, layer)
409 # open tif source file with raterio:
410 with rasterio.open(path, "r") as src:
411 # read cropped data from source file (src) to np.ndarray:
--> 412 out_image, transform = rasterio.mask.mask(src, [geometry], crop=True)
413 LOGGER.debug("Read cropped %s as np.ndarray.", path.name)
414 if out_image.shape[0] < layer:

File ~/miniforge3/envs/climada_env/lib/python3.11/site-packages/rasterio/mask.py:191, in mask(dataset, shapes, all_touched, invert, nodata, filled, crop, pad, pad_width, indexes)
188 else:
189 out_shape = (len(indexes), ) + shape_mask.shape
--> 191 out_image = dataset.read(
192 window=window, out_shape=out_shape, masked=True, indexes=indexes)
194 out_image.mask = out_image.mask | shape_mask
196 if filled:

File rasterio/_io.pyx:644, in rasterio._io.DatasetReaderBase.read()

File rasterio/_io.pyx:972, in rasterio._io.DatasetReaderBase._read()

RasterioIOError: Read failed. See previous exception for details.

[chahank]

Dear @Adventurer-E ,

Could it be that United States is not the proper name of the country? Could you please try with the iso3alpha code USA or the full name United States of America?

[Adventurer-E]

Dear @Adventurer-E ,

Could it be that United States is not the proper name of the country? Could you please try with the iso3alpha code USA or the full name United States of America?

Hi chahank,
I tried with both, giving the same error. In fact, if I indeed give a non-existent country name, it gives ValueError: No valid country identified in ['aaa'], aborting., not RasterioIOError.

[emanuel-schmid]

Dear @Adventurer-E

I can confirm that what you try to do does work, normally.
You may try and remove the file BlackMarble_2016_A1_geo_gray.tif from the ~/climada/data/ folder. It seems to be corrupt.
When you run

exp_litpop = LitPop.from_countries(
    ["United States"], res_arcsec=120
)

again, it will be automaticallly downloaded.