[Bug]: Error with standard script "walkthrough-basic" at searching cslcs

[sergeikharchenko]

Checked for duplicates

Yes - I've already checked

Describe the bug

Dear colleagues! The Dolphin works well for me with test case of Texas. But at the attempting to change AOI, track and dates I get the error:

---

IndexError Traceback (most recent call last)
Cell In[30], line 10
5 import opera_utils.download
7 aoi = (
8 "POLYGON((42.5569 43.2145,42.7224 43.2145,42.7224 43.2591,42.5569 43.2591,42.5569 43.2145))"
9 )
---> 10 results, options = opera_utils.download.search_cslcs(
11 aoi_polygon=aoi,
12 # We want to have the same set of dates for each Burst ID (spatial footprint)
13 check_missing_data=True,
14 track=50,
15 start="2024-06-01",
16 end="2024-10-01",
17 )
18 best_option = options[0]
20 slc_dir = Path("input_slcs")

File ~/miniconda3/lib/python3.11/site-packages/opera_utils/download.py:238, in search_l2(start, end, bounds, aoi_polygon, track, burst_ids, max_results, product, check_missing_data)
236 if not check_missing_data:
237 return results
--> 238 missing_data_options = get_missing_data_options(
239 slc_files=[r.properties["url"] for r in results]
240 )
241 return results, missing_data_options

File ~/miniconda3/lib/python3.11/site-packages/opera_utils/missing_data.py:96, in get_missing_data_options(slc_files, burst_id_date_tuples)
92 B = get_burst_id_date_incidence(burst_id_to_dates)
93 # In this matrix,
94 # - Each row corresponds to one of the possible burst IDs
95 # - Each column corresponds to one of the possible dates
---> 96 options = generate_burst_subset_options(B, all_burst_ids, all_dates)
97 # If they gave strings/files/urls, pick those out
98 if slc_files is not None:

File ~/miniconda3/lib/python3.11/site-packages/opera_utils/missing_data.py:255, in generate_burst_subset_options(B, burst_ids, dates)
252 break
253 date_idxs_to_exclude_combinations.append(set(flatten(combo)))
--> 255 all_column_idxs = set(range(B.shape[1]))
256 all_row_idxs = set(range(B.shape[0]))
258 # Track the row/col combinations that we've already

IndexError: tuple index out of range

What did you expect?

Reproducible steps

1.
2.
3.
...
If you have a `dolphin_config.yaml` that you used, you can attach that as well.

Environment

dolphin version: 0.38.0

Python deps:
        h5py: 3.10.0
         jax: 0.6.0
       numba: 0.59.1
       numpy: 1.26.4
 opera-utils: 0.19.0
    pydantic: None
      pyproj: 3.6.1
    rasterio: 1.3.9
 ruamel_yaml: None
       scipy: 1.11.4
threadpoolctl: 3.4.0
        tqdm: 4.66.1
  osgeo.gdal: 3.8.0

System:
      python: 3.11.8 | packaged by conda-forge | (main, Feb 16 2024, 20:53:32) [GCC 12.3.0]
  executable: /home/sergey/miniconda3/bin/python3
     machine: Linux-5.4.0-214-generic-x86_64-with-glibc2.31
Unwrapping packages:
      snaphu: None
       spurt: None
       isce3: None
       tophu: None
   whirlwind: None
optional GPU info:
         jax: 0.6.0
gpu_is_available: True
...

[scottstanie]

Hi, just checking that I'm reading it correctly- you're searching for a location in Europe?
The demo uses the OPERA geocoded slcs product (CSLC) since those are preprocessed to be in registered stacks. Unfortunately they only exist over North America at the moment, so the SAFE files (or Burst SLCs from ASF) have to be pre processed to become a coregistered stacks for dolphin. Some options for that processing are using isce2, isce3, GAMMA, or GMTSAR

[sergeikharchenko]

Yes. You are right, it's a site on the north slope of Greater Caucasus.
Thanks a lot, I'll try to use isce

[sergeikharchenko]

Hi again!
It's not the reason. I'm trying to work with Canada territory for now.

This code works:

ASF search

import subprocess
from os import chdir
from pathlib import Path

import opera_utils.download

aoi = (
"POLYGON((-135 68.2,-134 68.2,-134 68.6,-135 68.6,-135 68.2))"
)
results, options = opera_utils.download.search_cslcs(
aoi_polygon=aoi,
# We want to have the same set of dates for each Burst ID (spatial footprint)
check_missing_data=True,
track=108,
start="2020-05-31",
end="2024-05-31",
)
best_option = options[0]
#print(best_option)

This code doesn't works:

ASF search

import subprocess
from os import chdir
from pathlib import Path

import opera_utils.download

aoi = (
"POLYGON((-135 68.2,-134 68.2,-134 68.6,-135 68.6,-135 68.2))"
)
results, options = opera_utils.download.search_cslcs(
aoi_polygon=aoi,
# We want to have the same set of dates for each Burst ID (spatial footprint)
check_missing_data=True,
track=108,
start="2020-05-31",
end="2022-05-31",
)
best_option = options[0]

The one difference - is the day range. So is it can be related with gaps (or something else, because wider range works well) in Opera data even I'm interested in North America?

[scottstanie]

Ah sorry for not linking to the exact region covered:
https://github.com/nasa/opera-sds-pcm/blob/develop/geo/data/north_america_opera_expanded.geojson

it is actually more like "The United States + territories + cascadia subduction zone + 200 km into canada". Processing the entire country of Canada is outside the current budget