gdalinfo fails to recognize file on external mounted drive /media

[suatcoskun]

When i tried to run dolphin i took tihs error:

Traceback (most recent call last):
File "/home/suat/miniforge3/envs/dolphin-env/bin/dolphin", line 8, in
sys.exit(main())
^^^^^^
File "/home/suat/miniforge3/envs/dolphin-env/lib/python3.12/site-packages/dolphin/cli.py", line 32, in main
run_func(**arg_dict)
.
.
.

RuntimeError: NETCDF:"/media/suat/demir3/dolphin/dolphin/input_slcs/OPERA_L2_CSLC-S1_T071-151229-IW2_20240911T135300Z_20241012T053434Z_S1A_VV_v1.1.h5":"//data/VV": No such file or directory

Any advice?

[scottstanie]

Hi, two questions-

1. What happens when you run gdalinfo on that NETCDF string? Does the file exist and is it recognized?
2. Is that directory a mounted directory on your computer that is symlinked somewhere else?

[suatcoskun]

İf I give you some extra information:

1. Q: I installed Dolphin on my external ssd (media, not internal ssd) and tried running it there.
2. Q: When I run gdalinfo on my external ssd (media) l get an error message but if tried to run it on my internal ssd (home) everything looks good.

Maybe I tried to install Dolphin on my internal SSD, and I should try running it again.

Best.

[scottstanie]

Ok that's pretty weird haha.
It seems like this might not be a dolphin problem, but something with python/gdal and /media drive... That said, i'll turn this into a discussion, since a similar problem with /media also came up here: #405 (comment)
I'm interested to know what about these python installations can go wrong on ubuntu like this

[suatcoskun]

Hi @scottstanie

This time I tried to run Dolphin on my root disk. (Home) and I faced the same error again. But I caught an interesting thing about gdalinfo. If I change my path with HDF5 its work.

Then, I think to change my .yml files. ( in step 1 input_options). But I didn’t solve the issue

best
suat

[suatcoskun]

Anyway, it is a strange situation for me but I tried to run dolphin with docker and its done. Maybe docker installation will be the best choice for different systems. At that point, I have some questions about export products. Is that possible to plot time series on terminal and can we export results for .shp files?

[scottstanie]

So the timeseries folder has one geocoded raster per date; I haven't added many other export options, and I'm not sure what you're picturing for a Shapefile.

Are you trying to convert the raster data into some format like (x, y, time, value) and plot each point separately?
Do you have some example of the Vector data version of the output you wanted to see?

[suatcoskun]

Actually i have one answers for all this queastion YES . I am using mintpy, Stamps and many other opensource sowfware for InSAR processing. Many of them have diffrent export capibliltys to plot timeseries graphs. But in my opinon .shp export option is the most effective way for users. Because you can plot it in Qgis with a plugin and export them many diffrent file format. If you need some exapmles of this option you can look MintPy save_qgis.py script.

Best
Suat

[scottstanie]

If you're still looking for an example to do this, I tried one out here: https://gist.github.com/scottstanie/ac00b77b1d5e6b9dcaa904a9d14c8fc3
I might add a more refined version to the dolphin CLI at some point; for now, it does seem to work with https://github.com/luhipi/insar-explorer?tab=readme-ov-file when your QGIS has parquet support.

[suatcoskun]

Exciting news. I will test it and back to you soon.
Thanks

[suatcoskun]

I tried to run export_to_qgis.py code but I took this error message

My file directory looks like;

Any advice?

[scottstanie]

try making it --timeseries-pattern "timeseries/2*.tif", it's picking up the other geotiffs without dates in the names like conncomp_intersection.tif

[suatcoskun]

When I applied your suggestion it worked for me and I created output.parquet file. But when I try to open it in Qgis I took "/media/suat/demir3/dolphin/work-walkthrough/output.parquet is not a valid or recognized data source." error massage. Maybe it will be about my hole dolphin process so I will be start first step again. But I took attach my output file maybe you might take a look.
output.zip
Best...
Suat

[scottstanie]

ah that is probably a QGIS thing- it can be annoying to get geoparquet support.

I would just change the last line from wide_gdf.to_parquet(output_file, index=False) to wide_gdf.to_file("displacement.shp", index=False), or .gpkg if you want a geopackage

[suatcoskun]

Thank you. İt is done.

[suatcoskun]

Dear @scottstanie;

I had a chance to test dolphin at this weekend. Firstly I tested it most common area like Mexico City and these are results.

The first question; Is that possible to add coherence and std value option to export_to_qgis.py code?

How can remove low coherence value points? Which line can possible that in config.yml?

Is Dolphin apply atmospheric correction to our stack and how can we understand this? (I saw ERA5 option in config.yml)

Can we plotting wrap-unwrap phase, temporal coherence, avg_coherence, perpendicular baselines like Mintpy?

Best...

[scottstanie]

The first question; Is that possible to add coherence and std value option to export_to_qgis.py code?

There's a "temporal coherence" and "phase similarity" options which add those to the output; I don't know how the plotting works with those, perhaps @mahmud1 may know?

[scottstanie]

How can remove low coherence value points? Which line can possible that in config.yml?

I'm not sure what you mean by remove them. During the export, there's the --temporal-coherence-threshold option to not export those points.

Is Dolphin apply atmospheric correction to our stack and how can we understand this?
Right now the atmosphere portion of dolphin was very tailored toward the OPERA geocoded SLC inputs from Sentinel-1. If you want to apply if very soon, pyaps may be an easier option.
Same thing for the "Can it do <...> like Mintpy?" There's been a bunch of effort to export results to mintpy stacks for people who are already used to mintpy here: OPERA-Cal-Val/calval-DISP#2

[mahmud1]

There's a "temporal coherence" and "phase similarity" options which add those to the output; I don't know how the plotting works with those, perhaps @mahmud1 may know?

InSAR Explorer does not have any option yet to mask certain points based on a field value (certainly something useful to add in the future). Until then, you can either discard exporting those pixels with low temporal coherence (using --temporal-coherence-threshold) or removing them from the shapefile after export.