speed of point data access

[markjessell]

Hi I have a notebook where I can draw a line across the geology map and it extracts the grav and mag profiles. It all works, which is amazing, hower it is quite slow in the data extraction step.

For the following code:

sample_values = np.array(netcdf_grid_utils.get_value_at_coords(sample_points, 'EPSG:4326', max_bytes=100))

it takes 16 seconds to extract 200 grav points, around 90 seconds to extract 200mag points, and nearly 5 minutes to extract 2000 mag points from these two sources:

"https://dapds00.nci.org.au//thredds/dodsC/iv65/Geoscience_Australia_Geophysics_Reference_Data_Collection/national_geophysical_compilations/Gravmap2019/Gravmap2019-grid-grv_cscba_1vd.nc"

"https://dapds00.nci.org.au/thredds/dodsC/iv65/Geoscience_Australia_Geophysics_Reference_Data_Collection/national_geophysical_compilations/Magmap2019/Magmap2019-grid-tmi-Cellsize40m-AWAGS_MAG_2019.nc"

So my question is whether there is a better (faster) way of extracting this information.

cheers

Mark

[RichardScottOZ]

Could try this with cloud optimised geotiff versions of these, Mark. Considering this currently.

[RichardScottOZ]

Or something like you would with Zarr :- https://medium.com/pangeo/fake-it-until-you-make-it-reading-goes-netcdf4-data-on-aws-s3-as-zarr-for-rapid-data-access-61e33f8fe685

[markjessell]

It seems to be a specific problem with sampling points, if I simply access the grid of the bounding box defined by the line, it is completed in a few seconds, so I will download the grid and sample it myself

[richardt94]

Hi Mark,

I realise you posted this issue quite some time ago now but I've had a go at making this faster. If you're still interested in speeding up this type of data access, please see the pull request above and let me know if it helps. You will likely need to set the max_bytes argument in your function call to be larger (I had good results with 50 MB) to take advantage of the improvements.

[RichardScottOZ]

@richardt94 any idea of the limitations of size/area/time span for this type of approach?

[LSgeo]

Could try this with cloud optimised geotiff versions of these, Mark. Considering this currently.

Hey Richard, 4 years in the future - have you found anything interesting in this space? I've just arrived at this consideration myself - specifically looking at GeoParquet, unless I find Zarr getting more widely adopted.