ODD PI 26.1 Objective 2: 🌍 Add dynamic tiling and timeseries support for datasets in CMR

[abarciauskas-bgse]

Motivation

TiTiler-CMR enables visualization and timeseries of datasets already catalogued in CMR, providing a low-effort way to integrate datasets into VEDA instances.

Description

Building upon the existing titiler ecosystem, this objective will allow performant map and timeseries visualization of data stored as archival file formats (e.g., GRIB, HDF5/NetCDF4, TIFF).

Acceptance Criteria

• Assess compatibility: Develop the next iteration of titiler-cmr compatibility reporting
  • feat: add /concept_metadata and /compatibility endpoints developmentseed/titiler-cmr#80
  • Creating a list of titiler-cmr compatible datasets and complimentary documentation of methodology #254
  • Merging benchmarking documentation from the datacube guide and titiler-cmr #255
• Develop operational readiness: Implement comprehensive monitoring and observability using OpenTelemetry (OTEL) infrastructure in titiler-cmr.
• Support EDL-based access to DAAC buckets
• Stretch: Establish service level objectives (SLOs) for tile generation and time series endpoints

[Jeanne-le-Roux]

Dependency for GHG Center, once titiler-cmr can support timeseries visualization of NetCDF-4 archival format, we would like to utilize it for the MiCASA dataset: https://github.com/US-GHG-Center/ghgc-architecture/issues/788

[hrodmn]

Dependency for GHG Center, once titiler-cmr can support timeseries visualization of NetCDF-4 archival format, we would like to utilize it for the MiCASA dataset: US-GHG-Center/ghgc-architecture#788

@Jeanne-le-Roux which MiCASA dataset are you interested in? I tested out concept id C3273639213-GES_DISC and got it working for the NPP variable:
https://staging.openveda.cloud/api/titiler-cmr/bbox/-180,-90,180,90.png?concept_id=C3273639213-GES_DISC&datetime=2018-02-12T09%3A00%3A00Z&variable=NPP&backend=xarray&colormap_name=greens&rescale=0,5e-8

[Jeanne-le-Roux]

@hrodmn we have both daily (C3273639213-GES_DISC) and monthly (C3273638632-GES_DISC) combined under one dataset in the GHG Center. I believe @siddharth0248 had tested it previously and confirmed that it worked, but titiler-cmr enabled timeseries analysis capability was not yet integrated with the VEDA dashboard/E&A tool (we also want some measure to assess performance against COG). Has this changed?

[abarciauskas-bgse]

My understanding is that timeseries from titiler-cmr is now supported in VEDA E+A so I will add a task to demonstrate that it is working with MiCASA and we should be able to do that in the first sprint.

[abarciauskas-bgse]

Regarding support for NISAR, flagged by @wildintellect, which could be available as soon as 3 weeks, during the interdependencies meeting it was discussed that it would be high-value to have a titiler-cmr instance with access to the NISAR data files in ASF as well as to access the GEDI L4B data from ORNL DAAC. Based on the value of integrating datasets like NISAR via titiler-cmr and lack of certainty on when more requester-pays access will be rolled out, I am ok with adding EDL-credential based access through a new instance of titiler-cmr.

But that raises 2 questions:

1. What would need to be deprioritized? Given @hrodmn is the best candidate to do this work, I would suggest we need to deprioritize timeseries/statistics in titiler-multidim
2. What is the right implementation? Should we reimplement EDL credential rotation, which I believe will only work for one DAAC per deployment, or do we want to go with another solution? Quick pro/con assessment
   • pros for using edl-credential-rotation: quick and straightforward to implement
   • cons for using edl-credential-rotation: only works for one DAAC.
   • Another solution could use a service token (apparently not encouraged) or the same set of credentials to fetch and cache s3 credentials for each DAAC.
   • I also imagine @hrodmn may have some brilliant ideas on how to handle this.

@hrodmn @wildintellect please let me know if you think we should move forward with tiling NISAR via titiler-cmr. It is a significant change to both our PI objectives and our stated API service support. But I think I agree it is worth it for such a high-value dataset.

[abarciauskas-bgse]

Noting also that GEDI L4B access was mentioned as a reason to integrate with Worldview, so that would be another reason to implement EDL-based access in titiler-cmr

[wildintellect]

@jjfrench may have thoughts, and potentially could contribute.

[abarciauskas-bgse]

@jjfrench @hrodmn since it impacts our objective commitments, could I get your input into these questions today?

1. @hrodmn do you agree it makes the most sense to deprioritize the statistics/timeseries work to support titiler-cmr to have EDL-based access (to at least NISAR at ASF and GEDI L4B at ORNL DAAC)? Or is something else you suggest we deprioritize?
2. @hrodmn @jjfrench do you think we should just move forward with having a titiler-cmr instance-per-daac paired with an edl-credential-rotation instance or should we evaluate some other options?

[hrodmn]

1. do you agree it makes the most sense to deprioritize the statistics/timeseries work to support titiler-cmr to have EDL-based access (to at least NISAR at ASF and GEDI L4B at ORNL DAAC)? Or is something else you suggest we deprioritize?

I think it would be fine to deprioritize the titiler-multidim time series + statistics work since the usefulness of that feature depends on more virtual datasets being generated. Since development and deployment of new datasets will be ongoing throughout the PI 26.1 let's put the time series work on the backburner.

2. do you think we should just move forward with having a titiler-cmr instance-per-daac paired with an edl-credential-rotation instance or should we evaluate some other options?

I think having DAAC-specific deployments might be what we need to do, but I have been thinking about a few alternatives, here is what I have so far:

• Ask these DAACs to give the veda-data-reader-role privileges to do requester pays requests 🤠 . Maybe that is not feasible within a single PI but that would be the simplest solution on our side.
• It might be possible to switch the application over from using S3 direct access links to https access by defining EARTHDATA_USERNAME and EARTHDATA_PASSWORD and manually requesting tokens, but I haven't gotten that to work for ASF yet. I don't think this will result in much slower data access, but it might, so it is probably not worth pursuing.
• titiler-cmr already has a mechanism for requesting (and caching) DAAC-specific S3 credentials for use in the data read operations. We are not currently using this capability because we have it configured to use an IAM role instead, and since the application is deployed in a Lambda function the cache would only be useful for warm Lambda instances. I think it would be worth switching this on for our dev deployment to see how it affects latency of tile requests in normal browsing sessions. Maybe with some provisioned concurrency (extra $) this system would work nicely. We could also explore deploying the application using ECS or something that is always-on - this might improve other aspects of the application's performance, too.