Skip to content

[API][INGEST][DATASTORE] Observation Aggregation Durations #291

@fjugipe

Description

@fjugipe

This is to open the discussion on data published to E-SOH. Currently some of data ingested to the system is described with positive iso8601 duration when it actually according to the EDR spec should be described with negative iso8601 duration ( i.e. some PT10M data should actually be -PT10M since the timestamp of the data ingested is describing the END and not the START of measurement period )

I at least have a hunch that the end user is not that interested if the timestamp is the END or the START since the measurement periods in the system are short. But this will have some effects on how we handle for example range filtering with the duration custom dimension as users querying duration=PT1M/PT10M are probably interested in both the negative and the positive durations between that range.

Some previous discussion of the case from Slack of order of importance with the data:

  1. The data should be as it says (so for a parameter, the observation time being the beginning or end of the period should be consistent with the sign of the duration). This is basic EDR spec consistency.
  2. The data in a specific system (e.g E-SOH) should be consistent.
  3. The different systems (E-SOH and WP5) should be consistent.

Currently we do not fulfill any of these, nor have the possibility to confirm the correctness of the data and instead need to rely on the data supplier, since both should be allowed according to the spec.

Metadata

Metadata

Assignees

No one assigned

    Labels

    APIFor issues and PR related to the APIsdatastoreFor issues and PR related to the datastoreingest

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions