Attribute-value extension and standardisation

[ialarmedalien]

Some notes on extensions and standardisation for attribute-value set/pairs/whatever-you-want-to-call-them.

Add a type field to disambiguate between different types of value that may be stored and to allow extra validation.

The type should come from an enum so that we can control what's going in the field. Ideally the value in the attribute would dictate what appears in the type field, and we can use existing ontologies/controlled vocabularies to automatically populate it.

Examples:

properties:
  - attribute:
      id: MIXS:0000117
      label: total phosphorous
    raw_value: 2.2 ppm
    unit: ppm
    numeric_value: 2.2
    type: float

  - attribute:
      id: MIXS:0000011
      label: collection date
    raw_value: 12 Jun 2025
    value: 2025-06-12
    type: iso_datetime

  - attribute:
      label: n people on railway track
    value: 5
    type: integer

  - attribute:
      id: MIXS:0000012
      label: env_broad_scale
    value: terrestrial biome
    value_cv_id: ENVO:00000446
    type: cv_term   # controlled vocab term

  - attribute:
      label: smell
    value: completely disgusting
    type: text

  - attribute:
      label: size_of_bear
    value: big
    type: BearSizeEnum

  - attribute:
      label: random json data
    value: '{"this": "that", "the other": [1,3,5]}'
    type: json

Eventually this could be a form of data validation provided by BERtron itself on data integration; in these early stages, we would have to rely on data providers to add the appropriate type fields.

Capture units as text and as ontology IDs

Not everyone has an encyclopaedic knowledge of the unit ontology (more's the pity), so whilst it is useful to be able to use a controlled vocab to express units, it is not very user friendly. Providing both the controlled vocab ID for the unit and the text string might be a good compromise.

For example:

value: 2.2
unit: UO:0000008

would become

value: 2.2
unit: meter
unit_cv_id: UO:0000008

At a future time point, it would be good if the labels were populated on ingest into BERtron using preloaded reference ontologies. For now, we will have to rely on data providers to have the correct term names and term IDs.

Standardise representation of ontology IDs and their labels

attribute is split into label and id, whilst value has an accompanying value_cv_id sibling. Choose one representation or the other.

  - attribute:
      id: MIXS:0000097
      label: depth
    value:
      raw_value: 2.2m
      value: 2.2  # value.label / value.id doesn't make sense here but 'value.value' is not ideal!
    unit:
      id: UO:0000008
      label: meter
    type: float

  - attribute:
      id: MIXS:0000012
      label: env_broad_scale
    value: 
      id: ENVO:00000446
      label: terrestrial biome
    type: cv_term   # controlled vocab term - value is expressed as value.label and value.id

Another possibility:

  - attribute_label: depth
    attribute_cv_id: MIXS:0000097
    value: 2.2
    unit_label: meter
    unit_cv_id: UO:0000008
    type: float

  - attribute_label: env_broad_scale
    attribute_cv_id: MIXS:0000012
    value_label: terrestrial biome
    value_cv_id: ENVO:00000446
    type: cv_term

A simpler version:

  - attribute: depth
    attribute_cv_id: MIXS:0000097
    value: 2.2
    unit: meter
    unit_cv_id: UO:0000008
    type: float

  - attribute: env_broad_scale
    attribute_cv_id: MIXS:0000012
    value: terrestrial biome
    value_cv_id: ENVO:00000446
    type: cv_term

[jeff-cohere]

Re: types: 👍. Always nice to know what one is working with!

Re: Ontology IDs: I like the simpler version just above, but I worry that the accompanying _cv_id fields could introduce a version of the caching problem when I imagine the copy-pasting that is going to be happening, unless we don't envision humans directly editing these files. One way I've seen this addressed elsewhere is to e.g. have a "dictionary" of possible unit types mapped to their ontology IDs, which provides validation (you can't use it if it's not in there) and consistency (the unit maps unambiguously to its Ontology ID), but this may be too heavyhanded or fly in the face of established norms in LinkML.

I should probably learn more about LinkML. :-)

[jeff-cohere]

Thoughts on the value field:

• do we really need raw_value? If not, we can treat value as either a literal (in the case of float, say) or a mapping (if an enumerated type like cv_term)

[jeff-cohere]

@ialarmedalien and I are leaning toward the first of the three examples above, in which selected fields within an attribute can be mappings with label and id subfields, and thinking of jettisoning the raw_value field so that literal values are scalars.

[jeff-cohere]

Regarding units: should we consider using LinkML's enum -> vocabulary mappings?

[ialarmedalien]

Horrible view of prospective types of attribute-values:

  # require one of these to describe the attribute
  attribute_cv_id:       # attribute ID from controlled vocabulary / ontology / schema
  attribute_cv_label:    # term name from controlled vocab
  attribute_string:      # unformatted, uncontrolled attribute

  # optional
  raw_value:             # the original, unparsed version

  # text values - one of these is required
  value_cv_id:           # value ID from a controlled vocab / ontology / schema
  value_cv_label:        # term name from controlled vocab
  value_string:          # unformatted, uncontrolled string value

  # date/time - required
  datetime:              # ISO8601 compliant date time

  # numeric value - required
  numeric_value:         # a numeric value - int / float / etc.

  # unit - require either ID or label if present. Used with numeric values.
  unit_cv_id:            # unit ID from unit ontology / UCUM
  unit_cv_label:         # term name from unit ontology / UCUM
  unit_string:           # wtf? you haven't searched for this properly

possible attr-values:

  value_cv_source:       # source CV / enum (not sure if this is worthwhile or not)
  minimum_numeric_value: # could be minumum_<attr_name>
  maximum_numeric_value: # could be maximum_<attr_name>

Different classes have different combinations of the above slots - e.g. QuantityValue = an attribute, numeric_value, a unit (optional), and raw_value. An attr-value that uses terms from a controlled vocab could be an attribute, a value, and an optional raw_value.

[ialarmedalien]

(Above comment is an attempt to enumerate the possible fields if we were to have a single flat table for all attr/value representation.)

[mslarae13]

I'm pretty lost on this... to quote @turbomam I would like to see an example data set. 😅

[ialarmedalien]

Here's an example with various different data types:

properties:
    # QuantityValue where the quantity is an integer
  - type: integer
    attribute_string: n_people_on_railway_track
    numeric_value: 5
    raw_value: "FIVE"  # raw data, parsed to produce the numeric value

    # QuantityValue where the quantity is a float
  - type: float
    attribute_cv_id: MIXS:0000117
    attribute_cv_label: total phosphorous
    numeric_value: 2.2
    unit_cv_id: UO:0000169
    unit_cv_label: parts per million
    raw_value: "2.2 ppm"

    # DateTimeValue (proposed)
  - type: iso_datetime
    attribute_cv_id: MIXS:0000011
    attribute_cv_label: collection date
    datetime: 2025-06-12
    raw_value: "12 Jun 2025"

    # TextValue
  - type: text         # free text - no ontology or enumeration for potential values
    attribute_string: smell
    value_string: completely disgusting
    raw_value:   "  Completely disgusting!! "

    # TextValue - analogous to NMDC's ControlledTermValue
  - type: cv_term      # term from an enum
    attribute_string: size_of_bear
    value_cv_label: big
    value_source: BearSizeEnum

    # TextValue - analogous to NMDC's ControlledIdentifiedTermValue
  - type: cv_term      # term from a controlled vocab
    attribute_cv_id: MIXS:0000012
    attribute_cv_label: env_broad_scale
    value_cv_id: ENVO:00000446
    value_cv_label: terrestrial biome

    # example for capturing JSON data
  - type: json         # possible field for capturing more complex data
    attribute_string: random json data
    raw_value: '{"this": "that", "the other": [1,3,5]}'

NMDC links:

• ControlledTermValue
• ControlledIdentifiedTermValue

The different types of value listed above would have different validation rules; the *_cv_label and *_cv_id could be validated against the source ontology, schema, or enumeration. I could imagine having an interface to the schema that hid away a lot of the details, so that you could just do something like:

prop = sample.properties[0]
prop.attribute        # total_phosphorus
prop.value            # 2.2
prop.unit             # parts per million
prop.unit_cv_id       # UO:0000169
prop.formatted_value  # 2.2 parts per million

prop = sample.properties[2]
prop.attribute        # env_broad_scale
prop.value            # terrestrial biome
prop.unit             # None
prop.formatted_attr_value    
                      # env_broad_scale (MIXS:000012): terrestrial biome (ENVO:0000446)

This would be pretty simple to implement in pydantic.