Use modular JSON Schema for config

[rgov]

Currently, config validation uses type and optional information extracted from example.yaml.

We could consider making this more sophisticated by leveraging JSON Schema which can describe more complex rules, like enumeration choices, and externalize the validation logic to third-party dependencies.

Moreover, this would create the opportunity to create tools that can understand or generate configs, such as web UIs for operators, which could be less error-prone than editing complex YAML files.

This would require two components:

1. Have each node declare a JSON schema of what it reads from the parameter server. I suggest a directory of parameters/<node_name>.yaml. You could also provide schemata for third-party nodes.
2. From the launch file, extract the list of nodes, their names (for resolving ~), and the parameter values. I think this requires using the roslaunch Python package, not the command line.

As a nicety, we could consider using Pydantic in Python nodes to declare the schema as code, and then export it at build-time through a CMake macro.

from pydantic import BaseModel

class TaskA(BaseModel):
    interval: float
    count: int

class TaskB(BaseModel):
    interval: float
    threshold: float

class Parameters(BaseModel):
    tasks: list[TaskA|TaskB]

if __name__ == '__main__':
    print(json.dumps(MainModel.model_json_schema())

Then in the node, something like params: Parameters = load_parameters() would read and validate the parameters into an object representation.

Generated JSON Schema from example.yaml (in YAML)

$schema: http://json-schema.org/draft-07/schema#
properties:
  alerts:
    items:
      properties:
        type:
          type: string
        url:
          type: string
      required:
      - type
      - url
      type: object
    type: array
  arm_chanos:
    properties:
      ctd:
        properties:
          channels:
            items:
              type: string
            type: array
        required:
        - channels
        type: object
      ctd_topic:
        type: string
      motor:
        properties:
          address:
            type: string
          counts_per_turn:
            type: integer
          port:
            type: integer
          refresh_rate:
            type: integer
        required:
        - address
        - port
        - refresh_rate
        - counts_per_turn
        type: object
      tasks:
        properties:
          continuous_speed:
            type: number
          default_steps:
            items:
              type: number
            type: array
          downcast_type:
            type: string
          dwell_time:
            type: integer
          profiler_peak:
            properties:
              enabled:
                type: boolean
              offset_steps:
                items:
                  type: number
                type: array
              peak_expiration:
                type: integer
              threshold:
                type: number
            required:
            - enabled
            - threshold
            - peak_expiration
            - offset_steps
            type: object
          upcast_type:
            type: string
        required:
        - dwell_time
        - continuous_speed
        - downcast_type
        - upcast_type
        - profiler_peak
        - default_steps
        type: object
      winch:
        properties:
          epsilon:
            type: number
          gear_ratio:
            type: integer
          half_speed_dist:
            type: number
          max_speed:
            type: number
          range:
            properties:
              max:
                type: number
              min:
                type: number
            required:
            - min
            - max
            type: object
          safety_envelopes:
            properties:
              position:
                type: number
              time:
                properties:
                  extra_pct:
                    type: number
                  extra_sec:
                    type: number
                required:
                - extra_pct
                - extra_sec
                type: object
            required:
            - position
            - time
            type: object
          spool_circumference:
            type: number
        required:
        - range
        - spool_circumference
        - gear_ratio
        - max_speed
        - half_speed_dist
        - epsilon
        - safety_envelopes
        type: object
    required:
    - tasks
    - ctd_topic
    - ctd
    type: object
  arm_ifcb:
    properties:
      ctd_comms:
        properties:
          connection:
            properties:
              baud:
                type: integer
              data_bits:
                type: integer
              port:
                type: string
              type:
                type: string
            required:
            - type
            - port
            - baud
            - data_bits
            type: object
        required:
        - connection
        type: object
      ctd_topic:
        type: string
      ifcb_maintenance:
        properties:
          bead_interval:
            type: integer
          cartridge_debubble_interval:
            type: integer
          clean_interval:
            type: integer
        required:
        - cartridge_debubble_interval
        - clean_interval
        - bead_interval
        type: object
      motor:
        properties:
          address:
            type: string
          counts_per_turn:
            type: integer
          port:
            type: integer
          refresh_rate:
            type: integer
        required:
        - address
        - port
        - refresh_rate
        - counts_per_turn
        type: object
      profiler:
        properties:
          data_field:
            type: string
          data_topic:
            type: string
          peak_max_depth:
            type: number
          peak_min_depth:
            type: number
          resolution:
            type: number
        required:
        - resolution
        - data_topic
        - data_field
        - peak_min_depth
        - peak_max_depth
        type: object
      tasks:
        properties:
          profiler_peak:
            properties:
              threshold:
                type: number
            required:
            - threshold
            type: object
          scheduled_depth:
            properties:
              every:
                type: integer
              range:
                properties:
                  count:
                    type: integer
                  first:
                    type: number
                  last:
                    type: number
                required:
                - first
                - last
                - count
                type: object
            required:
            - every
            - range
            type: object
          wiz_probe:
            properties:
              default_depth:
                type: number
              duration:
                type: integer
              peak_offset:
                type: number
              preparation_window:
                type: integer
              times:
                items:
                  type: string
                type: array
              use_profiler_peak:
                type: boolean
            required:
            - times
            - duration
            - preparation_window
            - use_profiler_peak
            - peak_offset
            - default_depth
            type: object
        required:
        - profiler_peak
        - scheduled_depth
        - wiz_probe
        type: object
      winch:
        properties:
          epsilon:
            type: number
          gear_ratio:
            type: integer
          half_speed_dist:
            type: number
          max_speed:
            type: number
          range:
            properties:
              max:
                type: number
              min:
                type: number
            required:
            - min
            - max
            type: object
          safety_envelopes:
            properties:
              position:
                type: number
              time:
                properties:
                  extra_pct:
                    type: number
                  extra_sec:
                    type: number
                required:
                - extra_pct
                - extra_sec
                type: object
            required:
            - position
            - time
            type: object
          spool_circumference:
            type: number
        required:
        - range
        - spool_circumference
        - gear_ratio
        - max_speed
        - half_speed_dist
        - epsilon
        - safety_envelopes
        type: object
    required:
    - ifcb_maintenance
    - ctd_topic
    - ctd_comms
    - profiler
    type: object
  camera:
    properties:
      aft_camera:
        properties:
          video_stream_url:
            type: string
        required:
        - video_stream_url
        type: object
      fore_camera:
        properties:
          video_stream_url:
            type: string
        required:
        - video_stream_url
        type: object
    type: object
  classifier:
    properties:
      classifier_model:
        type: string
      image_topic:
        type: string
      triton_server_url:
        type: string
    required:
    - image_topic
    - triton_server_url
    - classifier_model
    type: object
  digital_logger:
    properties:
      address:
        type: string
      outlets:
        items:
          properties:
            name:
              type: string
            outlet:
              type: integer
          required:
          - name
          - outlet
          type: object
        type: array
      password:
        type: string
      username:
        type: string
    required:
    - username
    - password
    - address
    - outlets
    type: object
  gps:
    properties:
      host:
        type: string
    required:
    - host
    type: object
  ifcb:
    properties:
      address:
        type: string
      data_dir:
        type: string
      port:
        type: integer
      routines_dir:
        type: string
      serial:
        type: string
    required:
    - address
    - port
    - serial
    - routines_dir
    - data_dir
    type: object
  launch_args:
    properties:
      chanos_winch:
        type: boolean
      classifier:
        type: boolean
      digital_logger:
        type: boolean
      ifcb_winch:
        type: boolean
      log_dir:
        type: string
      rosbag_prefix:
        type: string
    required:
    - log_dir
    - rosbag_prefix
    - classifier
    - ifcb_winch
    - chanos_winch
    - digital_logger
    type: object
  lock_manager:
    properties:
      max_moving_winches:
        type: integer
    required:
    - max_moving_winches
    type: object
  name:
    type: string
  network_data_capture:
    properties:
      print_stats:
        type: boolean
      stats_interval:
        type: integer
      topics:
        properties:
          delimited_stream_example:
            properties:
              connection_type:
                type: string
              delimiter:
                type: string
              parsing_strategy:
                type: string
              port:
                type: integer
              subtopics:
                properties:
                  temperature:
                    properties:
                      field_id:
                        type: integer
                      type:
                        type: string
                    required:
                    - field_id
                    - type
                    type: object
                type: object
              use_regex_delimiter:
                type: boolean
            required:
            - connection_type
            - port
            - parsing_strategy
            - delimiter
            type: object
          rbr_udp_stream:
            properties:
              connection_type:
                type: string
              parsing_strategy:
                type: string
              port:
                type: integer
              subtopics:
                properties:
                  depth:
                    properties:
                      field_id:
                        type: string
                      type:
                        type: string
                    required:
                    - field_id
                    - type
                    type: object
                type: object
            required:
            - connection_type
            - port
            - parsing_strategy
            type: object
          ship_udp_example:
            properties:
              connection_type:
                enum:
                - udp
                - tcp
                type: string
              delimiter:
                type: string
              parsing_strategy:
                enum:
                - json_dict
                - json_array
                - raw
                - delimited
                type: string
              port:
                type: integer
              subtopics:
                properties:
                  example_subtopic:
                    properties:
                      field_id:
                        type: integer
                      type:
                        enum:
                        - str
                        - int
                        - float
                        - bool
                        - float[]
                        - int[]
                        - bool[]
                        type: string
                    required:
                    - field_id
                    - type
                    type: object
                  second_example:
                    properties:
                      field_id:
                        type: integer
                      type:
                        type: string
                    required:
                    - field_id
                    - type
                    type: object
                type: object
              use_regex_delimiter:
                type: boolean
            required:
            - connection_type
            - port
            - parsing_strategy
            type: object
        type: object
    required:
    - topics
    type: object
  web:
    properties:
      field_map:
        properties:
          commitHash:
            properties:
              default:
                type: string
              environment:
                type: string
            type: object
          ctdDepth:
            properties:
              default:
                type: number
              topic:
                type: string
              topic_field:
                type: string
            required:
            - default
            type: object
          defaultOnlyExample:
            properties:
              default:
                type: string
            required:
            - default
            type: object
          gpsLatitude:
            properties:
              default:
                type: number
              topic:
                type: string
              topic_field:
                type: string
            required:
            - topic
            - topic_field
            - default
            type: object
          gpsLongitude:
            properties:
              default:
                type: number
              topic:
                type: string
              topic_field:
                type: string
            required:
            - topic
            - topic_field
            - default
            type: object
        type: object
    required:
    - field_map
    type: object
required:
- name
- launch_args
- alerts
- gps
- lock_manager
- ifcb
- arm_ifcb
- camera
- web
type: object

[figuernd]

Hey I'm a big fan of JSON schema. I considered using that originally but wasn't sure whether this would create backwards compatibility issues with YAML so I just moved ahead with my hacky validation.

My other reason for using the example.yaml-as-schema was so that non-technical users could still understand and modify the schema on the fly if needed. JSON Schema syntax is pretty yucky so if we go down this path we will need to ensure we make our schemas flexible enough to accommodate every configuration scenario lest we create a validation lockout (but we could still provide the --no-validation escape hatch I have there already).

Pydantic would also obviate the ugliness of JSON schema, and I think Pydantic also can generate help strings? This would be critical for making the current schema understandable for end users, and may even let us move to a completely generated example.yaml.

All that to say: I think this is a really good idea.