Add advanced sensor models to Mosaico Ontology

[LuigiTer]

Description

Define and implement ontology models in Mosaico for the following advanced sensor types:

• LiDAR (e.g. Velodyne, Ouster, ...)
• RGB-D (depth cameras, e.g. Intel RealSense, Microsoft Azure Kinect)
• Radar (e.g. automotive or drone-grade radars)
• Laser (single/multi-echo laser rangefinders)

Each sensor type requires a dedicated, sensor-agnostic, framework-independent ontology model that becomes the canonical representation inside Mosaico.

---

Approach by Sensor Type

LiDAR, RGB-D, and Radar

For these three sensor types, the starting point is the analysis of the ROS PointCloud2 message format (sensor_msgs/PointCloud2).

Since no unified standard exists across manufacturers, the goal is to survey the most common vendors for each sensor type and identify which fields are most widely used in practice. Fields that are universally supported should become required (or strongly recommended) attributes of the ontology model, while vendor-specific fields should be stored in a dedicated extra_attributes field.

Vendors to survey (non-exhaustive):

Sensor Type	Example Vendors
LiDAR	Ouster, Velodyne, Livox, Hesai, Robosense
RGB-D	Intel RealSense, Microsoft Azure Kinect, Stereolabs ZED
Radar	Texas Instruments, Ainstein, Smartmicro, Continental

Laser (Single/Multi-echo Laser Rangefinders)

For laser sensors, the relevant ROS messages are:

• sensor_msgs/LaserScan
• sensor_msg/LaserEcho used with sensor_msgs/MultiEchoLaserScan

The ontology model for Laser should be defined starting from these message structures.

[LuigiTer]

LIDAR Proposed Solution

Before defining this ontology, we surveyed how different LiDAR vendors structure their data formats. Since no common standard exists across manufacturers, the LiDAR ontology covers the most widespread attributes, while any vendor-specific fields that don't match the defined schema are stored in a dedicated extra_attributes field.

The LiDAR ontology is split into two main classes:

LidarScan: represents the scans of the environment captured by a LiDAR sensor.
Lidar: represents the LiDAR sensor, holding a LidarScan with the captured data.

---

Following the Lidar ontology class, starting from LidarScan:

class LidarScan(Serializable):
    __msco_pyarrow_struct__ = pa.struct(
        [
            pa.field(
                "x",
                pa.list_(pa.float32()),
                nullable=False,
                metadata={"description": "x coordinates in meters"},
            ),
            pa.field(
                "y",
                pa.list_(pa.float32()),
                nullable=False,
                metadata={"description": "y coordinates in meters"},
            ),
            pa.field(
                "z",
                pa.list_(pa.float32()),
                nullable=False,
                metadata={"description": "z coordinates in meters"},
            ),
            pa.field(
                "intensity",
                pa.list_(pa.float32()),
                nullable=True,
                metadata={
                    "description": "strength of the returned signal."
                },
            ),
            pa.field(
                "reflectivity",
                pa.list_(pa.uint16()),
                nullable=True,
                metadata={"description": "surface reflectivity"},
            ),
            pa.field(
                "beam_id",
                pa.list_(pa.uint16()),
                nullable=True,
                metadata={
                    "description": "beam index (ring, channel), identifies which laser fired the point"
                },
            ),
            pa.field(
                "scan_range",
                pa.list_(pa.float32()),
                nullable=True,
                metadata={"description": "range in meters"},
            ),
            pa.field(
                "return_type",
                pa.list_(pa.uint8()),
                nullable=True,
                metadata={
                    "description": "single/dual return classification, manufacturer-specific"
                },
            ),
            pa.field(
                "timestamp",
                pa.list_(pa.float64()),
                nullable=True,
                metadata={
                    "description": "per-point acquisition time offset from scan start"
                },
            ),
            pa.field(
                "extra_attributes",
                pa.map_(pa.string(), pa.large_binary()),
                nullable=True,
                metadata={
                    "description": "extra attributes, manufacturer-specific"
                },
            ),
        ]
    )
    
    x: List[float]
    y: List[float]
    z: List[float]
    intensity: Optional[List[float]] = None
    reflectivity: Optional[List[int]] = None
    beam_id: Optional[List[int]] = None
    scan_range: Optional[List[float]] = None
    return_type: Optional[List[int]] = None
    timestamp: Optional[List[float]] = None
    extra_attributes: Optional[Dict[str, bytes]] = None

While the Lidar class is:

class Lidar(Serializable):
    __msco_pyarrow_struct__ = pa.struct([
        pa.field(
            "lidar_scans",
            LidarScan.__msco_pyarrow_struct__,
            nullable=False,
            metadata={
                "description": "lidar scans"
            },
        ),
    ])
    
    lidar_scans: LidarScan

---

Considerations

• The approach used for defining the structure of Lidar and LidarScan is the Struct of Array (SoA). SoA keeps each field (x, y, z, intensity…) in a separate contiguous array rather than grouping all fields per point. This maps naturally to LiDAR access patterns, eliminates per-point Python object overhead, and serializes to PyArrow columnar format with zero copies.

• Another point to consider is adding encode and decode methods to each ontology. These are particularly important for the Lidar ontology, where extra_attributes can grow large, encoding allows custom compression or binary packing strategies to reduce memory footprint during serialization.

[fdicorato]

lidar_scans: LidarScan should be a List

class Lidar(Serializable):
    __msco_pyarrow_struct__ = pa.struct([
        pa.field(
            "lidar_scans",
            pa.list_(value_type=LidarScan.__msco_pyarrow_struct__),
            nullable=False,
            metadata={
                "description": "lidar scans"
            },
        ),
    ])
    
    lidar_scans: List[LidarScan]

[fdicorato]

Ok, i understand my mistake: the LidarScan is made by attributes that already are lists! (prefer fixed number of columns in parquet)

• I would eliminate the Lidar class, which is just a 1-1 wrapper of the LidarScan
• I would rename the class LidarScanto Lidar
• The extra_attributes might be a (optional) List[Dict[str, Any]] (i.e. every point in the scan would have its own attributes, so a list)
  • I intend to keep the extra attributes as regular dict and not a dict of bytes (so the user can inspect their value)
• The pa.field("extra_attributes",... can be directly a binary and not a map?

[LuigiTer]

With this suggestion, the structure of the Lidar Ontology is defined. Since pydantic will raise an error on the matching with the type of extra_attributes, as mentioned in the considerations, it will be necessary to implement an encode and decode logic, also taking into account the possibility of using pydantic validators: https://docs.pydantic.dev/latest/concepts/validators/

[LuigiTer]

LaserScan Ontology Proposed solution

This proposes the canonical field set for the LaserScan ontology type within this project, aligned with the established ROS message definitions for sensor_msgs/LaserScan and sensor_msgs/MultiEchoLaserScan.

---

Proposed Field Set

The following fields are proposed for the LaserBaseScan ontology class. All names and semantics are derived directly from sensor_msgs/LaserScan and sensor_msgs/MultiEchoLaserScan.

class LaserScanBase(Generic[T]):
    angle_min:       float                              # Start angle of the scan [rad]
    angle_max:       float                              # End angle of the scan [rad]
    angle_increment: float                              # Angular distance between measurements [rad]
    time_increment:  float                              # Time between individual measurements [s]
    scan_time:       float                              # Time between scans (full rotation) [s]
    range_min:       float                              # Minimum valid range value [m]
    range_max:       float                              # Maximum valid range value [m]
    ranges:          T                                  # Range data — List[float] or List[List[float]]
    intensities:     Optional[T] = None                 # Intensity data, same shape as ranges

Type Parameter T

The use of a generic T allows the same ontology class to serve both ontology variants:

• LaserScan → T = List[float] (one range per beam)
• MultiEchoLaserScan → T = List[List[float]] (multiple echoes per beam)

[LuigiTer]

Radar Ontology

This proposes the canonical field set for the Radar ontology type within this project, grounded in the ROS (sensor_msgs/PointCloud2)[https://docs.ros2.org/foxy/api/sensor_msgs/msg/PointCloud2.html] message format.

---

Proposed Field Set

The following fields are proposed for the Radar ontology class. Mandatory fields map to the x, y, z channel convention in sensor_msgs/PointCloud2; optional fields correspond to named channels commonly added by radar drivers and post-processing nodes.

class RadarOntology:
    # --- Mandatory Cartesian position (PointCloud2 standard channels) ---
    x: List[float]                              # X position of each detection [m]
    y: List[float]                              # Y position of each detection [m]
    z: List[float]                              # Z position of each detection [m]
 
    # --- Spherical coordinates (derived or directly reported) ---
    range:     Optional[List[float]] = None     # Radial distance from sensor [m]
    azimuth:   Optional[List[float]] = None     # Azimuth angle [rad]
    elevation: Optional[List[float]] = None     # Elevation angle [rad]
 
    # --- Radar-specific quality metrics ---
    rcs: Optional[List[float]] = None           # Radar cross-section [dBsm]
    snr: Optional[List[float]] = None           # Signal-to-noise ratio [dB]
 
    # --- Velocity ---
    doppler_velocity: Optional[List[float]] = None  # Raw Doppler velocity [m/s]
    radial_speed:     Optional[List[float]] = None  # Radial speed (signed) [m/s]
 
    # --- Velocity Cartesian components ---
    vx: Optional[List[float]] = None            # Velocity in X, sensor frame [m/s]
    vy: Optional[List[float]] = None            # Velocity in Y, sensor frame [m/s]
 
    vx_comp: Optional[List[float]] = None       # Compensated velocity in X [m/s]
    vy_comp: Optional[List[float]] = None       # Compensated velocity in Y [m/s]
 
    # --- Acceleration ---
    ax: Optional[List[float]] = None            # Acceleration in X [m/s²]
    ay: Optional[List[float]] = None            # Acceleration in Y [m/s²]
 
    # --- Escape hatch for non-standard channels ---
    extra_attributes: Optional[Dict[str, Any]] = None  # Vendor-specific fields

[LuigiTer]

Depth Camera Ontology - RGBDCamera, ToFCamera, StereoCamera

This proposes the canonical ontology for depth-camera sensor data within this project. Three concrete subtypes are defined: RGBDCamera, ToFCamera, and StereoCamera. Each inheriting from a shared DepthCamera base.

---

Architecture Overview

The ontology is structured in three layers:

pa.Field lists          Python model hierarchy         Arrow schema
─────────────────       ────────────────────────       ──────────────────────────
_COMMON_FIELD     ──►  DepthCamera (BaseModel)  ──►   _build_schema(_COMMON_FIELD)
_TOF_FIELDS       ──►  ToFCamera                 ──►  _build_schema(_COMMON_FIELD, _TOF_FIELDS)
_STEREO_FIELDS    ──►  StereoCamera              ──►  _build_schema(_COMMON_FIELD, _STEREO_FIELDS)

_build_schema flattens any number of pa.Field groups and appends _EXTRA_FIELD, returning a pa.StructType. Each concrete subclass attaches the result as __msco_pyarrow_struct__.

---

Proposed Field Sets

Common fields (_COMMON_FIELD)

Field	Arrow type	Nullable	Description
x	list<float32>	No	Horizontal position derived from depth [m].
y	list<float32>	No	Vertical position derived from depth [m].
z	list<float32>	No	Depth value directly, distance along the optical axis [m].
intensity	list<float32>	Yes	Signal amplitude / intensity per point. Sensor-dependent scale.
rgb	list<float32>	Yes	Packed RGB colour value. Encoding convention should be specified per driver.

'rgb' is now the compressed value, but during implementation it can be considered split into 'red', 'green', 'blue' for easy entry and packed under the hood.

---

ToF-specific fields (_TOF_FIELDS)

Field	Arrow type	Nullable	Description
noise	list<float32>	Yes	Per-pixel noise estimate from the ToF sensor. Useful for confidence-weighted fusion.
grayscale	list<float32>	Yes	Grayscale amplitude image co-registered with the depth frame.

---

Stereo-specific fields (_STEREO_FIELDS)

Field	Arrow type	Nullable	Description
luma	list<uint8>	Yes	Luminance (Y channel) of the corresponding pixel in the rectified left image.
cost	list<uint8>	Yes	Stereo matching cost / disparity confidence. Lower values indicate higher confidence.

---

Escape hatch (_EXTRA_FIELD) appended to all schemas

Field	Arrow type	Nullable	Description
extra_attributes	large_binary	Yes	Serialized vendor-specific payload (e.g. confidence maps, temperature, calibration flags). Schema is unspecified at the ontology level.

large_binary is chosen over binary to accommodate vendors that embed large calibration blobs or compressed auxiliary images.

---

DepthCamera

class DepthCamera(BaseModel):
    x: List[float]
    y: List[float]
    z: List[float]
    rgb:              Optional[List[float]] = None
    intensity:        Optional[List[float]] = None
    extra_attributes: Optional[Dict[str, Any]] = None

Holds all fields from _COMMON_FIELD plus extra_attributes. Not instantiated directly.

---

RGBDCamera

class RGBDCamera(DepthCamera, Serializable):
    __msco_pyarrow_struct__ = _build_schema(_COMMON_FIELD)

---

ToFCamera

class ToFCamera(DepthCamera, Serializable):
    __msco_pyarrow_struct__ = _build_schema(_COMMON_FIELD, _TOF_FIELDS)
    noise:     Optional[List[float]] = None
    grayscale: Optional[List[float]] = None

---

StereoCamera

class StereoCamera(DepthCamera, Serializable):
    __msco_pyarrow_struct__ = _build_schema(_COMMON_FIELD, _STEREO_FIELDS)
    luma: Optional[List[int]] = None
    cost: Optional[List[int]] = None

---

Schema Builder

def _build_schema(*field_groups: List[pa.Field]) -> pa.StructType:
    fields = [field for group in field_groups for field in group]
    fields.append(_EXTRA_FIELD)
    return pa.struct(fields)

_build_schema accepts any number of pa.Field lists, flattens them in order, appends _EXTRA_FIELD, and returns a pa.StructType. This is the single point of schema construction, adding a new field group for a new sensor subtype requires only defining a new List[pa.Field] and passing it to _build_schema.

[fdicorato]

These data types have been implemented as 'futures'. Further developments and fixes will be targeted in dedicated issues. Discussion is closed