feat: uniform array-size policy across parser & streaming plugins#136
Open
pabloinigoblasco wants to merge 1 commit into
Open
feat: uniform array-size policy across parser & streaming plugins#136pabloinigoblasco wants to merge 1 commit into
pabloinigoblasco wants to merge 1 commit into
Conversation
Add a shared array-size policy contract (max_array_size + clamp/skip)
adopted by every parser and streaming plugin, so the option carries the
same defaults and JSON keys regardless of which plugin emits the config.
Contract lives in common/array_policy/ as a header-only INTERFACE
library:
common/array_policy/include/pj_array_policy/array_policy.hpp
common/array_policy/CMakeLists.txt (target: pj_array_policy)
common/array_policy/tests/array_policy_test.cpp
Canonical keys read/written on every config:
"max_array_size" : uint32 (0 means unlimited)
"array_policy" : "clamp" | "skip"
Legacy keys ("clamp_large_arrays", "discard_large_arrays") still work as
a fallback, so configs and layouts written by earlier plugin versions
keep loading without change.
Adoption:
parser_json, parser_protobuf, parser_ros — link contract,
new UI (Qt .ui file) exposing the option in the dialog.
data_stream_foxglove_bridge, data_stream_pj_bridge,
data_stream_ros2 (distro) — route the value
through ArrayLimit when forwarding the configuration to embedded
parsers; default is now Clamp/500.
data_load_mcap — drop a dead
array-size widget that is now handled by the embedded parser dialog.
The contract has no SDK dependency (only nlohmann/json) and is consumed
only by plugins in this repo, so it ships here rather than in
plotjuggler_sdk. Third-party plugins remain cross-compatible by honoring
the two JSON keys above; they do not have to link the library.
Contents:
- common/array_policy/{include,tests,CMakeLists.txt}: new shared lib + 8
gtest cases covering canonical keys, legacy fallback, zero-as-
unlimited, JSON round-trip.
- parser_json/json_parser{,_dialog}.{cpp,hpp} + .ui — adopt ArrayLimit,
add UI control with default Clamp/500.
- parser_protobuf/protobuf_parser{,_dialog}.{cpp,hpp} + .ui — same.
- parser_ros/ros_parser{,_dialog}.{cpp,hpp} — same.
- data_stream_foxglove_bridge/{foxglove_dialog.hpp,foxglove_source.cpp,
CMakeLists.txt} — propagate ArrayLimit to the inner parser config.
- data_stream_pj_bridge/{pj_bridge_dialog.hpp,pj_bridge_source.cpp,
CMakeLists.txt} — same.
- data_stream_ros2/distro/src/ros2_dialog.hpp + CMakeLists.txt — same.
- data_load_mcap/mcap_dialog.hpp — remove the now-redundant control.
- Root CMakeLists.txt — add_subdirectory(common/array_policy) in both
standalone and subdir modes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every parser and streaming plugin in this repo gains a uniform "maximum
array size + clamp/skip" option, with the same defaults and JSON keys
regardless of which plugin emits the configuration. Parsers flatten each
array element into its own scalar series (
field[0],field[1], …), soan unbounded array previously meant an unbounded number of series; this
contract caps that, and the new UI lets the user pick the limit per
parser dialog.
Why
Until now each parser/stream re-implemented the option with different
member names, different JSON keys, and different defaults
(
clamp_large_arraysvsdiscard_large_arrays). That drift made layoutsand configs non-portable between plugins and surprised users when the
same control behaved differently. A single contract removes the
divergence and lets every plugin read/write the option the same way.
Where the contract lives
The contract is header-only (struct + two JSON helpers, no SDK
dependency beyond
nlohmann/json) and only plugins in this repo consumeit. It ships as a local INTERFACE library:
Third-party plugins remain cross-compatible by honoring the two JSON
keys documented in the header — they do not have to link the library.
Canonical JSON keys
max_array_sizeuint325000means unlimited.array_policystring"clamp""clamp"or"skip".Legacy keys are read with lower priority as a fallback, so older configs
and layouts still load:
discard_large_arraysarray_policyskipclamp_large_arraysarray_policyclamp/skiparrayLimitToJsonalways writes both the canonical and legacy keys so abinary built before this contract can still honor the policy.
Adoption per plugin
parser_json,parser_protobuf,parser_ros— link the contract,expose the option in their dialog
.ui(default clamp/500).data_stream_foxglove_bridge,data_stream_pj_bridge,data_stream_ros2(distro) — propagate the value throughArrayLimitwhen forwarding the configuration to the embedded parser.
data_load_mcap— drop a now-redundant array-size control; theembedded parser dialog handles it.
Testing
All targets compile clean;
array_policy_testpasses 8/8 (defaults,canonical keys, legacy fallback, zero-as-unlimited, canonical-wins-over-
legacy, round-trip).
Test plan
large array, verify the UI shows the new control and that
changing clamp ↔ skip updates the materialized series accordingly.
plugin version (using
clamp_large_arrays) and verify thepolicy is restored as Clamp.