Commit 2dd0516
authored
feat: Build out Device and Data module
# Summary
This PR adds a centralized `Buffer` object which allows transposition from an Array of Structures (AOS) given by `Vec<Sample>` to a Structure of Array (SOA) given by an `AlignedWindow` struct. In doing so, it exposes three means of reading, two of which have not been tested and are bolded below:
1. Pull-based read - reads the most recent $N$ samples on request
2. **Cursor-based read** (`reader.rs`) - reads the next $N$ samples, keeping track of the location based on an iterator
3. **Push-based read** (`subscription.rs`) - reads the most recent $N$ samples as sent by a `SubscriptionManager`
This also includes a cascade of changes, such as:
- Moving the CLI arg parser from [getopts](https://github.com/rust-lang/getopts) to [clap](https://github.com/clap-rs/clap)
- Moving from [crossterm](https://github.com/crossterm-rs/crossterm) to [ratatui](https://github.com/ratatui/ratatui) for TUI tools (e.g., tio-monitor, the new tio-health)
- Exposing a `DeviceTree` to avoid instantiating multiple `Device`s (using `subtree_full()`)
- Support for `HDF5` export from a binary log file
- Elevating thread priority on macOS and Windows using `objc2` and `MMCSS` respectively (to minimizing dropped `Sample`s)
## New Files
### Device Module
> `tree.rs`, `rpc_registry.rs`, `reader.rs`, `subscription.rs`, `util.rs`
- `tree.rs` copies over the functionality of `device.rs` but adds the `DeviceRoute` as an argument
- `rpc_registry.rs` is for future functionality, as many tools would like to be exposed to the tree hierarchy of the RPC structure without having to reconstruct it themselves through a manual rpc.listinfo and building the entire query
- `util.rs` hosts the RPC parsing that was originally found exclusively in `tio-tool` that can be reused in other programs
### Data Module
> `buffer.rs`, `export.rs`, `filter.rs`, `util.rs`
The main area for future change is the developer API for the `Buffer` object. In supporting multiple use cases (monitoring, real-time signal processing, offline processing), there are possibly redundant structures. Thus far, the API is as follows:
- `OverflowPolicy` which determines whether to act like a ring buffer or, effectively, a batched streaming buffer
- `BufferEvent` structures derived from characteristics of the stream of `Sample`s (e.g. `SamplesSkipped`). It is sent over a crossbeam channel and, for now, overloads the batched streaming from `OverflowPolicy` in the form of a `DataChunk` and has an optional override to (copy) and forward all samples in a `BufferEvent::Samples`. **It should be possible to move away from this `DataChunk` and just use the proper subscription based `push` module.**
- `ReadError` a run-time error which determines if read queries are impossible, such as if the `Cursor` is pointing to a spot out of the `Buffer`, if it requested too many samples, or a column that doesn't exist
The whole idea is that a "run" attempts to denote a maximal contiguous region of samples for a given (route, stream_id) that share:
- `session_id`
- effective sampling rate (sampling_rate / decimation)
- time reference epoch (MetadataEpoch)
and do not exhibit large gaps or backward jumps in sample number / timestamp.
### OS Module
- `mod.rs`
This only has the thread priority functionality. In the future it would be appropriate to host any other OS dependent code here.
# Changelog
## Changes
- Update `tio-monitor` to support RPC execution, basic time-series and periodogram plotting, and multiple devices using [ratatui](https://github.com/ratatui/ratatui)
- Update all existing tools (`tio-proxy`, `tio-tool`, `tio-text-proxy`) to use [clap](https://github.com/clap-rs/clap) parser
- Change `lib.rs` inside `twinleaf-tools` to use [clap](https://github.com/clap-rs/clap) parser
## Added
- Add `buffer.rs` for multi device data transposition and alignment
- Thread priority elevation
- Add `tio-health.rs` for viewing the sample rate and statistics of all incoming samples
- Add `HDF5` support in `tio-tool` using `export.rs` and a flushing `Buffer`
- Add `BufferType` enum inside `proto.rs` to map to typed vector more easily
## Fixed
- Allow negative args in tio-tool rpc, solving #9File tree
35 files changed
+7215
-1030
lines changed- twinleaf-tools
- src
- bin
- twinleaf
- src
- data
- device
- tio
- os
- port
- proto
35 files changed
+7215
-1030
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
0 commit comments