Skip to content

Commit 21f844a

Browse files
PeterJohnsonPeterJohnson
authored
Add documentation of data logs and the data log tool (#1741)
* Add documentation of data logs and the data log tool * Move datalog to separate article * Address review comments, (hopefully) fix some links * Add a couple of notes about USB flash drives * exFAT is lowercase
1 parent 3d76597 commit 21f844a

File tree

7 files changed

+234
-6
lines changed

7 files changed

+234
-6
lines changed

‎source/docs/software/telemetry/datalog.rst

Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
On-Robot Telemetry Recording Into Data Logs
2+
===========================================
3+
4+
By default, no telemetry data is recorded (saved) on the robot. The ``DataLogManager`` class provides a convenient wrapper around the lower-level ``DataLog`` class for on-robot recording of telemetry data into data logs. The WPILib data logs are binary for size and speed reasons. In general, the data log facilities provided by WPILib have minimal overhead to robot code, as all file I/O is performed on a separate thread--the log operation consists of mainly a mutex acquisition and copying the data.
5+
6+
Structure of Data Logs
7+
----------------------
8+
9+
Similar to NetworkTables, data logs have the concept of entries with string identifiers (keys) with a specified data type. Unlike NetworkTables, the data type cannot be changed after the entry is created, and entries also have metadata--an arbitrary (but typically JSON) string that can be used to convey additional information about the entry such as the data source or data schema. Also unlike NetworkTables, data log operation is unidirectional--the ``DataLog`` class can only write data logs (it does not support read-back of written values) and the ``DataLogReader`` class can only read data logs (it does not support changing values in the data log).
10+
11+
Data logs consist of a series of timestamped records. Control records allow starting, finishing, or changing the metadata of entries, and data records record data value changes.
12+
13+
.. note: For more information on the details of the data log file format, see the `WPILib Data Log File Format Specification <https://github.com/wpilibsuite/allwpilib/blob/main/wpiutil/doc/datalog.adoc>`__.
14+
15+
Standard Data Logging using DataLogManager
16+
------------------------------------------
17+
18+
The ``DataLogManager`` class (`Java <https://first.wpi.edu/wpilib/allwpilib/docs/release/java/edu/wpi/first/wpilibj/DataLogManager.html>`__, `C++ <https://first.wpi.edu/wpilib/allwpilib/docs/release/cpp/classfrc_1_1_data_log_manager.html>`__) provides a centralized data log that provides automatic data log file management. It automatically cleans up old files when disk space is low and renames the file based either on current date/time or (if available) competition match number. The deta file will be saved to a USB flash drive if one is attached, or to ``/home/lvuser`` otherwise.
19+
20+
.. note: USB flash drives need to be formatted as FAT32 to work with the roboRIO. NTFS or exFAT formatted drives will not work.
21+
22+
Log files are initially named ``FRC_TBD_{random}.wpilog`` until the DS connects. After the DS connects, the log file is renamed to ``FRC_yyyyMMdd_HHmmss.wpilog`` (where the date/time is UTC). If the FMS is connected and provides a match number, the log file is renamed to ``FRC_yyyyMMdd_HHmmss_{event}_{match}.wpilog``.
23+
24+
On startup, all existing log files where a DS has not been connected will be deleted. If there is less than 50 MB of free space on the target storage, ``FRC_`` log files are deleted (oldest to newest) until there is 50 MB free OR there are 10 files remaining.
25+
26+
The most basic usage of DataLogManager only requires a single line of code (typically this would be called from ``robotInit``). This will record all NetworkTables changes to the data log.
27+
28+
.. tabs::
29+
30+
.. code-tab:: java
31+
32+
import edu.wpi.first.wpilibj.DataLogManager;
33+
34+
// Starts recording to data log
35+
DataLogManager.start();
36+
37+
.. code-tab:: c++
38+
39+
#include "frc/DataLogManager.h"
40+
41+
// Starts recording to data log
42+
frc::DataLogManager::Start();
43+
44+
DataLogManager provides a convenience function (``DataLogManager.log()``) for logging of text messages to the ``messages`` entry in the data log. The message is also printed to standard output, so this can be a replacement for ``System.out.println()``.
45+
46+
DataLogManager also records the current roboRIO system time (in UTC) to the data log every ~5 seconds to the ``systemTime`` entry in the data log. This can be used to (roughly) synchronize the data log with other records such as DS logs or match video.
47+
48+
For custom logging, the managed ``DataLog`` can be accessed via ``DataLogManager.getLog()``.
49+
50+
Logging Joystick Data
51+
^^^^^^^^^^^^^^^^^^^^^
52+
53+
DataLogManager by default does not record joystick data. The ``DriverStation`` class provides support for logging of DS control and joystick data via the ``startDataLog()`` function:
54+
55+
.. tabs::
56+
57+
.. code-tab:: java
58+
59+
import edu.wpi.first.wpilibj.DataLogManager;
60+
import edu.wpi.first.wpilibj.DriverStation;
61+
62+
// Starts recording to data log
63+
DataLogManager.start();
64+
65+
// Record both DS control and joystick data
66+
DriverStation.startDataLog(DataLogManager.getLog());
67+
68+
// (alternatively) Record only DS control data
69+
DriverStation.startDataLog(DataLogManager.getLog(), false);
70+
71+
.. code-tab:: c++
72+
73+
#include "frc/DataLogManager.h"
74+
#include "frc/DriverStation.h"
75+
76+
// Starts recording to data log
77+
frc::DataLogManager::Start();
78+
79+
// Record both DS control and joystick data
80+
DriverStation::StartDataLog(DataLogManager::GetLog());
81+
82+
// (alternatively) Record only DS control data
83+
DriverStation::StartDataLog(DataLogManager::GetLog(), false);
84+
85+
Custom Data Logging using DataLog
86+
---------------------------------
87+
88+
The ``DataLog`` class (`Java <https://first.wpi.edu/wpilib/allwpilib/docs/release/java/edu/wpi/first/util/datalog/DataLog.html>`__, `C++ <https://first.wpi.edu/wpilib/allwpilib/docs/release/cpp/classwpi_1_1log_1_1_data_log.html>`__) and its associated LogEntry classes (e.g. ``BooleanLogEntry``, ``DoubleLogEntry``, etc) provides low-level access for writing data logs.
89+
90+
.. note: Unlike NetworkTables, there is no change checking performed. **Every** call to a ``LogEntry.append()`` function will result in a record being written to the data log. Checking for changes and only appending to the log when necessary is the responsibility of the caller.
91+
92+
The LogEntry classes can be used in conjunction with DataLogManager to record values only to a data log and not to NetworkTables:
93+
94+
.. tabs::
95+
96+
.. code-tab:: java
97+
98+
import edu.wpi.first.util.datalog.BooleanLogEntry;
99+
import edu.wpi.first.util.datalog.DataLog;
100+
import edu.wpi.first.util.datalog.DoubleLogEntry;
101+
import edu.wpi.first.util.datalog.StringLogEntry;
102+
import edu.wpi.first.wpilibj.DataLogManager;
103+
104+
BooleanLogEntry myBooleanLog;
105+
DoubleLogEntry myDoubleLog;
106+
StringLogEntry myStringLog;
107+
108+
public void robotInit() {
109+
// Starts recording to data log
110+
DataLogManager.start();
111+
112+
// Set up custom log entries
113+
DataLog log = DataLogManager.getLog();
114+
myBooleanLog = new BooleanLogEntry(log, "/my/boolean");
115+
myDoubleLog = new DoubleLogEntry(log, "/my/double");
116+
myStringLog = new StringLogEntry(log, "/my/string");
117+
}
118+
119+
public void teleopPeriodic() {
120+
if (...) {
121+
// Only log when necessary
122+
myBooleanLog.append(true);
123+
myDoubleLog.append(3.5);
124+
myStringLog.append("wow!");
125+
}
126+
}
127+
128+
.. code-tab:: c++
129+
130+
#include "frc/DataLogManager.h"
131+
#include "wpi/DataLog.h"
132+
133+
wpi::log::BooleanLogEntry myBooleanLog;
134+
wpi::log::DoubleLogEntry myDoubleLog;
135+
wpi::log::StringLogEntry myStringLog;
136+
137+
void RobotInit() {
138+
// Starts recording to data log
139+
frc::DataLogManager::Start();
140+
141+
// Set up custom log entries
142+
wpi::log::DataLog& log = DataLogManager::GetLog();
143+
myBooleanLog = wpi::Log::BooleanLogEntry(log, "/my/boolean");
144+
myDoubleLog = wpi::log::DoubleLogEntry(log, "/my/double");
145+
myStringLog = wpi::log::StringLogEntry(log, "/my/string");
146+
}
147+
148+
void TeleopPeriodic() {
149+
if (...) {
150+
// Only log when necessary
151+
myBooleanLog.Append(true);
152+
myDoubleLog.Append(3.5);
153+
myStringLog.Append("wow!");
154+
}
155+
}
156+
157+
Downloading Data Logs from the Robot
158+
------------------------------------
159+
160+
If data log files are being stored to the roboRIO integrated flash memory instead of a removable USB flash drive, it's important to periodically download and delete data logs to avoid the storage from filling up.
161+
162+
To facilitate this, the DataLogTool desktop application integrates a SFTP client for downloading data log files from a network device (e.g. roboRIO or coprocessor) to the local computer.
163+
164+
This process consists of four steps:
165+
166+
1. Connect to roboRIO or coprocessor
167+
2. Navigate to remote directory and select what files to download
168+
3. Select download folder
169+
4. Download files and optionally delete remote files after downloading
170+
171+
Connecting to RoboRIO
172+
^^^^^^^^^^^^^^^^^^^^^
173+
174+
.. note:: The downloader uses SSH, so it will not be able to connect wirelessly if the radio firewall is enabled (e.g. when the robot is on the competition field).
175+
176+
.. image:: images/datalogtool/download-connecting.png
177+
:alt: Connection display showing team number, username, and password fields.
178+
179+
Either a team number, IP address, or hostname can be entered into the :guilabel:`Team Number / Address` field. This field specifies the remote host to connect to. If a team number is entered, ``roborio-TEAM-frc.local`` is used as the connection address.
180+
181+
The remote username and password are also entered here. For the roboRIO, the username should be ``lvuser`` with a blank password.
182+
183+
The tool also supports connecting to network devices other than the roboRIO, such as coprocessors, as long as the device supports SFTP password-based authentication.
184+
185+
Click :guilabel:`Connect` to connect to the remote device. This will attempt to connect to the device. The connection attempt can be aborted at any time by clicking :guilabel:`Disconnect`. If the application is unable to connect to the remote device, an error will be displayed above the :guilabel:`Team Number / Address` field and a new connection can be attempted.
186+
187+
Downloading Files
188+
^^^^^^^^^^^^^^^^^
189+
190+
After the connection is successfully established, a simplified file browser will be displayed. This is used to navigate the remote filesystem and select which files to download. The first text box shows the current directory. A specific directory can be navigated to by typing it in this text box and pressing Enter. Alternatively, directory navigation can be performed by clicking on one of the directories that are listed below the remote dir textbox. Following the list of directories is a table of files. Only files with a ``.wpilog`` extension are shown, so the table will be empty if there are no log files in the current directory. The checkbox next to each data log file indicates whether the file should be downloaded.
191+
192+
.. note: On the roboRIO, log files are typically saved to either ``/home/lvuser`` or ``/u`` (USB stick location).
193+
194+
.. image:: images/datalogtool/download-file-selection.png
195+
:alt: Remote file browser showing remote directory, list of directories, and list of files with checkboxes next to each one.
196+
197+
Click :guilabel:`Select Download Folder...` to bring up a file browser for the local computer.
198+
199+
If you want to delete the files from the remote device after they are downloaded, check the :guilabel:`Delete after download` checkbox.
200+
201+
Once a download folder is selected, a :guilabel:`Download` button will appear. After clicking this button, the display will change to a download progress display. Any errors will be shown next to each file. Click :guilabel:`Download complete!` to return to the file browser.
202+
203+
.. image:: images/datalogtool/download-downloading.png
204+
:alt: Download status showing 100% completion on the first file and a "file exists" error on the second file.
205+
206+
Converting Data Logs to CSV
207+
---------------------------
208+
209+
As data logs are binary files, the DataLogTool desktop application provides functionality to convert data logs into CSV files for further processing or analysis. Multiple data logs may be simultaneously loaded into the tool for batch processing, and partial data exports can be performed by selecting only the data that is desired to be output.
210+
211+
.. image:: images/datalogtool/download-connecting.png
212+
:alt: DataLogTool window showing two open files, a tree view of entries with corresponding checkboxes and information about each entry, and an output window with export style option.
213+
214+
The conversion process is started by opening data log files in the "Input Files" window. Files are opened by clicking the "Open File(s)..." button. Summary status on each file (e.g. number of records and entries) is displayed. The "X" button in the table row closes the file.
215+
216+
After at least one file is loaded, the "Entries" window displays a tree view of the entries (this can be changed to a flat view by right clicking on the "Entries" window title bar and unchecking "Tree View"). Individual entries or entire subtrees can be checked or unchecked to indicate whether they should be included in the export. The data type information and initial metadata for each entry is also shown in the table. As the "Entries" view shows a merged view of all entries across all input files, if more than one input file is open, hovering over an entry's name will highlight what input files contain that entry.
217+
218+
The output window is used to specify the output folder (via the "Select Output Folder..." button) as well as the output style (list or table). The list output style outputs a CSV file with 3 columns (timestamp, entry name, and value) and a row for every value change (for every exported entry). The table output style outputs a CSV file with a timestamp column and a column for every exported entry; a row is output for every value change (for every exported entry), but the value is placed in the correct column for that entry. Clicking "Export CSV" will create a ``.csv`` file in the output folder corresponding to each input file.
219+
220+
Custom Processing of Data Logs
221+
------------------------------
222+
223+
For more advanced processing of data logs (e.g. for processing of binary values that can't be converted to CSV), WPILib provides a ``DataLogReader`` class for reading data logs in `Java <https://first.wpi.edu/wpilib/allwpilib/docs/release/java/edu/wpi/first/util/datalog/DataLogReader.html>`__, `C++ <https://first.wpi.edu/wpilib/allwpilib/docs/release/cpp/classwpi_1_1log_1_1_data_log_reader.html>`__, or `Python <https://github.com/wpilibsuite/allwpilib/blob/main/wpiutil/examples/printlog/datalog.py>`__. For other languages, the `data log format <https://github.com/wpilibsuite/allwpilib/blob/main/wpiutil/doc/datalog.adoc>`__ is also documented.
224+
225+
DataLogReader provides a low-level view of a data log, in that it supports iterating over a data log's control and data records and decoding of common data types, but does not provide any higher level abstractions such as a NetworkTables-like map of entries. The printlog example in `Java <https://github.com/wpilibsuite/allwpilib/blob/main/wpiutil/src/printlog/java/printlog/PrintLog.java>`__ and `C++ <https://github.com/wpilibsuite/allwpilib/blob/main/wpiutil/examples/printlog/printlog.cpp>`__ (and the Python ``datalog.py``) demonstrates basic usage.

‎source/docs/software/telemetry/images/datalogtool/csv-export.png

87.8 KB
Loading

‎source/docs/software/telemetry/images/datalogtool/download-connecting.png

9.52 KB
Loading

‎source/docs/software/telemetry/images/datalogtool/download-downloading.png

22 KB
Loading

‎source/docs/software/telemetry/images/datalogtool/download-file-selection.png

32.8 KB
Loading

‎source/docs/software/telemetry/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,5 +6,6 @@ Telemetry
66

77
telemetry.rst
88
robot-telemetry-with-sendable.rst
9+
datalog.rst
910
writing-sendable-classes.rst
1011
3rd-party-libraries.rst

‎source/docs/software/telemetry/telemetry.rst

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,16 @@
1-
Telemetry: Recording Data on the Robot
2-
======================================
1+
Telemetry: Recording and Sending Real-Time Data
2+
===============================================
33

44
`Telemetry <https://en.wikipedia.org/wiki/Telemetry>`__ (literally "measuring remotely") is the process of recording and sending real-time data about the performance of your robot to a real-time readout or log file. Recording and viewing telemetry data is a crucial part of the engineering process - accurate telemetry data helps you tune your robot to perform optimally, and is indispensable for debugging your robot when it fails to perform as expected.
55

6-
Recording Data from Robot Code
7-
------------------------------
6+
By default, no telemetry data is recorded (saved) on the robot. However, recording data on the robot can provide benefits over recording on a dashboard, namely that more data can be recorded (there are no bandwidth limitations), and all the recorded data can be very accurately timestamped. WPILib has integrated support for on-robot recording of telemetry data via the ``DataLogManager`` and ``DataLog`` classes and provides a tool for downloading data log files and converting them to CSV.
7+
8+
.. note:: In addition to on-robot recording of telemetry data, teams can record their telemetry data on their driver station computer with :ref:`Shuffleboard recordings <docs/software/dashboards/shuffleboard/getting-started/shuffleboard-recording:Recording and Playback>`.
89

9-
.. note:: On-robot logging of telemetry data is not yet fully supported by WPILib (it is currently scheduled for a mid-season update). In the mean-time, teams can record their telemetry data on their driver station computer with :ref:`Shuffleboard recordings <docs/software/dashboards/shuffleboard/getting-started/shuffleboard-recording:Recording and Playback>`.
10+
Adding Telemetry to Robot Code
11+
------------------------------
1012

11-
WPILib supports several different ways to record and report telemetry data on the driver station.
13+
WPILib supports several different ways to record and send telemetry data from robot code.
1214

1315
At the most basic level, the :ref:`Riolog <docs/software/vscode-overview/viewing-console-output:Viewing Console Output>` provides support for viewing print statements from robot code. This is useful for on-the-fly debugging of problematic code, but does not scale as console interfaces are not suitable for rich data streams.
1416

0 commit comments

Comments
 (0)