Adjust probe-tracker offset on a per-object basis

[nielsleadholm]

Background

In the final version of the data collection pipeline, a rigid C-clamp was used to attach the Vive tracker puck to the ultrasound probe. The distance between the 3D center of the tracker puck and the tip of the ultrasound probe was then measured as accurately as possible using a manual caliper.

To the greatest extent possible, the C-clamp was then always attached at the same point on the ultrasound probe. However, the short battery life of the probe necessitated removing the C-clamp between object scans in order to recharge the device.

As such, there are two potential sources of error in the estimated distance between the tracker puck center and the probe tip (let us call this the tracker-probe-displacement). These are:

• Human error in manually measuring the 3D offset between the tracker-probe-displacement using a physical caliper.
• Human error in repositioning the C-clamp at the same position in between scans.

The estimated tracker-probe-displacement is defined in the code in the get_state method of ProbeTriggeredUltrasoundEnvironment, as well as load_next_data_point in JSONDatasetUltrasoundEnvironment, and is currently set to [0, 0.028, 0.105], where the unit is in meters.

Task Description

To address this, the following changes could be made:

1. Rather than having a single estimate for the tracker-probe-displacement, enable a different tracker-probe-displacement to be associated with each object.
2. Following (1), try to determine a more accurate estimate of the tracker-probe-displacement; this could be achieved by comparing the dense scan to the 3D model in TBP Robot Lab, and adjusting the values to better align these:

• For example, in the tuna can example below, the tuna can points appear misaligned, and the can is potentially abnormally "thin", suggesting the single value currently set in tbp.ultrasound_perception does not accurately match the physical displacement that was present at the time of the scan.
• Note this calibration should only be done for the dense scan of an object, since this represents training-type data; whatever calibrated value is arrived at should be used as is (without further adjustment) for the sparse, inference-time dataset of that object. Following this will ensure there is no leakage of this calibration into the main test set.
• Also note that, parallel to this issue, some of the 3D models in TBP Robot Lab themselves need appropriate re-scaling. Re-scaling the 3D models should ideally be done before tackling the issue of improving the tracker-probe-displacement estimate. See the associated Issue for this problem here (TODO link).

Outcome

Resolving the above would likely result in a significant boost in performance, particularly for the sim-to-real transfer task.

Figure: The Tuna Can as a Failure Case

The learned model of the tuna can from the dense scan. Note the offset points corresponding to the side of the can's wall, as well as the mismatch in the thickness vs. cross-section size of the tuna can, where the scanned model appears abnormally thin. Also note that the Tuna Can shows examples of anomalous points (see separate Issue here - TODO link).

Figure: Intuition for the Physical Displacement

To provide an intuition for the 3D displacement in y and z (displacement in x in principle should have been zero), markers for the approximate tracker puck and probe tip are added to the above images; note however that these images should not be used for improving the calibration - instead, we want to rely on the location information provided in the collected data points.