Calibration

NOTE: Use equidistant distortion model! radtan does not work properly with multicam

Calibration uses multicam_calibration and kalibr.

Before executing these steps, you should collect 2 rosbags with stereo camera and imu data. The first bag is for camera-camera calibration and consists of a static robot and moving calibration board. The second bag is for camera-imu calibration and consists of a static calibration board and moving robot. To see the motions, you can watch this video.

We recommend using rosbags because kalibr requires rosbags. multicam_calibration can use both a live stream or with a rosbag, but we also recommend using a rosbag.

Overview

Calibration consists of three steps:

1. Calibrate the stereo cameras to each other using either multicam_calibration or kalibr
2. Calibrate the cameras with IMU using kalibr
3. Convert the output kalibr yaml into the proper yaml format required by S-MSCKF

You may want to recalibrate the stereo cameras, but not the stereo-imu:

1. Calibrate the stereo cameras to each other using multicam_calibration
2. Update the current kalibr yaml using the output multicam_calibration yaml

Docker

All of the necessary packages and dependencies are located in a docker image

kumarrobotics/autonomy:calibration

You can either download this docker image:

docker pull kumarrobotics/autonomy:calibration

Or you can build it. Assuming the autonomy_stack repo is located in $HOME/autonomy_stack:

cd $HOME/autonomy_stack/calibration/docker
./build.sh

Run the docker image using the runGUI.sh script in the calibration folder. This script will launch a docker container with GUI support (may need to modify if your computer has no GPU). In addition, it will mount as a volume to the container:

1. ${PWD}/calib_files: contains the calibration files
2. ${HOME}/bags: contains the 2 rosbags for both the stereo-cam calibration, and the cam-imu calibration

Assuming the autonomy_stack repo is located in $HOME/autonomy_stack: cd $HOME/autonomy_stack/calibration/docker ./runGUI.sh

You will now be inside the docker container, which is where you will run all the following commands.

Camera-Camera calibration

There are two methods. If you have an initial guess of the camera parameters, you should use multicam_calibration. If you do not have an initial guess of the camera parameters, you should use Kalibr.

Initial guess for OVC3b

cam0:
  rectification_matrix:
  - [ 0.99998513429, -0.00072231833,  0.00540457690] 
  - [ 0.00071000439,  0.99999714874,  0.00228000146] 
  - [-0.00540620838, -0.00227613029,  0.99998279592] 
  projection_matrix:
  - [1016.28949887392,  0.00000000000, 652.56153679049,  0.00000000000] 
  - [ 0.00000000000, 1016.28949887392, 329.65089242489,  0.00000000000] 
  - [ 0.00000000000,  0.00000000000,  1.00000000000,  0.00000000000] 
  camera_model: pinhole
  intrinsics: [1067.96, 1068.1, 641.371, 343.293]
  distortion_model: equidistant
  distortion_coeffs: [-0.0964802, 0.00443556, 0.00765515, -0.0180741]
  resolution: [1280, 800]
  rostopic: /ovc/left/image_raw
cam1:
  T_cn_cnm1:
  - [ 0.99999976816,  0.00010526340, -0.00067275668, -0.11983986808] 
  - [-0.00010219715,  0.99998961548,  0.00455614903,  0.00009585961] 
  - [ 0.00067322929, -0.00455607922,  0.99998939440, -0.00072876351] 
  - [ 0.00000000000,  0.00000000000,  0.00000000000,  1.00000000000] 
  rectification_matrix:
  - [ 0.99998119046, -0.00079988241,  0.00608102979] 
  - [ 0.00081373344,  0.99999707971, -0.00227561086] 
  - [-0.00607919181,  0.00228051640,  0.99997892111] 
  projection_matrix:
  - [1016.28949887392,  0.00000000000, 652.56153679049, -121.79429175304] 
  - [ 0.00000000000, 1016.28949887392, 329.65089242489,  0.00000000000] 
  - [ 0.00000000000,  0.00000000000,  1.00000000000,  0.00000000000] 
  camera_model: pinhole
  intrinsics: [1064.59, 1064.79, 664.657, 350.615]
  distortion_model: equidistant
  distortion_coeffs: [-0.0963886, -0.00104029, 0.0133809, -0.017335]
  resolution: [1280, 800]
  rostopic: /ovc/right/image_raw

Multicam Calibration

The below commands mirror the instructions located in multicam_calibration, with slight modifications to automatically map the correct directories.

Note:

(1) multicam_calibration is using opencv3, if there are any packages that are using another version of opencv, there might be a conflict, please force their usage of opencv3 if possible.

(2) put individual pkgs in the apriltag repo into the src folder of the workspace before building.

Launch the calibration.launch file:

roslaunch calibration_launch calibration.launch

In a separate terminal, play the calibration bag:

rosbag play calibration.bag

When you have enough frames collected, start the calibration:

rosservice call /multicam_calibration/calibration

The calibration output will autonmatically be saved in the calib_files/multicam directory (on the native system).

Note that multicam_calibration requires an initial guess of the camera calibration. If this initial guess is not available, you can instead use Kalibr.

Kalibr camera-camera

The following steps make these assumptions:

1. calibration bag is located at ~/bags/calibration.bag (inside the docker container). This should be mapped correctly using the runGUI.sh script if you placed your rosbag in $HOME/bags directory in your host system
2. camera topics are /emu1/sync/cam0/image_raw and /emu1/sync/cam1/image_raw
3. aprilgrid.yaml is located at ~/calib_files/aprilgrid.yaml (inside the docker container). This should be mapped correctly using the runGUI.sh script

While inside the docker container, move into the calib_files directory (located at ~/calib_files), as Kalibr will save the output files into the current directory. If you are not in the calib_files directory, you will have to manually move the output files into this directory, otherwise they will disappear once you exit the docker container:

cd ~/calib_files

Calibrate the cameras, specifying the correct calibration bag, camera topics, and april tag configuration. This step will take a while (~10-15 minutes):

kalibr_calibrate_cameras --bag cam-cam.bag --topics /ovc/left/image_raw /ovc/right/image_raw --models pinhole-equi pinhole-equi --target aprilgrid.yaml

Camera-IMU calibration

You will use Kalibr to perform the camera-imu calibration.

The following steps make these assumptions:

1. calibration bag is located at ~/bags/calibration.bag (inside the docker container). This should be mapped correctly using the runGUI.sh script if you placed your rosbag in ${HOME}/bags in your host system.
2. output yaml from Camera-Camera calibration (either multicam_calibration or kalibr) is located at ~/calib_files/multicam/emu1-latest.yaml
3. imu parameters are located in `~/calib_files/kalibr/emu1-imu.yaml'

emu1-imu.yaml:

#Accelerometers
accelerometer_noise_density: 1.0e-03   #Noise density (continuous-time)
accelerometer_random_walk:   1.0e-04   #Bias random walk

#Gyroscopes
gyroscope_noise_density:     1.0e-04   #Noise density (continuous-time)
gyroscope_random_walk:       1.0e-05   #Bias random walk

rostopic:                    /emu1/sync/imu/imu      #the IMU ROS topic
update_rate:                 200.0     #Hz (for discretization of the values above)

4. aprilgrid.yaml is located at ~/calib_files/aprilgrid.yaml. This should be mapped correctly using the runGUI.sh script.

If you're using apriltags in the KumarLab, you can use the following aprilgrid.yaml

target_type: 'aprilgrid' #gridtype
tagCols: 7               #number of apriltags
tagRows: 5               #number of apriltags
tagSize: 0.04            #size of apriltag, edge to edge [m]
tagSpacing: 0.25         #ratio of space between tags to tagSize
                         #example: tagSize=2m, spacing=0.5m --> tagSpacing=0.25[-]

While inside the docker container, move into the calib_files directory:

cd ~/calib_files

Calibrate the cameras and IMU, specifying the correct calibration bag, camera topics, and april tag configuration:

kalibr_calibrate_imu_camera --bag cam-imu.bag --cam cam-cam-latest.yaml --imu imu.yaml --target aprilgrid.yaml

Convert kalibr yaml format

Move into the calib_files directory insider docker:

cd ~/calib_files

Run the convert_kalibr script. We assume kalibr.yaml is the kalibr output yaml, and output.yaml is where you want to save the converted yaml format.

./convert_kalibr -k kalibr.yaml -o output.yaml

Update kalibr yaml with new multicam calibration

You typically will have to update the camera-camera calibration more frequently than the camera-imu calibration using multicam-calibration.

Using the output file from the multicam_calibration step, you will thenneed to update the kalibr file to be used by S-MSCKF.

Move into the calib_files directory insider docker:

cd ~/calib_files

Run the update_kalibr script. We assume kalibr.yaml is the original kalibr output yaml, and multicam.yaml is recent multicam calibration yaml, and output.yaml is where you want to save the updated yaml format.

./update_kalibr -k kalibr.yaml -m multicam.yaml -o output.yaml