SFND 2D Feature Tracking

The idea of the camera course is to build a collision detection system - that's the overall goal for the Final Project. As a preparation for this, you will now build the feature tracking part and test various detector / descriptor combinations to see which ones perform best. This mid-term project consists of four parts:

• First, you will focus on loading images, setting up data structures and putting everything into a ring buffer to optimize memory load.
• Then, you will integrate several keypoint detectors such as HARRIS, FAST, BRISK and SIFT and compare them with regard to number of keypoints and speed.
• In the next part, you will then focus on descriptor extraction and matching using brute force and also the FLANN approach we discussed in the previous lesson.
• In the last part, once the code framework is complete, you will test the various algorithms in different combinations and compare them with regard to some performance measures.

See the classroom instruction and code comments for more details on each of these parts. Once you are finished with this project, the keypoint matching part will be set up and you can proceed to the next lesson, where the focus is on integrating Lidar points and on object detection using deep-learning.

Rubric Fulfillment

• MP.1 Data Buffer Optimization
  • Description: Implement a vector for dataBuffer objects whose size does not exceed a limit (e.g. 2 elements). This can be achieved by pushing in new elements on one end and removing elements on the other end.
  • Implementation: The Ring buffer is implemented in Line 93 in the main file(MidTermProject_Camera_Student.cpp).
• MP.2 Keypoint Detection
  • Description: Implement detectors HARRIS, FAST, BRISK, ORB, AKAZE, and SIFT and make them selectable by setting a string accordingly.
  • Implementation: Each detector above is implemented from Line 92 to Line 192 in the matching_2D_student.cpp file.
• MP.3 Keypoint Removal
  • Description: Remove all keypoints outside of a pre-defined rectangle and only use the keypoints within the rectangle for further processing.
  • Implementation: The process of only keeping keypoints on the preceding vehicle is from Line 112 to Line 122 in the main file.
• MP.4 Keypoint Descriptors
  • Description: Implement descriptors BRIEF, ORB, FREAK, AKAZE and SIFT and make them selectable by setting a string accordingly.
  • Implementation: Each descriptor above is implemented from Line 57 to Line 74 in the matching_2D_student.cpp file.
• MP.5 Descriptor Matching
  • Description: Implement FLANN matching as well as k-nearest neighbor selection. Both methods must be selectable using the respective strings in the main function.
  • Implementation: Matching using FLANN method is from Line 17 to Line 24 in the matching_2D_student.cpp file.
• MP.6 Descriptor Distance Ratio
  • Description: Use the K-Nearest-Neighbor matching to implement the descriptor distance ratio test, which looks at the ratio of best vs. second-best match to decide whether to keep an associated pair of keypoints.
  • Implementation: Using the KNN matching is in Line 164 in the main file.
• MP.7 Performance Evaluation 1
  • Description: Count the number of keypoints on the preceding vehicle for all 10 images and take note of the distribution of their neighborhood size. Do this for all the detectors you have implemented.
  • Implementation: There will be a MP7file in build folder after executing the program. All the data about keypoints will be saved in the file.
• MP.8 Performance Evaluation 2
  • Description: Count the number of matched keypoints for all 10 images using all possible combinations of detectors and descriptors. In the matching step, the BF approach is used with the descriptor distance ratio set to 0.8.
  • Implementation: There will be a MP8file in build folder after executing the program. The number of matched keypoints for all possible combination, except SIFT+ORB and non-AKAZE+AKAZE, will be recorded in the file.
• MP.9 Performance Evaluation 3
  • Description: Log the time it takes for keypoint detection and descriptor extraction. The results must be entered into a spreadsheet and based on this data, the TOP3 detector / descriptor combinations must be recommended as the best choice for our purpose of detecting keypoints on vehicles.
  • Implmentation: There will be a MP9file in build folder after executing the program. The computing time of both keypoint detection and descriptor extraction for all possible combination, except SIFT+ORB and non-AKAZE+non_AKAZE, will be recorded in the file.
  • Top 3 Combinations (Detector + Descriptor)
    1. FAST + ORB
    2. FAST + BRIEF
    3. ORB + BRIEF

Dependencies for Running Locally

1. cmake >= 2.8

• All OSes: click here for installation instructions

2. make >= 4.1 (Linux, Mac), 3.81 (Windows)

• Linux: make is installed by default on most Linux distros
• Mac: install Xcode command line tools to get make
• Windows: Click here for installation instructions

3. OpenCV >= 4.1

• All OSes: refer to the official instructions
• This must be compiled from source using the -D OPENCV_ENABLE_NONFREE=ON cmake flag for testing the SIFT and SURF detectors. If using homebrew: $> brew install --build-from-source opencv will install required dependencies and compile opencv with the opencv_contrib module by default (no need to set -DOPENCV_ENABLE_NONFREE=ON manually).
• The OpenCV 4.1.0 source code can be found here

4. gcc/g++ >= 5.4

• Linux: gcc / g++ is installed by default on most Linux distros
• Mac: same deal as make - install Xcode command line tools
• Windows: recommend using either MinGW-w64 or Microsoft's VCPKG, a C++ package manager. VCPKG maintains its own binary distributions of OpenCV and many other packages. To see what packages are available, type vcpkg search at the command prompt. For example, once you've VCPKG installed, you can install OpenCV 4.1 with the command:

c:\vcpkg> vcpkg install opencv4[nonfree,contrib]:x64-windows

Then, add C:\vcpkg\installed\x64-windows\bin and C:\vcpkg\installed\x64-windows\debug\bin to your user's PATH variable. Also, set the CMake Toolchain File to c:\vcpkg\scripts\buildsystems\vcpkg.cmake.

Basic Build Instructions

1. Clone this repo.
2. Make a build directory in the top level directory: mkdir build && cd build
3. Compile: cmake .. && make
4. Run it: ./2D_feature_tracking.