[RESEARCH] Image Filter for Repeating Grid Removal

[jorgenfj]

Motivation

Standard ArUco marker detection fails when markers are partially obscured by a repeating grid pattern. This is a common problem in robotic vision, especially when the camera's line of sight is obstructed by structural elements. We need a robust method to preprocess the image to "remove" the grid, making the ArUco markers fully detectable.

The approach:

• Use Fourier image processing to identify and remove the periodic grid pattern from the image.
• Apply perspective warp to the image to magnify the grid's repeating nature, making it easier to filter in the frequency domain.
• After filtering, the markers may appear as fragmented or disconnected regions. We need to implement a routine to merge these regions into a single, coherent marker.
• Finally, apply standard OpenCV ArUco detection to the processed image to locate and identify the markers.

Requirements:

• The final output must be an image where the standard ArUco detection algorithm successfully detects the marker with correct id.

---

Task

1. Implement Smart Filtering:
   • Perspective Warp: Create a function to apply a perspective warp to the image. This will normalize the grid pattern's appearance and enhance its periodicity.
   • Fourier Image Processing: Implement a routine that takes a warped image and applies a notch or band-stop filter in the frequency domain to remove the grid pattern. The output should be a processed image where the grid is suppressed.
2. Aruco Marker Reconstruction:
   • Region Merging: Develop a function that identifies fragmented ArUco marker regions in the filtered image and merges them to form a single, complete marker. This may involve using morphological operations like dilation or connectivity analysis.
3. Integrate with ArUco Detection:
   • Detection Pipeline: Create a pipeline that takes the raw camera feed, passes it through the smart filter and reconstruction steps, and then feeds the final output to a standard ArUco detection algorithm in https://github.com/vortexntnu/vortex-aruco-detection.

---

Suggested Workflow

1. Understand the proposed solution.
2. Research:
   • Perspective warp
   • Fourier filtering
   • Aruco detection pipeline in vortex-aruco-detection

---

Deliverables

• The whole filter can be implemented as a filter in https://github.com/vortexntnu/vortex-image-filtering or as a separate python node if python is the chosen language.
• A module for Fourier-based filtering (specifically, a notch filter) to remove the periodic noise.
• A function to merge fragmented marker regions in the processed image.
• An end-to-end demo/test script that showcases the complete process, from a gridded image to successful ArUco marker detection.

---

References

• [What Is Perspective Warping?](https://pub.towardsai.net/what-is-perspective-warping-opencv-and-python-750e7a13d386)
• [OpenCV Perspective Warping](https://docs.nvidia.com/vpi/algo_persp_warp.html)
• [Notch Filter for Periodic Noise](https://stackoverflow.com/questions/29235421/find-proper-notch-filter-to-remove-pattern-from-image)
• [Removing Periodic Noise with Fourier Transform](https://paulbourke.net/miscellaneous/imagefilter/)
• [Youtube Video about Fourier Transform for Periodic Noise Removal](https://www.youtube.com/watch?v=OOu5KP3Gvx0)

[kluge7]

Another potential solution could be to train a CNN model to directly classify aruco markers that are obscured by a grid. The main issue with this approach is that we dont know which markers will be used (we can assume id between 0 and 100) but it would still require a large dataset for the model to be effective. If we are able to either automate data generation in the simulator for this or find a dataset online for aruco classification (and add a grid to the images), this could be a possibility

[jorgenfj]

Another, Another solution could be to train a model to remove the grid and fill out the gaps of the remaining aruco marker, to allow the openCV algorithm to handle the identification. One challenge with this to produce sharp enough edges and contrasts for the openCV algorithm to work.