Tool not being tracked

[SWAGamaz]

Good day!
I need your help and advice. I sort of successfully built a project on OpenCV 4.8.0 for ARM64 and I'm trying to use it in Unity, but my tool is not tracked, the resulting ToolPosition array returns 0000000.

    void Start()
    {
#if ENABLE_WINMD_SUPPORT
        tracking = new HL2ToolTracking();
        tracking.Initialize(SpatialLocator.GetDefault().CreateStationaryFrameOfReferenceAtCurrentLocation().CoordinateSystem);
        tracking.StartTracking();
#endif
    }

    void Update()
    {
#if ENABLE_WINMD_SUPPORT
        string s = "";

        if (tracking != null && tracking.ToolPosition != null)
                foreach (var e in tracking.ToolPosition)
                    s += e + " ";

        text.text = s;
#else
        text.text = "ENABLE_WINMD_SUPPORT is not";
#endif
    }

I was unable to attach infrared LEDs to the glasses in such a way as to get a good reflection from the reflective sphere. I took the opposite approach and mounted LEDs with a diffuser on the tool. From what I can tell by the image, they are very visible.

It seems to me, perhaps, the problem is in the incorrect identification of the tool's points. I took the coordinates from the model in the Unity coordinate system. Should I use a different coordinate system? And how should I input the coordinates so that, for instance, ToolPosition would return the position of the tool tip?

	// definition of the model points
	cv::Mat pointsModel = (cv::Mat_<float>(4, 3) <<
		0.0f, 0.01023106f, -0.05009925f,
		0.0f, 0.0002310641f, 0.0f,
		0.03f, 0.0002310641f, 0.04990065f,
		-0.0299998f, 0.0002310641f, 0.0849006f);

I really hope for your advice!

P.S.
I'm providing a link to the compiled opencv 4.8.0 for ARM64.
I built only those modules that are required by the project's dependencies. During the compilation, following your instructions, I didn't encounter any errors. However, I'm not sure how to verify that everything is working correctly.

Hello,
Could you share what/how you mounted the markers onto your medical device?

Thanks
Saharsh

Did you calibrate the cameras? That could be a reason why the tracking isn't working properly.

[SWAGamaz]

Greetings, @ssbarve2!

For mounting the markers, I used a ready-made 3D model of the tool with designated spots for reflective spheres, which I printed with a 3D printer. In these allocated spots, I installed infrared LEDs with diffusers. I am wondering whether the order of x, y, z coordinates for each point affects the system's ability to accurately identify the tool.

As of now, I haven’t calibrated the cameras and am trying to achieve at least some recognition of the tool, even if the accuracy isn’t high.

May I ask if you have succeeded in tracking your tool effectively? Is there anything else you could recommend or clarify to help me get a more complete picture of the situation?

Thank you in advance for your assistance!

Hello @SWAGamaz,
I am currently in the initial stages of setting up the tool tracker myself, will keep you updated about my progress. We can connect via mail (if that's okay with you) to rectify each others doubts.

Currently, I am working on creating a basic instrument to track. Do you mind sharing your 3d model so that we can alter cross check our results.

Regards,
Saharsh

Also,
Any reason why you have placed markers on the same plane?

Regards,
Saharsh

[cgsaxner]

Hello @SWAGamaz,

there could be a couple of reasons as to why the tool is not detected. Since your camera images are a bit different from mine, maybe the parameters in the BlobFinder need to be adapted.
As @ssbarve2 pointed out, the camera calibration might also be an issue.
You also mentioned that you input the points in the Unity coordinate system. However, the OpenCV coordinate system is used within the Plugin, so you might have to invert the y-axis of your points!

For debugging it can be helpful to write a testing script that allows you to process pre-recorded frames offline. This makes it easier to visualize the intermediate steps.

Hello,
I had a couple of doubts:

1. What is the unit for marker coordinates with respect to the instrument origin(in your case the tip of the needle)?
2. How did you define the X,Y,Z coordinate system around the origin (how are the axes defined).n is X axis the vertical length of the tool, Y along the width and Z the height from base?
3. Can origin be any arbitrary point on the instrument or should it be a tip/corner?
4. In cv::Mat pointsModel which column is for which axis?

	// definition of the model points 
	cv::Mat pointsModel = (cv::Mat_<float>(5, 3) <<
		2.57336e-7f, 2.42391e-6f, 3.36463e-6f,
		0.0442368f, 0.0349529f, 0.0726091f,
		-0.0936104f, 0.0113667f, 0.0732692f,
		0.000176568f, -0.000130322f, 0.108399f,
		-0.00748328f, 0.0856192f, 0.0642646f);

This is how I am considering the instrument coordinate system.

Thanks