No significant improvement for symforce VS Ceres on SfM

[empty-spacebar]

Bug Discription

I replaced Ceres with symforce as solver on my SfM project, expecting faster runtime. However their performance had nearly no difference, no matter on timecost or reprojection error.

What I attach may help acquainting you with the details:

It shows the result of symforce, a pointcloud, correct and decent, along with some details of my SfM problem.

symforce_log.txt

Log generated by spdLog.

symforce_solver_config.txt

config perameters of symforce solver.

Environment

• OS and version: Ubuntu 20.04
• Python version: 3.8.10
• SymForce Version: 0.9.0

Thanks for your time.

[empty-spacebar]

I updated symforce from 0.9.0 to 0.10.1, then 13% speedup was observed, which is still far away from my anticipation.

By the way, I found Fixed-size optimizer, which is fast, but requires something like, fixed-size camera poses, fixed-size feature correspondences, etc. It seems impossible for SfM problems.

Is there any trick to modify SfM so it could fit fixed-size optimizer?

[zhanghua7099]

Hi! @empty-spacebar

I’m also curious about how SymForce compares with Ceres.
When using Ceres, do you rely on analytical derivatives or mainly on autodiff?
If you’ve tried both, did you find the runtime performance to be similar to SymForce?

Also, in your SFM projects, have you noticed any clear advantages when using SymForce over Ceres?

Thanks in advance for sharing your insights!

[empty-spacebar]

I found that you didn't implement Ceres version of BAL examples, so I implemented one for comparison between Symforce and Ceres.

Here's the results:

For problem-21-11315-pre, aka the smallest BAL example, Ceres(0.437s) is faster than Symforce(0.701s).
for problem-257-65132-pre, aka the largest BAL example, Ceres(77.076s) is slower than Symforce(32.868s).

Then I added loss function, which is necessary in practice, into my BAL programs. Both Ceres version and Symforce got accelerated.
However, Ceres was then faster than Symforce in all cases.

Another weird thing is, timecost of Symforce fructuated intensely, with a range from about 40s to 200s. While Ceres performed nearly the same every time.

My questions:

1. Have you ever try BAL examples with Ceres beforce? Any idea about the slower Symforce performance in my cases?
2. Why did Ceres surpass Symforce in BAL examples with loss function?
3. Why did the performance of Symforce fructuate so intensely?

[empty-spacebar]

Hi! @empty-spacebar

I’m also curious about how SymForce compares with Ceres. When using Ceres, do you rely on analytical derivatives or mainly on autodiff? If you’ve tried both, did you find the runtime performance to be similar to SymForce?

Also, in your SFM projects, have you noticed any clear advantages when using SymForce over Ceres?

Thanks in advance for sharing your insights!

Comparison between Ceres and Symforce remains to be discussed. I would like to hold my answer before getting comment from Aaron.

[empty-spacebar]

@aaron-skydio Hi, could you please check out this issue?

[aaron-skydio]

Generally I don't have time to look into this sort of open-ended question right now, but I can provide some high-level comments:

There are lots of options for both SymForce and Ceres that affect convergence and performance.

Yes, we've run BAL on both Ceres and SymForce. Canonically, the BAL benchmark doesn't use a robust loss, so any benchmarking we did on BAL wouldn't have used one. It's been a while, so I don't remember exactly the results and how they might have changed. You've maybe already looked at this, but see the SymForce reference implementation and the links from there to the Ceres and GTSAM reference implementations. Agree that using a robust loss is important for practical cases, and we use SymForce successfully with robust losses frequently.

SymForce is deterministic. If it produces different results run-to-run on the same optimization problem, that really is a bug, and if you provide a reproducible example (with code, and input data) that's something we can look at. On different optimization problems, the number of iterations may fluctuate, that's just a fact of life with nonlinear optimization. There are things you can do to make this more predictable, not much high-level advice I can give though.

By the way, I found Fixed-size optimizer, which is fast, but requires something like, fixed-size camera poses, fixed-size feature correspondences, etc. It seems impossible for SfM problems.

Is there any trick to modify SfM so it could fit fixed-size optimizer?

For this question specifically - no the fixed-size optimizer isn't something you'd typically use for a problem like BAL, or a similar SfM problem of unknown medium-to-large size. The regular optimizer is the thing to use.