Skip to content

PARALLEL_OPTIMIZATION.md

Latest commit

 

History

History
91 lines (65 loc) · 3.09 KB

PARALLEL_OPTIMIZATION.md

File metadata and controls

91 lines (65 loc) · 3.09 KB

Parallel Optimization Implementation

Summary

Successfully implemented multiprocessing support for the SkeletonOptimizer to speed up skeleton optimization when processing multiple polylines.

Implementation Details

Changes Made

  1. Added n_jobs parameter to SkeletonOptimizerOptions:

    • n_jobs=1: Sequential processing (default, backward compatible)
    • n_jobs=N: Use N parallel workers
    • n_jobs=-1: Use all available CPU cores

  2. Created parallel processing infrastructure:

    • Added _optimize_parallel() method using ProcessPoolExecutor
    • Created module-level _optimize_polyline_worker() function for pickling
    • Maintained backward compatibility with _optimize_sequential() method

  3. Automatic fallback:

    • Falls back to sequential processing for single polyline
    • Gracefully handles worker failures

Performance Results

Benchmark results on a workload with 40 polylines and 50 iterations:

Configuration Time Speedup
Sequential (baseline) 39.1s 1.00x
Parallel (2 workers) 10.6s 3.70x
Parallel (4 workers) 7.0s 5.60x
Parallel (all cores) 7.1s 5.49x

Key Findings

  • Best speedup: ~5.6x with 4 workers on 40 polylines
  • Diminishing returns: Beyond 4 workers, overhead starts to dominate
  • Overhead: Small workloads (<10 polylines) may be slower due to multiprocessing overhead
  • Sweet spot: 20+ polylines with 4 workers provides excellent speedup

Usage Example

from mcf2swc import SkeletonOptimizer, SkeletonOptimizerOptions

# Sequential (default)
opts = SkeletonOptimizerOptions(max_iterations=50)
optimizer = SkeletonOptimizer(skeleton, mesh, opts)
result = optimizer.optimize()

# Parallel with 4 workers
opts = SkeletonOptimizerOptions(max_iterations=50, n_jobs=4)
optimizer = SkeletonOptimizer(skeleton, mesh, opts)
result = optimizer.optimize()

# Parallel with all available cores
opts = SkeletonOptimizerOptions(max_iterations=50, n_jobs=-1)
optimizer = SkeletonOptimizer(skeleton, mesh, opts)
result = optimizer.optimize()

Testing

  • All original tests pass (18 tests)
  • Added 5 new parallel-specific tests
  • Created comprehensive benchmark script

Future Opportunities

Other optimization candidates identified but not yet implemented:

  1. RadiusOptimizer gradient computation (#2 priority)

    • Parallelize finite difference gradient calculation
    • Expected speedup: 4-8x for models with many nodes

  2. LocalRadiusOptimizer segment loop (#3 priority)

    • Parallelize segment-by-segment optimization
    • Requires Jacobi-style updates to avoid race conditions
    • Expected speedup: 2-8x depending on number of segments

  3. Surface area/volume computation (low priority)

    • Parallelize edge loops in compute_swc_surface_area() and compute_swc_volume()
    • Only worthwhile for very large skeletons (>1000 edges)

Notes

  • Multiprocessing overhead is significant on Windows due to process spawning
  • Performance scales well up to 4-8 workers depending on workload
  • The implementation maintains full backward compatibility