Fix camera jitter when looking straight up or down

[slimbuck]

Summary

• Fix degenerate orbit animation in createRotateTrack when the camera-to-target offset is aligned with the Y axis (rotating a Y-aligned vector around Y is a no-op, producing identical keyframes that cause spline ringing)
• Fix numerical instability in vecToAngles near the poles where near-zero horizontal components cause atan2 to produce wildly oscillating yaw values

Details

createRotateTrack: Decompose the camera offset into vertical and horizontal components instead of rotating the full 3D vector with a matrix. When the horizontal radius is near zero (looking straight up/down), a minimum orbit radius of 30% of the total distance is used so the camera actually traces a visible circle. For non-degenerate cases the result is mathematically identical.

vecToAngles: When the squared horizontal length is below 1e-8, yaw defaults to 0 instead of relying on atan2 with near-zero arguments. Also clamps the asin input to [-1, 1] to guard against floating-point edge cases.

Test plan

• Load a scene with the camera looking straight down the Y axis (position and target aligned in X/Z) and verify the rotation animation orbits smoothly
• Verify normal (non-vertical) camera setups still produce the same orbit behavior
• Test camera transitions and orbit interactions near vertical orientations for stable yaw

[Copilot]

Pull request overview

This PR addresses camera jitter/artifacts near vertical orientations by stabilizing angle extraction at the poles and by generating a non-degenerate orbit path when the camera offset is (nearly) Y-aligned.

Changes:

• Update vecToAngles to clamp asin input and avoid unstable yaw near the poles by falling back to 0 when horizontal magnitude is tiny.
• Rework createRotateTrack to build an orbit from horizontal/vertical decomposition (instead of rotating the full offset vector), with a minimum orbit radius for near-vertical setups.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/core/math.ts	Adds clamping and a pole-case fallback in vecToAngles to prevent yaw instability.
src/animation/create-rotate-track.ts	Replaces matrix-rotation orbit generation with an angle/radius approach and introduces a minimum orbit radius for near-vertical offsets.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.