Matrix4x4.CreateLookAt gives incorrect result in both LH & RH orientations.

[Gavin-Williams]

If I feed the following arguments to CreateLookAt ...

// a camera looking from the origin to the +z-axis
eye = 0,0,0
forward = 0,0,1
up = 0,1,0

result = Matrix4x4.CreateLookAt(eye, eye + forward, up);

I get

-1  0  0  0 
 0  1  0  0
 0  0 -1  0
 0  0  0  1

This is incorrect in both LH and RH systems.

The LH matrix should be

 1  0  0  0
 0  1  0  0
 0  0  1  0
 0  0  0  1

and the RH matrix should be

-1  0  0  0
 0  1  0  0
 0  0  1  0
 0  0  0  1

My understanding of the view matrix is as follows...
(r = right, u = up, f = forward, p = position)

r.x  u.x  f.x  p.x
r.y  u.y  f.y  p.y
r.z  u.z  f.z  p.z
0    0    0    1

From https://www.3dgep.com/understanding-the-view-matrix/

It's easy to see that the CreateLookAt function fails in both LH & RH systems because it inverts the forward <0,0,1> vector that I passed in to a <0,0,-1> vector.

What's going on?

[Gavin-Williams]

In #34859 it is stated...

They are functionally similar but with f being inversed. That is:

in left-handed you do target - position
in right-handed you do position - target

I don't understand that, forward is always target - position in any orientation. By definition. It's easy to see...

LH
position = <0,0,0> target = <0,0,1> forward = <0,0,1>

RH
position = <0,0,0> target = <0,0,1> forward = <0,0,1>

There is nothing different about them.

[tannergooding]

LookAtLH(eye, at, up) is functionally LookToLH(eye, at - eye, up);
LookAtRH(eye, at, up) is functionally LookToLH(eye, eye - at, up);

So the difference between LH and RH is simply the sign of the "direction" you're looking. This is true in DirectX Math: https://github.com/microsoft/DirectXMath/blob/main/Inc/DirectXMathMatrix.inl#L2181 and GLM: https://github.com/g-truc/glm/blob/b3f87720261d623986f164b2a7f6a0a938430271/glm/ext/matrix_transform.inl#L99

---

You'll note that for LH both DXM and GLM do f = normalize(at - eye), then s = normalize(cross(up, f)), then u = cross(f, s) (only differing in variable names).

Now one computes dot(..., -eye) and the other -dot(..., eye). However, since the algorithm is (lhs.X * rhs.X) + (lhs.Y * rhs.Y) + ... we know that these are equivalent.

The resulting matrix for both (aside from being transposed due to row-major vs column-major) is then:

X = s.X, s.Y, s.Z, dot(s, -eye)
Y = u.X, u.Y, u.Z, dot(u, -eye)
Z = f.X, f.Y, f.Z, dot(f, -eye)
W = 0,   0,   0,   1

---

Now if we take that logic and compare the two RH implementations, we see that DXM simply inverses the subtraction (doing eye - at instead).

GLM on the other hand keeps at - eye and then inverses the cross products instead. However, since we know cross is anticommutative and therefore (a * b) == -(b * a) and that (x - y) == -(y - x), we know that these are equivalent.

[tannergooding]

It does look like the Matrix4x4.CreateLookAt implementation might have a bug in that it is doing eye - at (RH) but then the cross and other logic is LH.

These types were largely supposed to mirror functionality from XNA, so I'd need to go back and double check that implementation to see if the reason for this difference.