btSubsimplexConvexCast usage

[velkyel]

Bullet-2.83-Alpha:

btCollisionWorld::rayTestInternal():

        btConvexCast* convexCasterPtr = 0;
        //use kF_UseSubSimplexConvexCastRaytest by default
        if (resultCallback.m_flags & btTriangleRaycastCallback::kF_UseGjkConvexCastRaytest)
            convexCasterPtr = &gjkConvexCaster;
        else
            convexCasterPtr = &subSimplexConvexCaster;

        btConvexCast& convexCaster = *convexCasterPtr;

        if (convexCaster.calcTimeOfImpact(rayFromTrans,rayToTrans,colObjWorldTransform,colObjWorldTransform,castResult))
        {
            //add hit
            if (castResult.m_normal.length2() > btScalar(0.0001))
            {
                if (castResult.m_fraction < resultCallback.m_closestHitFraction)
                {
#ifdef USE_SUBSIMPLEX_CONVEX_CAST
                    //rotate normal into worldspace
                    castResult.m_normal = rayFromTrans.getBasis() * castResult.m_normal;
#endif //USE_SUBSIMPLEX_CONVEX_CAST

there is default subSimplexConvexCaster used but USE_SUBSIMPLEX_CONVEX_CAST is not defined so normal is not rotated into worldspace (compare with btTriangleConvexcastCallback::processTriangle). Not sure -- is it bug?

[erwincoumans]

Good point. In most cases the basis in rayFromTrans is identity, so it doesn't matter. In the case of a compound shape, it could be an issue. I need to spend more time to review this. Thanks for bringing this up.