MT crash

[stolk]

After the bullet upgrade, I also noticed MT crashes that I did not have before.
It could be my app's miss-use of bullet, but in case it is not, here are the details...

Thread 1 "frag" received signal SIGABRT, Aborted.
─── Assembly ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
0x00007ffff65f0e8b __GI_raise+187 mov    $0x2,%edi
0x00007ffff65f0e90 __GI_raise+192 mov    $0xe,%eax
0x00007ffff65f0e95 __GI_raise+197 syscall 
0x00007ffff65f0e97 __GI_raise+199 mov    0x108(%rsp),%rcx
0x00007ffff65f0e9f __GI_raise+207 xor    %fs:0x28,%rcx
0x00007ffff65f0ea8 __GI_raise+216 mov    %r8d,%eax
0x00007ffff65f0eab __GI_raise+219 jne    0x7ffff65f0ecc <__GI_raise+252>
─── Expressions ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
─── History ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
─── Memory ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
─── Registers ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   rax 0x0000000000000000      rbx 0x0000000000000000      rcx 0x00007ffff65f0e97      rdx 0x0000000000000000      rsi 0x00007fffffffd3e0      rdi 0x0000000000000002      rbp 0x00007ffff67697d8  
   rsp 0x00007fffffffd3e0       r8 0x0000000000000000       r9 0x00007fffffffd3e0      r10 0x0000000000000008      r11 0x0000000000200246      r12 0x000000000069c608      r13 0x000000000069c645  
   r14 0x00000000000000aa      r15 0x0000000000000000      rip 0x00007ffff65f0e97   eflags [ PF ZF IF ID ]          cs 0x00000033               ss 0x0000002b               ds 0x00000000          
    es 0x00000000               fs 0x00000000               gs 0x00000000          
─── Source ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Cannot display "/build/glibc-OTsEL5/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c" ([Errno 2] No such file or directory: '/build/glibc-OTsEL5/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c')
─── Stack ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
[0] from 0x00007ffff65f0e97 in __GI_raise+199 at ../sysdeps/unix/sysv/linux/raise.c:51
arg sig = 6
[1] from 0x00007ffff65f2801 in __GI_abort+321 at abort.c:79
(no arguments)
[+]
─── Threads ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
[5] id 5610 name frag from 0x00007ffff73e86d6 in futex_abstimed_wait_cancelable+40 at ../sysdeps/unix/sysv/linux/futex-internal.h:205
[4] id 5609 name frag from 0x00007ffff73e86d6 in futex_abstimed_wait_cancelable+40 at ../sysdeps/unix/sysv/linux/futex-internal.h:205
[3] id 5608 name frag from 0x00007ffff73e86d6 in futex_abstimed_wait_cancelable+40 at ../sysdeps/unix/sysv/linux/futex-internal.h:205
[2] id 5605 name SDLTimer from 0x00007ffff73e86d6 in futex_abstimed_wait_cancelable+40 at ../sysdeps/unix/sysv/linux/futex-internal.h:205
[1] id 5601 name frag from 0x00007ffff65f0e97 in __GI_raise+199 at ../sysdeps/unix/sysv/linux/raise.c:51
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
>>> where
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff65f2801 in __GI_abort () at abort.c:79
#2  0x00007ffff65e239a in __assert_fail_base (fmt=0x7ffff67697d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x69c645 "n >= 0", file=file@entry=0x69c608 "/home/bram/src/bullet3/src/LinearMath/btAlignedObjectArray.h", line=line@entry=170, function=function@entry=0x69c680 <btAlignedObjectArray<btSimulationIslandManagerMt::Island*>::operator[](int)::__PRETTY_FUNCTION__> "T& btAlignedObjectArray<T>::operator[](int) [with T = btSimulationIslandManagerMt::Island*]") at assert.c:92
#3  0x00007ffff65e2412 in __GI___assert_fail (assertion=0x69c645 "n >= 0", file=0x69c608 "/home/bram/src/bullet3/src/LinearMath/btAlignedObjectArray.h", line=170, function=0x69c680 <btAlignedObjectArray<btSimulationIslandManagerMt::Island*>::operator[](int)::__PRETTY_FUNCTION__> "T& btAlignedObjectArray<T>::operator[](int) [with T = btSimulationIslandManagerMt::Island*]") at assert.c:101
#4  0x0000000000493130 in btAlignedObjectArray<btSimulationIslandManagerMt::Island*>::operator[] (this=0x1856ac0, n=-1) at /home/bram/src/bullet3/src/LinearMath/btAlignedObjectArray.h:170
#5  0x0000000000491385 in btSimulationIslandManagerMt::getIsland (this=0x18569f0, id=-1) at /home/bram/src/bullet3/src/BulletDynamics/Dynamics/btSimulationIslandManagerMt.cpp:174
#6  0x0000000000491f0d in btSimulationIslandManagerMt::addManifoldsToIslands (this=0x18569f0, dispatcher=0x1a55ab0) at /home/bram/src/bullet3/src/BulletDynamics/Dynamics/btSimulationIslandManagerMt.cpp:437
#7  0x00000000004929a3 in btSimulationIslandManagerMt::buildAndProcessIslands (this=0x18569f0, dispatcher=0x1a55ab0, collisionWorld=0x16d1850, constraints=..., solverParams=...) at /home/bram/src/bullet3/src/BulletDynamics/Dynamics/btSimulationIslandManagerMt.cpp:680
#8  0x000000000048fa19 in btDiscreteDynamicsWorldMt::solveConstraints (this=0x16d1850, solverInfo=...) at /home/bram/src/bullet3/src/BulletDynamics/Dynamics/btDiscreteDynamicsWorldMt.cpp:187
#9  0x00000000004a6494 in btDiscreteDynamicsWorld::internalSingleStepSimulation (this=0x16d1850, timeStep=0.00416666688) at /home/bram/src/bullet3/src/BulletDynamics/Dynamics/btDiscreteDynamicsWorld.cpp:480
#10 0x00000000004a62d7 in btDiscreteDynamicsWorld::stepSimulation (this=0x16d1850, timeStep=0.00416666688, maxSubSteps=1, fixedTimeStep=0.00416666688) at /home/bram/src/bullet3/src/BulletDynamics/Dynamics/btDiscreteDynamicsWorld.cpp:434
#11 0x000000000048fe19 in btDiscreteDynamicsWorldMt::stepSimulation (this=0x16d1850, timeStep=0.00416666688, maxSubSteps=0, fixedTimeStep=0.00416666688) at /home/bram/src/bullet3/src/BulletDynamics/Dynamics/btDiscreteDynamicsWorldMt.cpp:256
#12 0x000000000041ebed in wld_bullet_update (dt=0.00416666688) at PI/wld_bullet.cpp:236
#13 0x000000000041c487 in wld_update (dt=0.00416666688) at PI/wld.cpp:148
#14 0x000000000041ba2f in ctrl_simulate () at PI/ctrl_sim.cpp:197
#15 0x00000000004072b5 in iterate () at XWin/main.cpp:431
#16 0x0000000000408562 in main (argc=1, argv=0x7fffffffdd78) at XWin/main.cpp:642
>>> up
#1  0x00007ffff65f2801 in __GI_abort () at abort.c:79
79	abort.c: No such file or directory.
>>> up
#2  0x00007ffff65e239a in __assert_fail_base (fmt=0x7ffff67697d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x69c645 "n >= 0", file=file@entry=0x69c608 "/home/bram/src/bullet3/src/LinearMath/btAlignedObjectArray.h", line=line@entry=170, function=function@entry=0x69c680 <btAlignedObjectArray<btSimulationIslandManagerMt::Island*>::operator[](int)::__PRETTY_FUNCTION__> "T& btAlignedObjectArray<T>::operator[](int) [with T = btSimulationIslandManagerMt::Island*]") at assert.c:92
92	assert.c: No such file or directory.
>>> up
#3  0x00007ffff65e2412 in __GI___assert_fail (assertion=0x69c645 "n >= 0", file=0x69c608 "/home/bram/src/bullet3/src/LinearMath/btAlignedObjectArray.h", line=170, function=0x69c680 <btAlignedObjectArray<btSimulationIslandManagerMt::Island*>::operator[](int)::__PRETTY_FUNCTION__> "T& btAlignedObjectArray<T>::operator[](int) [with T = btSimulationIslandManagerMt::Island*]") at assert.c:101
101	in assert.c
>>> up
#4  0x0000000000493130 in btAlignedObjectArray<btSimulationIslandManagerMt::Island*>::operator[] (this=0x1856ac0, n=-1) at /home/bram/src/bullet3/src/LinearMath/btAlignedObjectArray.h:170
170			btAssert(n >= 0);
>>> print n
$1 = -1
>>> 

Was MT changed in the last few years?

        scheduler = btCreateDefaultTaskScheduler();
        LOGI( "Setting task scheduler." );
        btSetTaskScheduler( scheduler );
...
        dispatcher = new btCollisionDispatcherMt(collisionConfiguration);
...
        // A pool of 4 solvers for MT operation.
        solverpool = new btConstraintSolverPoolMt( 4 );
        world = new btDiscreteDynamicsWorldMt( dispatcher, broadphase, solverpool, solver, collisionConfiguration );

Doesn't happen in single threaded mode. Nor with the old bullet code base.

[erwincoumans]

That's a shame. There are no tests to avoid regression in the multithreaded parts. Possibly some of the recent deformable simulation work caused it. Is there any way to reproduce this in the Example Browser?

[stolk]

Speaking of the Example Browser, all the soft body examples crash on me.
If you build for DEBUG, and start a soft body example:

App_ExampleBrowser: /home/bram/src/bullet3/src/BulletDynamics/ConstraintSolver/btSequentialImpulseConstraintSolver.cpp:739: int btSequentialImpulseConstraintSolver::getOrInitSolverBody(btCollisionObject&, btScalar): Assertion `body.isStaticOrKinematicObject()' failed.
b3Printf: Aborting and deleting SharedMemoryCommon object

It seems kinda similar to my crash. I get manifolds for which one of the collision bodies points to garbage.

[stolk]

I think I solved this.

Does Bullet support collision objects living on the stack?

I was doing a quick proximity test by testing a sphere against the world, with the sphere living on the stack.

What happens, I think, is that manifolds can persist between frames?

So next frame, it can still be pointing to a collision object that is no longer alive?

Two years ago, it happened to work, I guess. But no longer. It is probably a bug in my app for not making it a GHOST OBJECT?

int  wld_bullet_scan_neighbourhood( vec3_t pt, float radius, int cnt, int* walls, bool* hastank )
{
        btSphereShape sph( radius );
        btCollisionObject obj;
        obj.setCollisionShape( &sph );
        obj.setCollisionFlags( btCollisionObject::CF_STATIC_OBJECT );
        obj.setUserPointer( TAG_GRP( 0, GRP_QUER ) );
        btTransform trf;
        trf.setIdentity();
        btVector3 origin( pt[0], pt[1], pt[2] );
        trf.setOrigin( origin );
        obj.setWorldTransform( trf );
        TestCallback tcb( cnt, walls, hastank );
        world->contactTest( &obj, tcb );
        return tcb.cnt;
}

[stolk]

Or maybe I should do world->contactTest() on a btDiscreteDynamicsWorldMt object?

[erwincoumans]

indeed, contact manifolds keep pointer to persistent collision shapes/objects, so you need to keep the lifetime of the collision shapes/objects longer than the manifolds.

[stolk]

I'm still having crashes, even with all object on heap.

What is m_defaultManifoldResult used for?

[stolk]

I'm seeing weird values in there, but I guess that's because it is defined, but not used anywhere in the code?

$ find . -type f -exec grep -H m_defaultManifoldResult {} \;
./BulletCollision/CollisionDispatch/btCollisionDispatcher.h:	btManifoldResult m_defaultManifoldResult;

[erwincoumans]

I've never used or touched any of that multithreaded code. So the only few options: do bisection on git commits to see what commit caused the regression. Or create a small repro case that we can help debugging.

[stolk]

Thanks, I'll attempt that.