Mujoco mjv_movePerturb: how does it work? There is no detailed documentation

[jacopotdsc]

Intro

Hi!

I am a graduate student and I use MuJoCo for a project

My setup

C++ with mujoco 3-3-4, ubuntu 22.04

My question

I have to implement drag feature in a Mujoco simulator with mouse but I am struggling to do it: the aim to achieve is to move an object as the mouse move. Following I will put some code.
When i click on an object and I move the mouse, the object start move but not as the mouse. I also tried to use the following function to compute the delta using a function to compute the delta in 3D coordinate.
My question is: where am I wrong? is projectClick function working well? how should I compute the delta? Furthemore, in what reference system the delta's values to given to mjv_movePerturb? I tried different ways but none of them are working.
I want to understand how compute reldx and reldx. I tried with delta of the mouse, delta with projectClick, projectClick and intersectRayAndPlane but I can't compute the right delta to have the effect to drag an object as the mouse

Minimal model and/or code that explain my question

void SimulationViewport::mousePressEvent(QMouseEvent* event) {
    lastMousePosition = event->position();

    dragSelection = getClickedBodyIdFromMouse(event);
    if ( dragSelection > 0 ){

        perturb->active = event->modifiers() == Qt::ShiftModifier ? mjPERT_ROTATE : mjPERT_TRANSLATE;  
        perturb->select = dragSelection;
        mjv_initPerturb(model, data, scene, perturb);
        mouseAction = mjMOUSE_SELECT;
    }
    else if (event->button() == Qt::LeftButton) {
        if (event->modifiers() & Qt::ShiftModifier) {
            mouseAction = mjMOUSE_ROTATE_V;
        } else {
            mouseAction = mjMOUSE_MOVE_H;
        }
    }
}

void SimulationViewport::mouseMoveEvent(QMouseEvent* event) {
    if (mouseAction == mjMOUSE_NONE)
        return;
    
    if(mouseAction == mjMOUSE_SELECT){

        double scale = 1;

        // Delta mouse in pixel (posizione attuale - precedente)
        QPointF deltaMouse = (event->position() - lastMousePosition) * devicePixelRatioF();

        mjtNum deltaX = deltaMouse.x();
        mjtNum deltaY = -deltaMouse.y();


        mjtMouse perturbationType = event->modifiers() == Qt::ShiftModifier ? mjMOUSE_ROTATE_V : mjMOUSE_MOVE_H;
        mjv_movePerturb(model, data, perturbationType, deltaX, deltaY, scene, perturb);
        mjv_applyPerturbPose(model, data, perturb, 1);
    
    }
    else{
        const QPointF delta = (event->position() - lastMousePosition) * devicePixelRatioF();
        mjv_moveCamera(model, mouseAction, 0.003*delta.x(), 0.003*delta.y(), scene, cam);
    }
        

    lastMousePosition = event->position();
    update();
}

Vector3f SimulationViewport::computeDelta(QMouseEvent* event) {
    
    Vector3f bodyTranslation = getBodyTranslation(dragSelection);
    Matrix3f bodyRotation = getBodyRotationMatrix(dragSelection);
    Vector3f cameraPos = getCameraInWorld();
    calcDragPlaneVector();

    // Intersection for current mouse position
    Vector3f currentPos;
    Vector3f projectedClick = projectClick( event->position().x() * devicePixelRatio(), event->position().y() * devicePixelRatio());
    bool currBool = intersectRayAndPlane(cameraPos, (projectedClick - cameraPos).normalized(), bodyTranslation, dragPlaneVector, currentPos);

    //  Intersection for current mouse position
    Vector3f previousPos;
    Vector3f prevProjectedClick = projectClick(lastMousePosition.x() * devicePixelRatio(), lastMousePosition.y() * devicePixelRatio());
    bool prevBool = intersectRayAndPlane(cameraPos, (prevProjectedClick - cameraPos).normalized(), bodyTranslation, dragPlaneVector, previousPos);

    return (currBool && prevBool) ? (currentPos - previousPos).eval() : Vector3f::Zero();
}

Vector3f SimulationViewport::projectClick(int x, int y) const{
  GLdouble mvMatrix[16];
  GLdouble projMatrix[16];
  for(int i = 0; i < 16; ++i){
    mvMatrix[i] = static_cast<GLdouble>(cameraTransformation[i]);
    projMatrix[i] = static_cast<GLdouble>(projection[i]);
  }

  GLdouble tx, ty, tz;
  gluUnProject(static_cast<GLdouble>(x), static_cast<GLdouble>(height - y), 1.0, mvMatrix, projMatrix, viewport, &tx, &ty, &tz);
  return Vector3f(static_cast<float>(tx), static_cast<float>(ty), static_cast<float>(tz));
}

intersectTayAndPlane: and been pasted and copy from a working simulator. It inteserct with normal to the xy plane.
projectClick: it also have been pasted and copy but I notice that sometime it depend on the camera zoom. Here is how I retreived the matrix ( I also struggled with it cause initially they were wrong in some ways due to wrong initializzation )

Confirmations

• I searched the latest documentation thoroughly before posting.
• I searched previous Issues and Discussions, I am certain this has not been raised before.