Touch sensor plugin which allows end-user access to a sensors associated taxel positions and force expended

[tumultousRamen]

I am using the touch sensor plugin to develop simulations for an Allegro Hand v3 (https://www.wonikrobotics.com/research-robot-hand) robot developed by Wonik Robotics (https://www.wonikrobotics.com/).

While working with the plugin and going through its source code, I think that giving the end-user access to each individual taxel associated with a sensor would be quite beneficial and was working on implementing a plug-in that could help us achieve that. I am relatively new to MuJoCo, so please pardon some basic questions! I was wondering if:

1. Can we make plug-ins for MuJoCo using Python?
2. If not, for broader impact, does implementing a Python wrapper for a C++ plug-in make sense?
3. Would making the computation differentiable be beneficial?

[yuvaltassa]

1. Yes.
2. No, since the C++ plugins are directly used by the engine. If you wrap them in Python you might as well write them in Python.
3. I don't know, would it? I don't think that's really possible since contacts are binary events, and even if you make the contacts soft the gradients would be very very large.

[tumultousRamen]

Thanks for the reference and the insights, @yuvaltassa !

[tumultousRamen]

Hi @yuvaltassa ,

I decided to go forward with a C++ implementation, keeping in mind the performance issues that might arise from implementing it in Python.

The plug-in sensor example (here) that I've been using has methods like compute, visualize etc that have their own callbacks that are triggered by the engine and compiler.

I was wondering if it makes sense to write a new callback for a method that I wrote that calculates the taxel centers. If not, what are some other ways we can let end users access the calculated taxel center points?

void TouchGrid::ComputeTaxelCenters(const mjModel* m, mjData* d, int instance) {
  // Get site id and frame
  int site_id = m->sensor_objid[m->sensor_plugin[instance]];
  mjtNum* site_pos = d->site_xpos + 3*site_id;
  mjtNum* site_mat = d->site_xmat + 9*site_id;

  // Allocate space for taxel centers (3 coordinates per taxel)
  taxel_centers_.resize(size_[0] * size_[1] * 3);

  // Allocate bin edges
  mjtNum x_edges[size_[0] + 1];
  mjtNum y_edges[size_[1] + 1];

  // Make bin edges
  BinEdges(x_edges, y_edges, size_, fov_, gamma_);

  // Compute center points
  for (int i = 0; i < size_[0]; i++) {
    for (int j = 0; j < size_[1]; j++) {
      mjtNum aer[3];
      aer[0] = 0.5*(x_edges[i+1] + x_edges[i]);
      aer[1] = 0.5*(y_edges[j+1] + y_edges[j]);
      aer[2] = 1.0;  // Unit distance

      mjtNum pos[3];
      SphericalToCartesian(aer, pos);
      mju_mulMatVec3(pos, site_mat, pos);
      mju_addTo3(pos, site_pos);

      int idx = (j*size_[0] + i) * 3;
      taxel_centers_[idx] = pos[0];
      taxel_centers_[idx+1] = pos[1];
      taxel_centers_[idx+2] = pos[2];
    }
  }

  delete[] x_edges;
  delete[] y_edges;
}

[yuvaltassa]

The generic thing is to use mjData.user_data, you declare it like this