Motor burned due to overcurrent during position control with modified current control gains

[isorrentino]

📝 Description:

One of the motors burned out during a test. When it failed, it was extremely hot to the touch.

🧪 Steps leading to the failure:

1. The motor was being position-controlled.
2. I increased the current control gains.
3. With those gains, the motor immediately started whistling, so I quickly set the gains to zero.
4. During the time the gains were zero, the motor was still under position control.
5. The motor went into hardware fault due to overcurrent.

⚠️ Likely causes:

• The current control gains are not only used for closed-loop current control, but also in PWM (open-loop) mode to zero the direct current and to limit the maximum voltage for the quadrature axes.
  A possible solution would be to separate the gains used for closed-loop control (user-modifiable) from those used internally in other modes (hidden from the user).
• The current limits in the configuration files were incorrect. One suggestion to avoid motor damage in these cases could be adding runtime checks on both the requested current and motor temperature. This could improve safety.
• Some integrator-related variables are not reset when gains are updated. This can lead to accumulation and unintended behavior.

✅ Suggested sub-issues / action points:

• Separate current control gains used in closed-loop from those used in open-loop or internal modes.
• Ensure that all integrator-related variables are reset when gains are modified.
• Add runtime checks for output voltage over time and motor temperature.

cc @ale-git @valegagge

[pattacini]

Hi @isorrentino

I think that the service devoted to protecting the motor is the logic that triggers the HF-overload-current, which in turn needs to rely on correct values stored in the configuration files. This operation is instantaneous.

Add runtime checks for output voltage over time and motor temperature.

This has something to do with the I2T and, more thoroughly thought, with the thermal model of the motor, which both exhibit a slow intervention rate. They also need more difficult fine-tuning than a (relatively) simple overload current threshold.

We could revise the algorithm implemented in the 2FOC to verify that it is effective and correctly configured.

Ensure that all integrator-related variables are reset when gains are modified.

This may be somehow arguable as in control we often favor bumpless transitions (e.g., to enable gain scheduling), which are not guaranteed 100% as of now when updating the gains I think, but they would be even more impacted by an integral reset. It very depends on the scenario at stake.

Separate current control gains used in closed-loop from those used in open-loop or internal modes.

I'm not sure what could happen if we would control Id and Iq with 2 different sets of gains. I'm not smart enough to figure this out in my head. Some tests are required at least.
True that they're 2 different paths and we may be more conservative on one wrt the other, but they're still tightly related because of FOC.

Also, I'm pretty uncertain whether the direct path could have generated the current that burnt the motor. The integral is more likeley to be the culprit here, I guess (but my previous comments on bumpless transfers hold as well).

---

To sum up, a couple of final comments:

• Probably, setting a good estimate of the overload current (even taking some safety margin) is the quickest and safest way to go.
• I'm not saying that there's no room to improve the present situation; however, I deem there's the need to interact/iterate and collect stronger evidence that these options can serve the purpose.

[isorrentino]

Hi @pattacini, thank you for your reply.
@ale-git was with me when this happened, and since he is more experienced with the 2FOC firmware, he might be able to provide additional insights or clarify further what I’ve already mentioned.

[pattacini]

Good that @ale-git is already in the loop 👍🏻

Yeah, I think further insights are required indeed.

[ale-git]

About the problem of the runtime PID change, we could reset the integral only if Ki is 0. It seems reasonable because if Ki=0 it means that we don't want integral part, but above all it is also unlikely that Ki is 0, so it would just prevent that the motor burns if Ki is set to 0 by mistake.

I think that the combinations of two things made the motor burn: the fixed integral part contributing to Vq not resetted, and the current limit set for R1, otherwise the I2T mechanism would have raised a fault.

About the overcurrent limit, it is managed by the 2FOC hardware and set to 21 A or more (depending on board version) as a protection against short-circuits. We can add a software overcurrent protection acting on user defined values under 21 A, but currents over currentPeak are already prevented by software.

[pattacini]

About the problem of the runtime PID change, we could reset the integral only if Ki is 0. It seems reasonable because if Ki=0 it means that we don't want integral part, but above all it is also unlikely that Ki is 0, so it would just prevent that the motor burns if Ki is set to 0 by mistake.

Resetting the integral only if Ki=0 shouldn't interfere with the requirement of bumpless transitions 👍🏻

Nonetheless, if current caps and I2T are correctly configured I don't think it is really necessary to do so, if I'm not mistaken. Are there cases where the opposite holds?

On the other hand, if caps and I2T are NOT correctly configured, having the further strategy on the integral reset won't be sufficient at all.

[pattacini]

On the other hand, if caps and I2T are NOT correctly configured, having the further strategy on the integral reset won't be sufficient at all.

The reset actually would have discharged the integral 👍🏻
Anyhow, intervening by setting Ki=0 to recover from a whistling motor instead of pressing the fault button seems a bit stretched.
At any rate, it could be an option.

[ale-git]

Nonetheless, if current caps and I2T are correctly configured I don't think it is really necessary to do so, if I'm not mistaken. Are there cases where the opposite holds?

Basically the I2T starts growing as soon as the current exceeds Inom, and at a certain point it will raise a fault.
Many years ago I implemented it in the way it was requested, though I was skeptical because it has no physical meaning at all. In the real world also a current under Inom generates heat. Moreover this mechanism doesn't take into account thermal capacity and conductivity.

void setMaxCurrent(int nom, int peak, int ovr)
{
    Inom = nom;
    Ipeak = peak;
    I2nom = (long)nom*(long)nom;
    I2Tlimit = ((long)peak*(long)peak - I2nom) * 10;
    Iovr = ovr;
}

inline void I2Twatcher(void)
{
    I2Tacc += (__builtin_mulss(I2Tdata.IQMeasured,I2Tdata.IQMeasured) - I2nom) >> 10;

    if (I2Tacc < 0) I2Tacc = 0;

    if (I2Tacc > I2Tlimit)
    {
        //The temperature grew too much. Protect!
        SysError.I2TFailure = TRUE;
        FaultConditionsHandler();
    }
}

Empirically it works, more or less, but I would be very happy to get rid of it for a simple thermal model like

$$ \dot{T} = \frac {R I_{meas}^2}{C_{th}} - \frac {T-20}{R_{th}} $$

Where $C_{th}$ = thermal capacity and $R_{th}$ = thermal resistance.

[pattacini]

Yeah, the thermal model is the principled way to go, instead of relying on the simplistic I2T approach.

However, fiddling with thermal quantities (i.e. resistances, capacitances) is not an easy job.

We made the attempt to implement a thermal model for the AMCBLDC in the past with @mfussi66:

• Here's the thermal model.
• Here's the original underlying discussion stemming from the Faulhaber white-paper/datasheet, where the capacitance is subsumed somehow in the time constant.
• Our attempts to fit the quantities vs. datasheet:
  • AMCBLDC – Validate the thermal model further and link it to the control logic icub-firmware-models#34
  • Fit the thermal model icub-firmware-models#36
  • Refine thermal model with inductive load icub-firmware-models#42

As of now, we haven't reached a good trade-off we're happy of yet¹ and we put it in the icebox because of other more urgent developments.

That being said, a thermal model is an important piece of SW that requires some care and perhaps is a bit beyond the scope of this task.

Footnotes

1. It turned out that we may also need the motor inductance to better model the current behavior. ↩

[octo-reminder]

We couldn't respond to your request in time. Sorry about that. Please try resubmitting your request and contact us if the problem persists.

[isorrentino]

Hi @pattacini and @ale-git, thank you both for the detailed responses. If I understood correctly and to summarize:

• The current overcurrent limit was too high, and the I2T mechanism didn’t trigger in time.
• The integral contribution to Vq was not reset when the gains were set to zero.
• Implementing a more realistic thermal model would be ideal in the long term, but I understand that it's a non-trivial effort and probably outside the scope of this issue.

In the short term, do you think it would be reasonable to automatically reset the integrator when Ki = 0? This might help prevent unintended accumulation.

Let me know if I can help with testing or implementation.

[pattacini]

In the short term, do you think it would be reasonable to automatically reset the integrator when Ki = 0? This might help prevent unintended accumulation.

This could be implemented in the 2FOC 👍🏻
I've been added to a meeting on Wed afternoon. We can discuss it.

Warning

Just remember: only if Ki = 0 (to guarantee bumpless transfers in all the other cases).