Troubleshooting Idle Robot on the Field

[jasondaming]

I want a better name for this. This should all be information that a CSA / FTA knows but this will help let teams inside our head and let us know what we are looking for when a robot goes idle on the field and what additional troubleshooting should be done based on this problem discovery.

[Starlight220]

I think this can be expanded into a more general troubleshooting section.

Some issues I'm thinking of:

• Code crashes: stacktrace/error messages usually indicate the issue (and the fix). Standard Java exceptions that are thrown from deep inside WPILib (and not from team-written code, which is more accessible to fix) are either (1) NPEs from user-given nulls (shouldn't really happen now that there are requireNonNull calls almost everywhere in WPILib), (2) overriding methods that shouldn't be overridden without calling super. Both of these are caused by team code interacting with WPILib in an unsupported way. Unless I'm forgetting something, an allwpilib Issue should be opened for other stdlib Java exceptions thrown inside WPILib.

• Tripped main breaker: the breaker is physically open and all LEDs are off; high current use in DS logs. Usually fixable with current limiting (or fixing bad wiring).

• Brownouts causing high latency / packet loss.

• Commands don't run: missing CommandScheduler.run() call.

• missing some CAN frames but not others from a specific configured motor controller but not others: tripping specific (40A etc) breakers (see story below); usually fixable with current limiting (or fixing bad wiring). I'm expanding on this story below because it's an unintuitive diagnosis (at least it was to me).

Story

A motor controller (SparkMax in this instance) erratically stopped working in the middle of matches (recurred for quite a few matches), with error messages being printed about status frame 2 being missing from that motor controller (but no error messages about frames 0 and 1). The issue was that we were tripping the 40A breaker (and when we thought of checking DS logs for this, it was evident), causing the motor controller to restart and lose the status frame configs (causing a discrepancy between how often the code expects to receive a status frame and how often the motor controller sends a status frame): Frames 0 and 1 were configured to be sent less frequently than the default, so the code didn't expect them to be sent -- so no "missing frame" errors; status frame 2 was configured to be sent more frequently than the default, so the code expected status frames that weren't sent -- thus a "missing frame" error.

[Daltz333]

Code crashes: stacktrace/error messages usually indicate the issue (and the fix). Standard Java exceptions that are thrown from deep inside WPILib (and not from team-written code, which is more accessible to fix) are either (1) NPEs from user-given nulls (shouldn't really happen now that there are requireNonNull calls almost everywhere in WPILib), (2) overriding methods that shouldn't be overridden without calling super. Both of these are caused by team code interacting with WPILib in an unsupported way. Unless I'm forgetting something, an allwpilib Issue should be opened for other stdlib Java exceptions thrown inside WPILib.

Honestly, I've never seen this happen on a field. In the pits? Sure. I would not classify this as a field problem, as most issues are going to be identified at compile time.

Tripped main breaker: the breaker is physically open and all LEDs are off; high current use in DS logs. Usually fixable with current limiting (or fixing bad wiring).

Yep

Brownouts causing high latency / packet loss.

Yep.

Commands don't run: missing CommandScheduler.run() call.

Again, assuming the team tests basic robot functionality in the pits. I would not classify this as a "field" problem.

[jasondaming]

I have encountered code crashing quite often and would say it is probably one of the most common causes of a team being bypassed. I completely agree that the vast majority of the time this is caught before the field but when it isn't the field can't really let them troubleshoot it or push new code. If it is constantly crashing their robot program there is not much to be done.

To be fair though there is not too much advice we can provide on this other than "fix the error" and make sure to test all autos and different permutations of scenarios that could happen on the field.

[Starlight220]

@Daltz333 I've seen both of the cases you've doubted happen on the field, and helped teams with them at competitions. The first is definitely not catched at compile-time (unless we make those methods final, which is something I've suggested in the past--there's a PR open for that). The second is less likely, but I've seen teams put the Scheduler.run() call in teleopPeriodic instead of robotPeriodic and then their autos don't run.

[Daltz333]

What I specifically think should happen is code specific issues being in a separate troubleshooting article. I added a troubleshooting section in #1814 that can be expanded to identifying and troubleshooting common code related issues.

[jasondaming]

Yes I don't think this should delve into code issues as much as say that code issues could leave you stranded on the field if certain things happen and possibly some advice for mitigating that chance.