modules: add task watchdog

[alexcekay]

Solved Problem

• We had some drones with a certain task priority frozen leading to degraded performance. As we use MAVLink logging only on our Skynode-S the watchdog provided by the data-logger can not be used.
• In addition it is hard to debug what got stuck when only having a top output.

Solution

• Adds a new module that monitors if a low priority task is scheduled using an ISR (same idea as in data-logger)
• If task starvation is detected it dumps the registers and stack of all tasks to a file. In addition a top style output is written
• The data is dumped to a task_watchdog directory
• Added a health check that reports that a task starvation occured

Changelog Entry

For release notes:

Feature Add a task watchdog for mavlink logging only boards

Alternatives

• Tried to unify this with the logic already existing in data-logger and add all of the dumps to the .ulg file. This is not possible in our case due to the FRAM not allowing to creating even a bare-bones log. So not really possible to enable the file backend on the Skynode-S.

Test coverage

• Tested on the bench with an Auterion v6s

Note

• If one already uses the data-logger watchdog it is not recommended to also use this module as they would both trigger nearly at the same time and both try to run with increased priority
• If the reg and stack dumping logic would also be interesting for our boards we could invest more time in the future to unify this in the future

[github-actions]

🔎 FLASH Analysis

px4_fmu-v5x [Total VM Diff: 288 byte (0.01 %)]

    FILE SIZE        VM SIZE    
--------------  -------------- 
+0.0%    +288  +0.0%    +288    .text
  +0.1%    +160  +0.1%    +160    g_cromfs_image
  +2.2%     +76  +2.2%     +76    Commander::handle_command()
  [NEW]     +56  [NEW]     +56    CSWTCH.263
  [NEW]     +32  [NEW]     +32    CSWTCH.259
  +1.5%     +28  +1.5%     +28    HealthAndArmingChecks::HealthAndArmingChecks()
  +0.0%     +28  +0.0%     +28    [section .text]
   +45%     +20   +45%     +20    SdCardChecks::updateParamsImpl()
  +0.9%     +12  +0.9%     +12    Commander::dataLinkCheck()
  +3.3%     +12  +3.3%     +12    Commander::disarm()
  +3.3%      +8  +3.3%      +8    Commander::checkForMissionUpdate()
  +2.1%      +8  +2.1%      +8    Commander::executeActionRequest()
  +2.8%      +8  +2.8%      +8    Commander::manualControlCheck()
  +0.1%      +8  +0.1%      +8    px4::parameters
 -99.9%      +4 -99.9%      +4    [12 Others]
  [DEL]      -4  [DEL]      -4    CSWTCH.830
  -1.4%      -8  -1.4%      -8    Commander::arm()
  -0.8%      -8  -0.8%      -8    Commander::run()
  -3.3%     -12  -3.3%     -12    Commander::handleAutoDisarm()
  [DEL]     -32  [DEL]     -32    CSWTCH.258
  -9.2%     -52  -9.2%     -52    Commander::control_status_leds()
  [DEL]     -56  [DEL]     -56    CSWTCH.262
+0.0%     +82  [ = ]       0    .debug_abbrev
-0.0%      -4  [ = ]       0    .debug_frame
+0.0% +3.26Ki  [ = ]       0    .debug_info
-0.0%     -39  [ = ]       0    .debug_line
   +33%      +1  [ = ]       0    [Unmapped]
  -0.0%     -40  [ = ]       0    [section .debug_line]
-0.0%     -86  [ = ]       0    .debug_loclists
-0.0%     -39  [ = ]       0    .debug_rnglists
  [NEW]      +3  [ = ]       0    [Unmapped]
  -0.0%     -42  [ = ]       0    [section .debug_rnglists]
+0.0%   +1017  [ = ]       0    .debug_str
+0.0%     +16  [ = ]       0    .symtab
  [DEL]     -32  [ = ]       0    CSWTCH.258
  [NEW]     +32  [ = ]       0    CSWTCH.259
  [DEL]     -32  [ = ]       0    CSWTCH.262
  [NEW]     +32  [ = ]       0    CSWTCH.263
  [DEL]     -32  [ = ]       0    CSWTCH.830
  [NEW]     +32  [ = ]       0    CSWTCH.833
  +100%     +16  [ = ]       0    Commander::checkWorkerThread()
   +33%     +16  [ = ]       0    Commander::control_status_leds()
  +3.8%     +16  [ = ]       0    Commander::handle_command()
 -50.0%     -16  [ = ]       0    Commander::updateControlMode()
 -33.3%     -16  [ = ]       0    Commander::updateTunes()
 -25.0%     -16  [ = ]       0    SdCardChecks::checkAndReport()
  +100%     +16  [ = ]       0    SdCardChecks::updateParamsImpl()
   +11%     +32  [ = ]       0    ___ZL19param_get_cplusplustPf.isra.0_veneer
   +50%     +16  [ = ]       0    ___ZN4ListIP13MavlinkStreamE8IteratorppEv.isra.0_veneer
 -40.0%     -32  [ = ]       0    __nxsched_get_files_veneer
 -33.3%     -16  [ = ]       0    __stm32_recvfifo_veneer
 -20.0%     -16  [ = ]       0    matrix::Matrix<>::operator+=()
   +33%     +16  [ = ]       0    matrix::Matrix<>::operator/()
-2.5%    -288  [ = ]       0    [Unmapped]
+0.0% +4.19Ki  +0.0%    +288    TOTAL

px4_fmu-v6x [Total VM Diff: 248 byte (0.01 %)]

    FILE SIZE        VM SIZE    
--------------  -------------- 
+0.0%    +248  +0.0%    +248    .text
  +0.1%    +124  +0.1%    +124    g_cromfs_image
  +2.2%     +76  +2.2%     +76    Commander::handle_command()
  [NEW]     +56  [NEW]     +56    CSWTCH.263
  [NEW]     +32  [NEW]     +32    CSWTCH.259
  +1.5%     +28  +1.5%     +28    HealthAndArmingChecks::HealthAndArmingChecks()
   +45%     +20   +45%     +20    SdCardChecks::updateParamsImpl()
  +0.0%     +16  +0.0%     +16    [section .text]
  +0.9%     +12  +0.9%     +12    Commander::dataLinkCheck()
  +3.3%     +12  +3.3%     +12    Commander::disarm()
 -99.6%     +12 -99.6%     +12    [12 Others]
  +3.3%      +8  +3.3%      +8    Commander::checkForMissionUpdate()
  +2.1%      +8  +2.1%      +8    Commander::executeActionRequest()
  +2.8%      +8  +2.8%      +8    Commander::manualControlCheck()
  +0.1%      +8  +0.1%      +8    px4::parameters
  [DEL]      -4  [DEL]      -4    CSWTCH.830
  -1.4%      -8  -1.4%      -8    Commander::arm()
  -0.8%      -8  -0.8%      -8    Commander::run()
  -3.3%     -12  -3.3%     -12    Commander::handleAutoDisarm()
  [DEL]     -32  [DEL]     -32    CSWTCH.258
  -9.2%     -52  -9.2%     -52    Commander::control_status_leds()
  [DEL]     -56  [DEL]     -56    CSWTCH.262
+0.0%     +82  [ = ]       0    .debug_abbrev
-0.0%      -4  [ = ]       0    .debug_frame
+0.0% +3.20Ki  [ = ]       0    .debug_info
-0.0%     -39  [ = ]       0    .debug_line
  [NEW]      +1  [ = ]       0    [Unmapped]
  -0.0%     -40  [ = ]       0    [section .debug_line]
+0.0%      +2  [ = ]       0    .debug_loclists
-0.0%     -39  [ = ]       0    .debug_rnglists
  [NEW]      +3  [ = ]       0    [Unmapped]
  -0.0%     -42  [ = ]       0    [section .debug_rnglists]
+0.0%   +1017  [ = ]       0    .debug_str
+0.0%     +16  [ = ]       0    .symtab
  [DEL]     -32  [ = ]       0    CSWTCH.258
  [NEW]     +32  [ = ]       0    CSWTCH.259
  [DEL]     -32  [ = ]       0    CSWTCH.262
  [NEW]     +32  [ = ]       0    CSWTCH.263
  [DEL]     -32  [ = ]       0    CSWTCH.830
  [NEW]     +32  [ = ]       0    CSWTCH.833
  +100%     +16  [ = ]       0    Commander::checkWorkerThread()
   +33%     +16  [ = ]       0    Commander::control_status_leds()
  +3.8%     +16  [ = ]       0    Commander::handle_command()
 -50.0%     -16  [ = ]       0    Commander::updateControlMode()
 -33.3%     -16  [ = ]       0    Commander::updateTunes()
 -25.0%     -16  [ = ]       0    SdCardChecks::checkAndReport()
  +100%     +16  [ = ]       0    SdCardChecks::updateParamsImpl()
 -20.0%     -16  [ = ]       0    matrix::Matrix<>::operator+=()
   +33%     +16  [ = ]       0    matrix::Matrix<>::operator/()
-4.4%    -248  [ = ]       0    [Unmapped]
+0.0% +4.21Ki  +0.0%    +248    TOTAL

Updated: 2026-02-19T17:20:47

[dakejahl]

Do we need to guard this on #if CONFIG_MODULES_TASK_WATCHDOG?