[NXDART] Test Scenarios.

[cederom]

• In order to provide generic approach and universal architecture we need to define a standard set of test cases.
• There are two main categories of tests: Build and Runtime.
• Runtime tests can be: Internal (working as the NuttX binary application "inside" the MCU, no external components required, even onboard), and External (NuttX interaction with external world outside MCU takes place, i.e. sensors, network, WIFI, BLE, GPIO, other interfaces and peripherals).

Build Tests

• Mandatory: Build current nuttx and nuttx-apps master. This ensures current HEAD quality.
• Optional(?): Build current nuttx with older nuttx-apps (i.e. release package/tag or just N-steps back in repo history). This ensures backward compatibility between nuttx and nuttx-apps.
• Mandatory: Cover each architecture (MCUs to be selected).
• Optional: Cover each possible arch/board.
• Others?

Runtime Tests

• For now we use at least :nsh and :ostest runtime tests on existing configurations. nsh is the quickest way to build and verify board bringups. ostest validates various NuttX mechanisms (and will/should be extended with more).
• TODO: :nsh needs standardization and cleanup on all boards, so it only contains nsh and no other applications (build time), and exactly the same set of built-in features (cohoerence).

In future when basics are finished, test groups may be created according to concepts presented below.

Runtime Tests: Base

• These tests are mandatory.
• Must be implemented and supported on every board under evaluation.
• Ensures quality of builds provided by Build Tests.
• Test Cases (to be defined):
  • nsh.
  • uname -a.
  • help.
  • ostest.
  • free.

Runtime Tests: Extended

• These tests are optional.
• Ensures performance validation (stability/improvement/degradation).
• Test Cases (to be defined):
  • stress tests.
  • benchmarks.

Runtime Tests: Specific

• These tests are optional, they may be hardware specific (thus name).
• Ensures architecture / board / peripheral validation.
• If generic tests are used on different boards with no hardware support then UNAVAILABLE should be returned rather than FAILED.
• Test Cases (to be defined):
  • Ethernet.
  • WIFI.
  • Bluetooth.
  • LoRa.
  • LCD.
  • Touchscreen.
  • Joystick.

Runtime Tests: Custom

• Vendors must be able to re-use our framework on-premises with custom hardware to perform their own tests in a generic way.
• These tests can also reveal problems with our code (i.e. compatibility, efficiency, etc)

Comments below are welcome :-)