RFC: Improve Sensor Testing Insfrastructure

[ubieda]

Introduction

In the latest Sensor WG Meeting (240819), it was discussed the current state of sensor testing and there was interest in improving this to make it more effective without implying a significant effort.

In the context of migrating Sensor drivers to the Sensor model V2 (Asynchronous, Read/Decode APIs) it becomes more important to have a solution in place to keep track of regressions, as well as validate the porting process along the way.

Problem description

Currently, there are two main approaches for testing sensor drivers:

1. Build-all sensors Testsuite -> https://github.com/zephyrproject-rtos/zephyr/tree/main/tests/drivers/build_all/sensor
2. Sensor Emulator Test-suites -> https://github.com/zephyrproject-rtos/zephyr/tree/main/tests/drivers/sensor

The issues with only relying on these two approaches:

• Build-all sensors:
  • Build-time only verification (for sensors with no emulator).
• Sensor Emulator Test-suites
  • Requires developing and maintaining an emulator + test-suite per sensor driver (non-negligible effort).
  • Limited number of emulators in-tree (and all most of them are based on Sensor model v1 -> Fetch/Get approach).
  • There have been bugs identified for some of the emulators themselves.

In general, there are challenges with the existing :

• We've identified some issues not covered by none of the above. Some examples:
  • Default Sensor Decoder returning incorrect values #72059
  • sensor_shell: strange and unexpected output when getting single axis from 3-axis data #72840
  • drivers: sensors: sensor_shell: fix hang & crash #72815
  • drivers: invalid trigger default value for LIS2DH (and LIS3DH etc.) #71227
• The existing testing infrastructure does not cover Read-Stream/Decode APIs nor triggers (or at least not in a significant way).
• The rate at which users contribute new sensors is very high (+10 PRs introducing new drivers at this moment).

Proposed change

HIL Testing Infrastructure in CI

Incorporate HIL sensor testing infrastructure in CI, in which run-time sensor operations are performed and verified. This implies test-beds provided by vendors and trusted community members are deployed and maintained operational.

To achieve this, we'd need to:

• Coordinate with the Testing WG and understand what options we have.
• Contact Sensor Vendors and trusted community members to provide and maintain a test-bed with supported sensors.
• Start deploying and adding test-beds nodes in CI infrastructure.

Testing Approach

• Go with generic tests (akin the existing generic sensor samples??) that is applicable to all sensors that support that type.
• Evaluate the following parameters
  • Sensor Polling (read/decode with results validation)
  • Attributes (e.g: change ODR or resolution and verify it is effective)
  • Triggers (to the extend that is possible in a static setup, e.g: Watermark, Stationary, FIFO Full).
  • Preferrably, evaluate the sensor on all supported transports (e.g: BME280 both SPI and I2C).

Concerns and Unresolved Questions

• Having enough vendors and community members to provide testing setup.
• Whether we use the Sensor Shell for this or stick to the aforementioned generic testsuites.

Alternatives

Not doing any change in sensor testing and keep on catching bugs as people report them.

[ubieda]

@hakehuang we'd like to loop-in the Testing WG to understand what options we have in making this happen. Could you provide your thoughts?

[teburd]

I completely agree with the HIL testing approach for sensors. I know @bperseghetti might have some thoughts on this as well.

[yperess]

The sensor build_all test actually has a generic test which will run against any sensor driver that has a paired emulator (https://github.com/zephyrproject-rtos/zephyr/blob/main/tests/drivers/build_all/sensor/testcase.yaml#L32). The idea when we wrote it was to reduce the friction of testing. When you add a new driver instead of driver + emulator + test you just write driver + emulator.

Here's an example of a driver which only implements the V2 API https://github.com/zephyrproject-rtos/zephyr/blob/main/drivers/sensor/bosch/bma4xx/bma4xx.c

Downstream in ChromeOS I'm also looking at test libraries which can run either by HIL or emulators. The idea is that you write the tests in a library which you can link against either an emulated environment or a hardware binary. When you run/flash you should see the same behavior. The idea was to run the same test in CI then a nightly run on hardware.

[yperess]

We're also working with the Fuchsia team on an open source sensor trace/playback system which should be RTOS agnostic. So we're hoping to be able to run a HIL test while capturing the data and replay it in CI emulation.

[bperseghetti]

I completely agree with the HIL testing approach for sensors. I know @bperseghetti might have some thoughts on this as well.

I think that having real hardware testing is critical for the health and reliability of sensors in Zephyr.

While I'm a big simulation/emulation fan in general, I think the risks are too great of "writing a test to the answers you know" and the amount of time to write them vs what they provide might not be a good cost value for the majority of sensors. With real hardware testing you sometimes might even find quirks/discrepancies from the assumed correct datasheets or implementations that look great in code but just don't function properly on device.

I think for us one of those learning experiences was the icm42688, we were able to write a rudimentary test for it for HIL and it was the only way we could work through getting it to properly function with streaming API and adjusting the FS/ODRs.

[hakehuang]

@hakehuang we'd like to loop-in the Testing WG to understand what options we have in making this happen. Could you provide your thoughts?

@yperess , I basically agree with this proposal. can you attend in next week TWG meeting, and we can have discussion on how to support those.

HIL Testing Infrastructure in CI
Incorporate HIL sensor testing infrastructure in CI, in which run-time sensor operations are performed and verified. This > implies test-beds provided by vendors and trusted community members are deployed and maintained operational.

To achieve this, we'd need to:

Coordinate with the Testing WG and understand what options we have.

-> the emulator solution in HIL running on bus simulator is far from enough. it has many assumptions. so it is only fit for sensor API change regression. and not fit for sensor driver modelling testing.

-> using qemu / renode with adding sensor support could be an option as the driver modelling can be added but need sensor vender support this.

-> real board testing, we can affording weekly testing on sensors we have from TWG prospective. but we need have those sensor available. this task can be further allocated to each vender, and we allow them to upload results as well.

Contact Sensor Vendors and trusted community members to provide and maintain a test-bed with supported sensors.
Start deploying and adding test-beds nodes in CI infrastructure.
Testing Approach
Go with generic tests (akin the existing generic sensor samples??) that is applicable to all sensors that support that type.
Evaluate the following parameters
Sensor Polling (read/decode with results validation)
Attributes (e.g: change ODR or resolution and verify it is effective)
Triggers (to the extend that is possible in a static setup, e.g: Watermark, Stationary, FIFO Full).

-> NXP can offer to testing sensors function in weekly regression, providing NXP platform is supported, or sensor team to port to NXP platform. we can send test report to sensor group

[yperess]

@yperess , I basically agree with this proposal. can you attend in next week TWG meeting, and we can have discussion on how to support those.

Yup, I'll be there.

[hakehuang]

Yup, I'll be there.

Thanks, I will add this to agenda.