Migration and road map from the deprecated testing API to the new ZTEST API.

[aaronemassey]

Migration and road map from deprecated testing API to the current API.

Each test suite should be refactored to the new ZTEST API.

Some example API migrations

• Bluetooth AT
• CMSIS DSP
• Condition Variables
• Posix

Migration status

The list below was initially generated by finding tests without a ZTEST_SUITE
macro usage and roughly categorizing them with respect to their directories and
tested features. For example, tests/application_development and
tests/arch/arm contain unmigrated tests.

• application_development
  • code_relocation [@hakehuang ]
  • gen_inc_file [@enjiamai ]
• arch/arm [@shaoanxli ] (12/12)
• arch/arm64 [@carlocaione] (3/3)
• arch/x86 + others [@enjiamai, @edersondisouza ] (3/3)
• benchmarks (@enjiamai )
• bluetooth
  • @kruithofa for controller/ctrl_
  • @ppryga for connection{less}_*
  • @carlescufi for the rest
• boards (5/5)
  • boards/altera_max10 (@enjiamai )
  • boards/espressif_esp32 (@enjiamai )
  • boards/intel_adsp (@enjiamai )
  • boards/mec15xxevb_assy6853 (@enjiamai )
  • boards/native_posix (@enjiamai )
  • boot/uefi (This is not using ztest, skip)
• cmake/config_dir
• crypto (@enjiamai )
• drivers [ ] (37/39)
  • adc (@hakehuang )
  • drivers/bbram (@mengxianglinx)
  • drivers/can (@henrikbrixandersen )
  • drivers/clock_control (#mengxianglinx )
  • drivers/counter (@enjiamai )
  • drivers/dac (@enjiamai )
  • dma (@hakehuang )
  • drivers/disk (@enjiamai )
  • eeprom (@henrikbrixandersen )
  • drivers/entropy (@enjiamai )
  • drivers/espi (@enjiamai )
  • drivers/flash (@enjiamai )
  • drivers/flash_simulator (@enjiamai )
  • drivers/gpio (@enjiamai )
  • drivers/hwinfo (@shaoanli )
  • drivers/i2c (@enjiamai )
  • drivers/i2s/i2s_api (@hunterhu3000 , tests: drivers: i2s: i2s_api: move to new ztest API #48231)
  • drivers/i2s/i2s_speed (@hunterhu3000 , tests: drivers: i2s: i2s_speed: move to new ztest API #48239)
  • drivers/i2s/ipm (This is not using ztest, should we apply it?)
  • drivers/kscan/kscan_api (@hunterhu3000 )
  • drivers/led/led_api (@enjiamai )
  • drivers/memc/stm32_sdram/ (@pdgendt )
  • drivers/mipi_dsi/api/ (@enjiamai )
  • drivers/pinctrl/api (@yperess )
  • drivers/pinctrl/gd32 (@enjiamai )
  • drivers/pinctrl/nrf (@enjiamai )
  • drivers/pwm/ (@enjiamai , tests: drivers: pwm: move the tests to new ztest API #49511 )
  • drivers/regulator/ (@enjiamai )
  • drivers/sdhc (@danieldegrasse )
  • drivers/sensor/generic/ (@aaronemassey )
  • drivers/sensor/accel (@enjiamai )
  • drivers/sensor/sbs_gauge (@aaronemassey )
  • drivers/spi/ (@enjiamai )
  • drivers/syscon (@enjiamai )
  • drivers/timer/nrf_rtc_timer (@enjiamai )
  • drivers/uart (@enjiamai )
  • drivers/virtualization/ivshmem (@enjiamai )
  • drivers/w1/w1_api ( @str4t0m )
  • drivers/watchdog/wdt_basic_api (@aaronemassey )
• kernel ( @nashif , @smrtos, @LixinGuoX ) (38/38)
  • kernel/condvar (@nashif,)
  • kernel/common (@LixinGuoX,)
  • kernel/context (@LixinGuoX)
  • kernel/device (@LixinGuoX)
  • kernel/early_sleep (#LixinGuoX )
  • kernel/events (@LixinGuoX)
  • kernel/fatal (@LixinGuoX)
  • kernel/fifo (@LixinGuoX)
  • kernel/fpu_sharing (@LixinGuoX)
  • kernel/gen_isr_table (@LixinGuoX)
  • kernel/interrupt (@enjiamai)
  • kernel/lifo (@LixinGuoX)
  • kernel/mbox (@enjiamai )
  • kernel/mem_heap (@Zhaoningx )
  • kernel/mem_protect (@enjiamai)
  • kernel/mem_slab (@Zhaoningx )
  • kernel/mp (@enjiamai
  • kernel/msgq (@mengxianglinx )
  • kernel/mutex (@smrtos)
  • kernel/obj_tracking (@LixinGuoX)
  • kernel/pending (@LixinGuoX)
  • kernel/pipe (@LixinGuoX)
  • kernel/poll (@Zhaoningx )
  • kernel/profiling (@LixinGuoX)
  • kernel/queue (@Zhaoningx )
  • kernel/sched (@smrtos )
  • kernel/semaphore (@mengxianglinx)
  • kernel/sleep (@LixinGuoX)
  • kernel/smp (@LixinGuoX)
  • kernel/smp_boot_delay (@LixinGuoX)
  • kernel/spinlock (@LixinGuoX)
  • kernel/stack (@LixinGuoX)
  • kernel/threads (@Zhaoningx)
  • kernel/tickless (@LixinGuoX)
  • kernel/timer (@enjiamai )
  • kernel/usage (@LixinGuoX)
  • kernel/workq (@Zhaoningx )
  • kernel/xip @LixinGuoX
• lib [@yerabolu, @1xlibiao ] (28/29, 97%)
  • lib/cmsis_dsp/filtering[@enjiamai ]
  • lib/cmsis_dsp/interpolation[@enjiamai ]
  • lib/cmsis_dsp/matrix[@enjiamai ]
  • lib/cmsis_dsp/quaternionmath (@1xlibiao)
  • lib/cmsis_dsp/statistics[@enjiamai ]
  • lib/cmsis_dsp/support[@enjiamai ]
  • lib/cmsis_dsp/svm (@1xlibiao )
  • lib/cmsis_dsp/transform[@enjiamai ]
  • lib/devicetree (@1xlibiao)
  • lib/fdtable (@1xlibiao)
  • lib/gui/lvgl (@1xlibiao)
  • lib/heap (@1xlibiao)
  • lib/heap_align (@1xlibiao)
  • lib/json (@1xlibiao)
  • lib/cmsis_nn (@yerabolu)
  • lib/mem_alloc (#yerabolu , tests: lib: mem_alloc: migrating mem_alloc tests to new ztest API #50164 )
  • lib/mem_blocks (@yerabolu)
  • lib/mpsc_pbuf (@yerabolu)
  • lib/newlib/heap_listener (@yerabolu )
  • lib/newlib/thread_safety (@yerabolu )
  • lib/notify (@yerabolu)
  • lib/onoff (@yerabolu)
  • lib/p4workq (@yerabolu)
  • lib/ringbuffer (@1xlibiao )
  • lib/smf (@1xlibiao )
  • lib/sprintf (@yerabolu)
  • lib/spsc_pbu (@nordic-krch )
  • lib/sys_util (@yerabolu )
  • lib/time (@yerabolu)
• misc [@chen-png ] (2/2)
• net [@hunterhu3000, @xiaos2x ] ( 66/73, 90%)
  • net/6lo (@xiaos2x)
  • net/all (@xiaos2x)
  • net/arp (@xiaos2x)
  • net/bridge (@xiaos2x)
  • net/buf (@xiaos2x)
  • net/checksum_offload (@xiaos2x)
  • net/context (@hunterhu3000 )
  • net/dhcpv4 (@xiaos2x)
  • net/ethernet_mgmt (@hunterhu3000 )
  • net/hostname (@xiaos2x)
  • net/udp (@xiaos2x)
  • net/icmpv4 (@hunterhu3000 , tests: net: icmpv4 move to new ztest API #49381 )
  • net/icmpv6 (@xiaos2x)
  • net/ieee802154/custom_l2 (@xiaos2x)
  • net/ieee802154/fragment (@xiaos2x)
  • net/ieee802154/l2 (@xiaos2x)
  • net/iface (@hunterhu3000 )
  • net/igmp (@hunterhu3000 )
  • net/ip-addr (@xiaos2x)
  • net/ipv6 (@hunterhu3000 )
  • net/ipv6_fragment (@xiaos2x)
  • net/lib/coap (@xiaos2x)
  • net/lib/dns_addremove (@xiaos2x )
  • net/lib/dns_packet (@xiaos2x)
  • net/lib/dns_resolve (@xiaos2x)
  • net/lib/dns_sd (@xiaos2x)
  • net/lib/http_header_fields (@xiaos2x)
  • net/lib/lwm2m/content_json (@hunterhu3000 )
  • net/lib/lwm2m/content_link_format (@hunterhu3000 )
  • net/lib/lwm2m/content_oma_tlv (@hunterhu3000 )
  • net/lib/lwm2m/content_plain_text (@hunterhu3000 )
  • net/lib/lwm2m/content_raw_cbor (@hunterhu3000 )
  • net/lib/lwm2m/content_senml_cbor (@hunterhu3000 , tests: net: lib: lwm2m: content_senml_cbor: move to new ztest API #49895 )
  • net/lib/mqtt_packet (@xiaos2x)
  • net/lib/mqtt_publisher (@hunterhu3000 )
  • net/lib/mqtt_pubsub (@hunterhu3000 )
  • net/lib/mqtt_subscriber (@hunterhu3000 )
  • net/lib/tls_credentials (@hunterhu3000 )
  • net/mgmt (@hunterhu3000)
  • net/mld (@hunterhu3000)
  • net/neighbor (@hunterhu3000)
  • net/net_pkt (@hunterhu3000)
  • net/npf (@hunterhu3000)
  • net/pm (@hunterhu3000)
  • net/ppp (@hunterhu3000)
  • net/promiscuous (@hunterhu3000)
  • net/ptp (@hunterhu3000)
  • net/route (@hunterhu3000)
  • net/route_mcast/ (@hunterhu3000)
  • net/shell (@hunterhu3000)
  • net/socket/af_packet (@hunterhu3000)
  • net/socket/af_packet_ipproto_raw (@hunterhu3000) suffle fail, cannot run twice
  • net/socket/getaddrinfo (@hunterhu3000) suffle fail, cannot run twice
  • net/socket/getnameinfo (@hunterhu3000)
  • net/socket/misc (@hunterhu3000)
  • net/socket/net_mgmt (@hunterhu3000) unable to suffle
  • net/socket/offload_dispatcher (@hunterhu3000 )
  • net/socket/poll (@hunterhu3000) suffle fail, cannot run twice
  • net/socket/register (@hunterhu3000) unable to suffle
  • net/socket/select (@hunterhu3000)
  • net/socket/socketpair (@hunterhu3000 )
  • net/socket/tcp (@hunterhu3000 )
  • net/socket/tls (@hunterhu3000 )
  • net/socket/tls_ext (@hunterhu3000, tests: net: socket: tls_ext move to new ztest API #48212)
  • net/socket/udp (@hunterhu3000 )
  • net/socket/websocket (@hunterhu3000, tests: net: socket: websocket move to new ztest API #48224)
  • net/tcp (@hunterhu3000 , net: lib: coap: update method_from_code() to report success/failure #49498 )
  • net/traffic_class (@hunterhu3000 )
  • net/trickle/ (@hunterhu3000, tests: net: trickle move to new ztest API #48226)
  • net/tx_timestamp (@hunterhu3000, tests: net: tx_timestamp move to new ztest API #48227)
  • net/utils (@hunterhu3000, tests: net: utils move to new ztest API #48228)
  • net/virtual (@hunterhu3000 )
  • net/vlan (@hunterhu3000) suffle fail, cannot run twice
• subsys [ ] ( 38/38 , 100%)
  • subsys/logging/log_msg (@1xlibiao )
  • subsys/logging/log_immediate (@1xlibiao )
  • subsys/logging/log_backend_init (@1xlibiao )
  • subsys/logging/log_switch_format (@1xlibiao )
  • subsys/logging/log_benchmark (@shaoanxli )
  • subsys/logging/log_backend_fs (@shaoanxli )
  • subsys/logging/log_core_additional (@shaoanxli )
  • subsys/logging/log_api (@yperess )
  • subsys/logging/log_output (@nordic-krch )
  • subsys/logging/log_stack (@nordic-krch )
  • subsys/logging/log_stress (@nordic-krch )
  • subsys/logging/log_syst (@1xlibiao )
  • subsys/sd/ (@Zhaoningx)
  • subsys/jwt (@shuai1x)
  • subsys/mgmt (@yinfangw )
  • subsys/canbus (@henrikbrixandersen )
  • subsys/usb (@shuai1x)
  • subsys/cpp (@shuai1x)
  • subsys/edac (@yinfangw )
  • subsys/pm/ (@yinfangw )
  • subsys/storage (@yinfangw )
  • subsys/shell (@yinfangw)
  • subsys/modbus (@Zhaoningx )
  • subsys/ipc/ (@Zhaoningx)
  • subsys/settings/fcb_init (@Kunli123 )
  • subsys/settings (@mengxianglinx )
  • subsys/rtio/ (@teburd)
  • subsys/fs/fat_fs_api (@shuai1x)
  • subsys/fs/fat_fs_dual_drive (@shuai1x )
  • subsys/fs/fs_api (#mengxianglinx )
  • subsys/fs/littlefs (@Kunli123 )
  • subsys/fs/multi-fs (@shuai1x )
  • subsys/fs/fcb (@yinfangw )
  • subsys/fs/nvs (@simhein )
  • subsys/portability/ (#shuai1x )
  • subsys/dfu (@shuai1x)
  • subsys/tracing (@mengxianglinx )
  • subsys/debug (@yinfangw)
• unit
  • list (@Zhaoningx )
  • rbtree (@Zhaoningx )
  • util (@enjiamai )
  • others (@enjiamai )
• ztest

Simple migration workflow

This does not cover all migration cases nor the entirety of the ZTEST API. This
is just a simple workflow get started on simple tests.

• Replace ztest_test_suite() and its containing function with a
  ZTEST_SUITE() defined here with documented
  parameters.
• Replace prior test functions invoking zassert with ZTEST or ZTEST_USER.
• Add CONFIG_ZTEST_NEW_API=y to the test's prj.conf.

Difficulty or questions

Usually, a single test's migration shouldn't require more than 10 minutes. If
you're having difficulty porting a test or understanding how to use the new
ZTEST API, then make a comment on this issue.

Some suggestions

• Try the new CONFIG_ZTEST_SHUFFLE to check if the ported test functions are self-contained/independent.
• Keep the old CONFIG_MULTITHREADING as the old test cases.
• Do not have a “test_” prefix in the test suite name.
• One test suite per commit. One PR to hold all related commits.

[nashif]

for all of those working on this, do not have a test_ prefix in the testsuite name. This is just confusing and redundant.

[hakehuang]

@aaronemassey , I have a problem here, there used to be some test fucntions are registerred as shall command see below code

extern void test_dma_m2m_chan0_burst8(void);
extern void test_dma_m2m_chan1_burst8(void);
extern void test_dma_m2m_chan0_burst16(void);
extern void test_dma_m2m_chan1_burst16(void);

#ifdef CONFIG_SHELL
TC_CMD_DEFINE(test_dma_m2m_chan0_burst8)
TC_CMD_DEFINE(test_dma_m2m_chan1_burst8)
TC_CMD_DEFINE(test_dma_m2m_chan0_burst16)
TC_CMD_DEFINE(test_dma_m2m_chan1_burst16)

SHELL_CMD_REGISTER(test_dma_m2m_chan0_burst8, NULL, NULL,
			TC_CMD_ITEM(test_dma_m2m_chan0_burst8));
SHELL_CMD_REGISTER(test_dma_m2m_chan1_burst8, NULL, NULL,
			TC_CMD_ITEM(test_dma_m2m_chan1_burst8));
SHELL_CMD_REGISTER(test_dma_m2m_chan0_burst16, NULL, NULL,
			TC_CMD_ITEM(test_dma_m2m_chan0_burst16));
SHELL_CMD_REGISTER(test_dma_m2m_chan1_burst16, NULL, NULL,
			TC_CMD_ITEM(test_dma_m2m_chan1_burst16));
#endif

shall we create a marco for ZTEST_SH which looks like below

#define Z_TEST_WITH_SH(suite, fn, t_options, use_fixture)                                                  \
	static void _##suite##_##fn##_wrapper(void *data);                                         \
	static void suite##_##fn(                                                                  \
		COND_CODE_1(use_fixture, (struct suite##_fixture *fixture), (void)));              \
	static STRUCT_SECTION_ITERABLE(ztest_unit_test, z_ztest_unit_test_##suite##_##fn) = {      \
		.test_suite_name = STRINGIFY(suite),                                               \
		.name = STRINGIFY(fn),                                                             \
		.test = (_##suite##_##fn##_wrapper),                                               \
		.thread_options = t_options,                                                       \
	};                                                                                         \
	static void _##suite##_##fn##_wrapper(void *data)                                          \
	{                                                                                          \
		COND_CODE_1(use_fixture, (suite##_##fn((struct suite##_fixture *)data);),          \
			    (ARG_UNUSED(data); suite##_##fn();))                                   \
	}                                                                                          \
       TC_CMD_DEFINE(suite##_##fn)                                                           \
       SHELL_CMD_REGISTER(suite##_##fn, NULL, NULL, TC_CMD_ITEM(suite##_##fn));  \
	static inline void suite##_##fn(                                                           \
		COND_CODE_1(use_fixture, (struct suite##_fixture *fixture), (void)))

[aaronemassey]

@hakehuang

@aaronemassey , I have a problem here, there used to be some test fucntions are registerred as shall command see below code

Looking at this still, but I'm a bit confused as to actually how the SHELL_COMMAND_REGISTER macro is actually being leveraged in this test. It looks like the test is simply running the test functions themselves without respect to the shell command macro. Maybe we can discuss this at the next working group meeting (tomorrow).

Apologies for late response. Been moving.