Skip to content

RFC: twister: Include coverage support for device testing #31245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Feb 15, 2021
Merged

RFC: twister: Include coverage support for device testing #31245

merged 2 commits into from
Feb 15, 2021

Conversation

@finikorg
Copy link
Contributor

Include coverage support for device testing. It works by switching
capture_coverage to True (as it would meet coverage data start
pattern). Then we continue tot read coverage data until we get
coverage data end pattern. Otherwise, after receiving test result
pattern, we close serial console and do not get time to find start
coverage data pattern.

@finikorg finikorg requested a review from nashif January 12, 2021 08:04
@github-actions github-actions bot added the area: Twister Twister label Jan 12, 2021
@zephyrbot zephyrbot added the platform: X86 x86 and x86-64 label Jan 12, 2021
@zephyrbot zephyrbot requested a review from jhedberg January 12, 2021 11:35
@nashif nashif assigned nashif and unassigned jenmwms Jan 12, 2021
@finikorg finikorg force-pushed the coverage branch 2 times, most recently from d72a791 to 1ef8646 Compare January 13, 2021 14:09
@nashif nashif requested review from LeiW000, chen-png and enjiamai and removed request for andrewboie January 30, 2021 13:13
@nashif
Copy link
Member

nashif commented Feb 1, 2021

@finikorg please rebase

@finikorg
boards: ehl_crb: Add coverage support for the board
87fed42 
Allow to include coverage support for ehl_crb board.

Signed-off-by: Andrei Emeltchenko <[email protected]>
@finikorg
twister: Include coverage support for device testing
ba712f6 
Include coverage support for device testing. It works by switching
capture_coverage to True (as it would meet coverage data start
pattern). Then we continue tot read coverage data until we get
coverage data end pattern. Otherwise, after receiving test result
pattern, we close serial console and do not get time to find start
coverage data pattern.

Signed-off-by: Andrei Emeltchenko <[email protected]>
@finikorg finikorg changed the title RFC: twister: Include coverage support for device testing twister: Include coverage support for device testing Feb 1, 2021
@chen-png
Copy link

chen-png commented Feb 2, 2021

tested on ehl board, it could get coverage data, but for some testcases, the default timeout is not enough to get test results and all coverage data, it will timeout like below, the testcase is passed, but have not got all coverage data.

DEBUG - Timed out while monitoring serial output on ehl_crb
DEBUG - run status: ehl_crb/tests/kernel/threads/thread_apis/kernel.threads.apis passed
INFO - 1/1 ehl_crb tests/kernel/threads/thread_apis/kernel.threads.apis PASSED (device 77.924s)

@finikorg
Copy link
Contributor Author

finikorg commented Feb 2, 2021

tested on ehl board, it could get coverage data, but for some testcases, the default timeout is not enough to get test results and all coverage data, it will timeout like below, the testcase is passed, but have not got all coverage data.

DEBUG - Timed out while monitoring serial output on ehl_crb
DEBUG - run status: ehl_crb/tests/kernel/threads/thread_apis/kernel.threads.apis passed
INFO - 1/1 ehl_crb tests/kernel/threads/thread_apis/kernel.threads.apis PASSED (device 77.924s)

Yes, I found that for some tests we need to increase timeout.

@jenmwms
Copy link
Contributor

jenmwms commented Feb 2, 2021

Yes, I found that for some tests we need to increase timeout.

How much increase is needed, and for which tests? It sounds like this patch introduces this timeout issue. In the past it was important to address root cause of any significant increase to timeout needed for tests to complete, rather than make a significant change for only 1 HW platform that will effect all platforms and arches. If it is minimal increase in timeout, is that a separate PR to address this?

@nashif
Copy link
Member

nashif commented Feb 3, 2021

How much increase is needed, and for which tests? It sounds like this patch introduces this timeout issue. In the past it was important to address root cause of any significant increase to timeout needed for tests to complete, rather than make a significant change for only 1 HW platform that will effect all platforms and arches. If it is minimal increase in timeout, is that a separate PR to address this?

where is the timeout being changed?

@nashif nashif added this to the v2.6.0 milestone Feb 3, 2021
@finikorg
Copy link
Contributor Author

finikorg commented Feb 3, 2021

Yes, I found that for some tests we need to increase timeout.

How much increase is needed, and for which tests? It sounds like this patch introduces this timeout issue. In the past it was important to address root cause of any significant increase to timeout needed for tests to complete, rather than make a significant change for only 1 HW platform that will effect all platforms and arches. If it is minimal increase in timeout, is that a separate PR to address this?

What I mean is that:

  • By default nothing actually changes.
  • If you enable --coverage then after test execution we get coverage data on console, in my case pseudo-console over network
  • I tested for ibecc tag and I am able to get coverage data for my tests but I believe if there are lots of coverage data it may take some time to receive them over console / network so we need to check timeout

@finikorg finikorg changed the title twister: Include coverage support for device testing RFC: twister: Include coverage support for device testing Feb 3, 2021
@nashif nashif merged commit 10fa48d into zephyrproject-rtos:master Feb 15, 2021
@finikorg finikorg deleted the coverage branch February 15, 2021 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nashif nashif nashif approved these changes

@aasthagr aasthagr Awaiting requested review from aasthagr

@jenmwms jenmwms Awaiting requested review from jenmwms

@jhedberg jhedberg Awaiting requested review from jhedberg

+3 more reviewers

@enjiamai enjiamai enjiamai approved these changes

@LeiW000 LeiW000 LeiW000 approved these changes

@chen-png chen-png chen-png approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@nashif nashif

Labels

area: Boards area: Twister Twister platform: X86 x86 and x86-64

Projects

None yet

Milestone

v2.6.0

Development

Successfully merging this pull request may close these issues.

7 participants

@finikorg @nashif @chen-png @jenmwms @enjiamai @LeiW000 @zephyrbot