Incorrect Success rate Calculation deducting SkippedScenario

[ymouhat]

Describe the Bug

Hello
it seems allure calculates the success rate = (SucceededScenario / (TotalScenario -SkippedScenario))
As a result we get different values than what is calculated in Azure devops & Allure report
( there is a sligth different in number of scenario counted by Azure & Allure because the QA used the same scenario name several times in one feature)

Example:

The tooltip has the right success rate calculated but not the main graph.

the same issue has been noticed using allure report & Jenkins pipelines

in our context, When test are executed, if an error is found in one scenario it skipped the next following scenarios of the given feature;
it has been activated with

cucumberOpts: [
failFast: true
}

Steps to Reproduce

Run several cucumber feature tests containing sevaral scenario that succeed, failed and skipped.
it seems allure calculates the success rate = 'SucceededScenario / (TotalScenario -SkippedScenario) )
SkiipedScenario shouldn't be deducted from the total number of scenario

Expected Behaviour

success rate = (SucceededScenario / TotalScenario)
it it doesn't fit everybody an option should be considered to take into account SkippedScenario in the calculation or not.

Screenshots or Additional Context

this issue has been discovered while using https://marketplace.visualstudio.com/items?itemName=qameta.allure-azure-pipelines&ssr=false#overview

What Language are you using?

TypeScript

What Framework/Allure Integration you are using?

wdio/allure-reporter": "8.24.0 / allure-commandline": "^2.29.0 /

What version of Allure Integration you are using?

2.29.0

What version of Allure Report you are using?

2.29.0

Code of Conduct

• I agree to follow this project's Code of Conduct

[baev]

It was like that in older versions, and lots of our users asked to exclude skipped tests from the success rate calculation. Actually, it was one of the most popular requests from the community.

Please see allure-framework/allure-java#412

[ymouhat]

HI @baev
Thank you for redirecting to the issue . I looked for an existing ticket but I may have missed it .
I agree this is the same issue .
however it seems the tooltip and the overal status doesn't show the same values (which doesn't seem correct)

The links you provided seems to indicate it is fixed with version 2.25.

Are we missing somehing ?

I made a quick test directly running the test locally to avoid any additional 3rd party component used in between .
And I reproduced the issue.

"@wdio/allure-reporter": "8.12.1"
"allure-commandline": "^2.29.0",
"webdriverio": "8.12.0",

Same issue noticed with

"@wdio/allure-reporter": "8.24.1"
"allure-commandline": "^2.29.0",
"webdriverio": "8.24.0",

[ymouhat]

Hi @baev
Could you please look at my answer
I don't get why the tooltip and the overal success rate are different
This is confusing :(

[ymouhat]

Hi @baev

I come back on this issue

• the passed scenario shows 94,56%
• The success rate shows 99,21%

Why both values are different ? while there showing the same thing % of scenario that passed.

From a quality perspective, to not consider the skipped scenario improve the success rate ratio but biase the reallity
It shows a success rate calculalted based on the scenario executed and not on the total of scenario

In my opinion we should have an option to choose whatever calculation is expected.

Can this improvment considered?

THank you in advance.