Allure.Reqnroll with MSTest | Not able to update test status in the -result.json file

[singh-swati28]

Hello,

In my C# Reqnroll + MSTest project, test scenario execution is being intentionally skipped based on some flag, for eg: execute test for US and skip using Assert.Inconclusive("This test is intentionally skipped for other countries") if any other country is specified in Feature file test data.
Since Asert.Inconclusive throws exception, the test is marked as broken when allure results files are generated.
In order to display intentionally failed tests as Skipped in Allure Report, I am using AllureLifecycle UpdateTestCase method to update Steps and overall Test status to Skipped

After the step is executed (where the test is being intentionally skipped), I am updating test status in AfterScenario hook using following code

AllureLifecycle.Instance.UpdateTestCase(testResult =>
{
    foreach (var x in testResult.steps)
    {
        if ((x.status.Equals(Status.failed) || x.status.Equals(Status.broken)) && x.statusDetails != null && !string.IsNullOrWhiteSpace(x.statusDetails.message) && (x.statusDetails.message.Contains("intentionally skipped") || x.statusDetails.message.Contains("ignored tests")))
        {
            x.status = Status.skipped;    
        }
    }
    testResult.status = Status.skipped;
    testResult.statusDetails.message = "intentionally skipped";
});

While debugging, I can see that the testResult.Status is updated to skipped in the Current TestContext .
However, when the execution is over and the -results.json is generated, only steps[].status got updated to 'skipped' and 'status' field is still set to 'broken'.

Can anyone please look into this and suggest the right way to update 'status' field?

Note: This is working fine with Specflow+Xunit/MSTest and Allure.Specflow plugin using AllureLifecycle.Instance.UpdateTestcase. In specflow, the status is set to broken initially and I was able to set it to skipped based on status details message but with Reqnroll and MSTest, initially the test status is none and then its getting overwritten )from skipped to broken) once the test exited the AfterScenario hook

@baev
@delatrie
@fescobar

[singh-swati28]

Hi @delatrie
Could you please look into this ?

[delatrie]

Hi, @singh-swati28
This is a bug, which we will eventually fix. See allure-framework/allure-csharp#557.

As for the ability to post-process Allure result objects, it will become possible as soon as allure-framework/allure-csharp#488 gets implemented.

Sorry for not giving specific dates

[singh-swati28]

Hi, @delatrie
Thank you for confirming that this is a bug and for planning to address it. Since, this is a priority for our project, could you kindly provide any estimated timeline for when the fix might be available? If there's a workaround in the meantime that I can implement to continue working, that would be greatly appreciated.
Additionally, please notify me once the bug #557 and feature #488 has been resolved, so that I can test the fix as soon as it's available.

Thank you for your help!

Best regards,
Swati