AssertFailedException - JustMock xUnit method occurrence intermittently fails.

[adison88]

We are trying to run unit tests via xUnit and mocking method occurence using JustMockLite. Method under arrange is creating a underlying Task however, once in a while test fails with following error

build	11-May-2021 14:36:09	               Telerik.JustMock.Xunit.AssertFailedException : Multiple assertion failures:
build	11-May-2021 14:36:09	               1. Occurrence expectation failed. Expected exactly 1 call. Calls so far: 0
build	11-May-2021 14:36:09	         Arrange expression: x => x.CallAsync(IsAny(), IsAny())
build	11-May-2021 14:36:09	               2. Occurrence expectation failed. Expected exactly 1 call. Calls so far: 0
build	11-May-2021 14:36:09	         Arrange expression: x => x.NotifyAsync(IsAny())
build	11-May-2021 14:36:09	               
build	11-May-2021 14:36:09	               ---- Telerik.JustMock.Diagnostics.DebugViewDetailsException : State:
build	11-May-2021 14:36:09	         Elevated mocking: disabled
build	11-May-2021 14:36:09	         
build	11-May-2021 14:36:09	         Arrangements and expectations:
build	11-May-2021 14:36:09	                   Arrangement (id=0) x => x.Load(IsAny()):
build	11-May-2021 14:36:09	                 Met: Occurences must be in [1, 1]; calls so far: 1. 
build	11-May-2021 14:36:09	             Arrangement (id=1) x => x.InitializePolicy(IsAny(), IsAny()):
build	11-May-2021 14:36:09	                 Met: Occurences must be in [1, 1]; calls so far: 1. 
build	11-May-2021 14:36:09	             Arrangement (id=2) x => x.RequestBulkSync(IsAny(), IsAny(), IsAny()):
build	11-May-2021 14:36:09	                 Met: Occurences must be in [any, 1]; calls so far: 1. 
build	11-May-2021 14:36:09	             Arrangement (id=3) x => x.CallAsync(IsAny(), IsAny()):
build	11-May-2021 14:36:09	                 Unmet: Occurences must be in [1, 1]; calls so far: 0. 
build	11-May-2021 14:36:09	             Arrangement (id=4) x => x.NotifyAsync(IsAny()):
build	11-May-2021 14:36:09	                 Unmet: Occurences must be in [1, 1]; calls so far: 0. 
build	11-May-2021 14:36:09	         
build	11-May-2021 14:36:09	         Invocations:

[ivo-stoilov]

Hello @adison88, thanks for letting us know about this problem. I suppose that is somehow related to async processing, but I am not certain enough. Could you please give a short code snippet that reproduces the issue.

[adison88]

@ivo-stoilov Please refer to below snippet. Occurence for IBusiness4 fails with there's IBusiness3 method is being run as a part of another task.

[Fact]
        public void DoWorkTest()
        {
            using (var container = this.GetContainerInstance())
            {
                container.Arrange<IBusiness1>(x => x.Load(Arg.AnyInt))
                    .DoNothing().OccursOnce();
                container.Arrange<IBusiness2>(x => x.Load(Arg.AnyInt))
                    .DoNothing().OccursOnce();
                container.Arrange<IBusiness3>((x => x.Load(Arg.AnyInt))
                    .DoNothing().OccursAtMost(1);
                container.Arrange<IBusiness4>(x => x.Load(Arg.AnyInt))
                    .DoNothing().OccursOnce(); 
                // Act
                container.Instance.DoWork();
                // Assert
                container.AssertAll();
            }
        }
// Actual method
public void DoWork()
        {
            this.business1.Load(10);
            this.business2.Load(10);
            this.business3.Load(10);
            Task task = new Task(
                        delegate
                        {
                            try
                            {
                                this.business3.Load(10);
                            }
                            catch (Exception ex)
                            {
                                throw new Exception(ex.Message);
                            }
                        });
            task.Start();
            this.business4.Load(10);
        }

[ivo-stoilov]

Hello @adison88, I am unable to observe the issue using the latest JustMock NuGet (2021.2.511.1, see attached sample project). However, I meet another error:

Telerik.JustMock.XUnit.AssertFailedException : Occurrence expectation failed. Expected at most 1 call. Calls so far: 2
Arrange expression: x => x.Load(Arg.AnyInt)
---- Telerik.JustMock.Diagnostics.DebugViewDetailsException : State:
Elevated mocking: disabled

Arrangements and expectations:
    Arrangement (id=0) x => x.Load(Arg.AnyInt):
        Met: Occurences must be in [1, 1]; calls so far: 1. 
    Arrangement (id=1) x => x.Load(Arg.AnyInt):
        Met: Occurences must be in [1, 1]; calls so far: 1. 
    Arrangement (id=2) x => x.Load(Arg.AnyInt):
        Unmet: Occurences must be in [any, 1]; calls so far: 2. 
    Arrangement (id=3) x => x.Load(Arg.AnyInt):
        Met: Occurences must be in [1, 1]; calls so far: 1. 

Invocations:
    (ByRef (IBusiness1) Castle.Proxies.IBusiness1Proxy).Load((int) 10) called 1 time; (signature: Void Load(Int32))
    (ByRef (IBusiness2) Castle.Proxies.IBusiness2Proxy).Load((int) 10) called 1 time; (signature: Void Load(Int32))
    (ByRef (IBusiness3) Castle.Proxies.IBusiness3Proxy).Load((int) 10) called 1 time; (signature: Void Load(Int32))
    (ByRef (IBusiness4) Castle.Proxies.IBusiness4Proxy).Load((int) 10) called 1 time; (signature: Void Load(Int32))

It seems that the following arrangement fails:

container.Arrange<IBusiness3>(x => x.Load(Arg.AnyInt)) .DoNothing().OccursAtMost(1);

since the task is able to call IBusiness3.Load by a chance.

I will recommend you to enable debug trace for the failing test, rerun it and send us the produced output as a text file, Also it would be good to know which versions of JustMock and Xunit you are using.

I look forward to your reply, so we can plan further steps in resolving the issue.

[adison88]

@ivo-stoilov

Thanks for looking into it. I'll get you the logs soon once we reproduce the issue again on our side.

Versions that are in use -

JustMock - 2019.3.910.4
Xunit - 2.4.1
Xunit.Runner.VisualStudio - 2.3.1

[adison88]

@ivo-stoilov Please find the attached trace and let me know if anything else is needed.

DebugTrace.txt

[ivo-stoilov]

Hello @adison88, looking at the log file did not help so much, it looks like it was produced by the test with code, which does not match the snippet you have already sent, the method signatures of the arrangements are different (in bold):

Arrangements and expectations:
Arrangement (id=0) x => x.IBusiness1.Load(IsAny()):
Met: Occurences must be in [1, 1]; calls so far: 1.
Arrangement (id=1) x => x.IBusiness2.Load(IsAny(), IsAny()):
Met: Occurences must be in [1, 1]; calls so far: 1.
Arrangement (id=2) x => x.IBusiness3.Load(IsAny(), IsAny(), IsAny()):
Met: Occurences must be in [any, 1]; calls so far: 1.
Arrangement (id=3) x => x.IBusiness4.Load(IsAny(), IsAny()):
Unmet: Occurences must be in [1, 1]; calls so far: 0.

I also tried to experiment with the sample project from the previous message by downgrading the JustMock and Xunit.Runner.VisualStudio versions to the corresponding ones, but with no luck.

I am afraid that without stable reproduction it would be hard to identify the root cause of the issue. However, I encourage you to set up a minimal project (you can use sample one as a good starting point) that reproduces the problem. Any other relevant information that might help is also greatly appreciated.

[adison88]

@ivo-stoilov Could you please try with below project if you can reproduce it? Our builds run with more than 6000 unit tests and in parallel so it's little challenging for us to reproduce it locally as the failure occurred intermittently.

MySolution.zip

[ivo-stoilov]

Hello @adison88, I think that the key to resolving this issue is parallel execution. It might be a surprise for you, but there is a good reason that JustMock is not fully supporting parallelism, please take a look at the following explanation. You can try to disable parallel execution at the assembly level to see whether it would fix the issue. I agree that it is a kind of trade-off, but I hope that now you have a better understanding of the current framework limitations.