TestCluster: wait for cluster stabilization before starting tests by ReubenBond · Pull Request #4343 · dotnet/orleans

ReubenBond · 2018-03-28T23:35:38Z

This PR adds a check in TestCluster to verify that each silo has observed the expected number of active silos

ReubenBond · 2018-03-29T02:34:12Z

@dotnet-bot test this please

xiazen · 2018-03-29T21:53:57Z

System.InvalidOperationException : Cannot find generated GrainReference class for interface 'Orleans.Runtime.TestHooks.ITestHooksSystemTarget'\r\n--
seems like a valid test failure

xiazen · 2018-03-29T22:13:21Z

src/Orleans.TestingHost/TestCluster.cs

+
+                    if (totalWait.Elapsed < TimeSpan.FromSeconds(60))
+                    {
+                        WriteLog($"Warning! {remainingSilos} silos do not have a consistent cluster view after {totalWait.ElapsedMilliseconds}ms, continuing without stabilization.");


I think you may miss a break here. If while loop runs longer than a certain time, then break from the loop?

xiazen · 2018-03-29T22:16:07Z

src/Orleans.TestingHost/TestCluster.cs

+                // Poll each silo to check that it knows the expected number of active silos.
+                // If any silo does not have the expected number of active silos in its cluster membership oracle, try again.
+                // If the cluster membership has not stabilized after a certain period of time, give up and continue anyway.
+                var totalWait = Stopwatch.StartNew();


can you capsulate this section of code into a function WaitForClusterStabilizing or MakeSureAllSiloJoinedCluster method. Just to make it cleaner and more readable

sergeybykov · 2018-04-04T17:00:00Z

Two tests are failing consistently:

UnitTests.RuntimeCodeGenTests.RuntimeCodeGen_BasicEndToEnd

MESSAGE:
System.AggregateException : One or more errors occurred.\r\n---- System.InvalidOperationException : Cannot find generated GrainReference class for interface 'Orleans.Runtime.TestHooks.ITestHooksSystemTarget'\r\n---- The following constructor parameters did not have matching fixture data: Fixture fixture
+++++++++++++++++++
STACK TRACE:
----- Inner Stack Trace #1 (System.InvalidOperationException) ----- at Orleans.Runtime.TypeMetadataCache.GetGrainReferenceType(Type interfaceType) in D:\j\workspace\functional_pr---83d31c72\src\Orleans.Core\AssemblyLoader\TypeMetadataCache.cs:line 53 at Orleans.GrainFactory.MakeCaster(Type interfaceType) in D:\j\workspace\functional_pr---83d31c72\src\Orleans.Core\Core\GrainFactory.cs:line 252 at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory) at Orleans.GrainFactory.Cast(IAddressable grain, Type interfaceType) in D:\j\workspace\functional_pr---83d31c72\src\Orleans.Core\Core\GrainFactory.cs:line 239 at Orleans.GrainFactory.Cast[TGrainInterface](IAddressable grain) in D:\j\workspace\functional_pr---83d31c72\src\Orleans.Core\Core\GrainFactory.cs:line 223 at Orleans.GrainFactory.GetSystemTarget[TGrainInterface](GrainId grainId, SiloAddress destination) in D:\j\workspace\functional_pr---83d31c72\src\Orleans.Core\Core\GrainFactory.cs:line 286 at Orleans.TestingHost.TestCluster.d__36.MoveNext() in D:\j\workspace\functional_pr---83d31c72\src\Orleans.TestingHost\TestCluster.cs:line 205 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.TestingHost.TestCluster.Deploy() in D:\j\workspace\functional_pr---83d31c72\src\Orleans.TestingHost\TestCluster.cs:line 126 at TestExtensions.BaseTestClusterFixture..ctor() in D:\j\workspace\functional_pr---83d31c72\test\TestExtensions\BaseClusterFixture.cs:line 43 ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----

UnitTests.RuntimeCodeGenTests.RuntimeCodeGen_AddsSupportClasses

System.AggregateException : One or more errors occurred.\r\n---- System.InvalidOperationException : Cannot find generated GrainReference class for interface 'Orleans.Runtime.TestHooks.ITestHooksSystemTarget'\r\n---- The following constructor parameters did not have matching fixture data: Fixture fixture
+++++++++++++++++++
STACK TRACE:
----- Inner Stack Trace #1 (System.InvalidOperationException) ----- at Orleans.Runtime.TypeMetadataCache.GetGrainReferenceType(Type interfaceType) in D:\j\workspace\functional_pr---83d31c72\src\Orleans.Core\AssemblyLoader\TypeMetadataCache.cs:line 53 at Orleans.GrainFactory.MakeCaster(Type interfaceType) in D:\j\workspace\functional_pr---83d31c72\src\Orleans.Core\Core\GrainFactory.cs:line 252 at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory) at Orleans.GrainFactory.Cast(IAddressable grain, Type interfaceType) in D:\j\workspace\functional_pr---83d31c72\src\Orleans.Core\Core\GrainFactory.cs:line 239 at Orleans.GrainFactory.Cast[TGrainInterface](IAddressable grain) in D:\j\workspace\functional_pr---83d31c72\src\Orleans.Core\Core\GrainFactory.cs:line 223 at Orleans.GrainFactory.GetSystemTarget[TGrainInterface](GrainId grainId, SiloAddress destination) in D:\j\workspace\functional_pr---83d31c72\src\Orleans.Core\Core\GrainFactory.cs:line 286 at Orleans.TestingHost.TestCluster.d__36.MoveNext() in D:\j\workspace\functional_pr---83d31c72\src\Orleans.TestingHost\TestCluster.cs:line 205 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.TestingHost.TestCluster.Deploy() in D:\j\workspace\functional_pr---83d31c72\src\Orleans.TestingHost\TestCluster.cs:line 126 at TestExtensions.BaseTestClusterFixture..ctor() in D:\j\workspace\functional_pr---83d31c72\test\TestExtensions\BaseClusterFixture.cs:line 43 ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----

…t is accessible from TestCluster

ReubenBond · 2018-04-04T23:51:13Z

@dotnet-bot test bvt

We see this deps.json failure sometimes

…tnet#4343)

)

TestCluster: wait for cluster stabilization before starting tests

b7c8143

xiazen reviewed Mar 29, 2018

View reviewed changes

xiazen mentioned this pull request Apr 2, 2018

Intermittent failures of ControllableAdapter* tests #4220

Closed

ReubenBond added 2 commits April 5, 2018 09:01

Add Orleans.Runtime as a framework part so that ITestHooksSystemTarge…

b52e22e

…t is accessible from TestCluster

Extract WaitForInitialStabilization into its own method

cf2e403

Good point, Xiao

0fc4b8a

sergeybykov merged commit 868926f into dotnet:master Apr 5, 2018

ReubenBond deleted the test-wait-for-stabilization branch April 5, 2018 21:01

sergeybykov pushed a commit to sergeybykov/orleans that referenced this pull request May 5, 2018

TestCluster: wait for cluster stabilization before starting tests (do…

24d6bd5

…tnet#4343)

ReubenBond added a commit that referenced this pull request May 9, 2018

TestCluster: wait for cluster stabilization before starting tests (#4343

293b5db

)

github-actions bot locked and limited conversation to collaborators Dec 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TestCluster: wait for cluster stabilization before starting tests#4343

TestCluster: wait for cluster stabilization before starting tests#4343
sergeybykov merged 4 commits intodotnet:masterfrom
ReubenBond:test-wait-for-stabilization

ReubenBond commented Mar 28, 2018

Uh oh!

ReubenBond commented Mar 29, 2018

Uh oh!

xiazen commented Mar 29, 2018

Uh oh!

xiazen Mar 29, 2018 •

edited

Loading

Uh oh!

xiazen Mar 29, 2018

Uh oh!

sergeybykov commented Apr 4, 2018

Uh oh!

ReubenBond commented Apr 4, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ReubenBond commented Mar 28, 2018

Uh oh!

ReubenBond commented Mar 29, 2018

Uh oh!

xiazen commented Mar 29, 2018

Uh oh!

xiazen Mar 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xiazen Mar 29, 2018

Choose a reason for hiding this comment

Uh oh!

sergeybykov commented Apr 4, 2018

Uh oh!

ReubenBond commented Apr 4, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xiazen Mar 29, 2018 •

edited

Loading