Deployment error handling was reworked#272
Open
istalker2 wants to merge 4 commits intoMirantis:masterfrom
Open
Deployment error handling was reworked#272istalker2 wants to merge 4 commits intoMirantis:masterfrom
istalker2 wants to merge 4 commits intoMirantis:masterfrom
Conversation
istalker2
force-pushed
the
unblock-deployment
branch
from
a1923ba to
5730a94
Compare
added 4 commits
June 13, 2017 16:10
This change adds ability to replicate dependency with index parameters iterated over arbitrary number of lists. For each dependency it is now possible to specify map of indexVariableName -> listExpression listExpression := range|item + [, listExpression] range := number '..' number item := STRING for example, if for "i: 1..3" the dependency will be replicated into 3 clones, each one of them having argument i set to value in range [1, 3] This also allows to consume N flow replicas by replicating the dependency that leads to the consumed flow
For sequential flow, each next replica is attached to the leafs of previous one so that they will be deployed sequentially
* stopChan is now passed to the graph finalizers so that deployment can be canceled on the final stages * never write to stopChan. The only correct way to cancel deployment is to close the channel * pass nil instead of real chanel for unit tests that do not cancel deployment
In some cases deployment could hang forever: * if graph vertex depends on vertex which will never be created (because of timeout or permanent error) * if graph vertex was set to be created only if parent fails, but it didn't Because deployment algorithm waits for all vertexes to be created if any one of them remained blocked, Deploy() is going to run forever blocking AC process from handling other deployment tasks. Also on-error processing could be triggered by intermediate resource status. For example it could happen, if resource status was obtained prior to resource.Create() call. Another case if resource was set to have several deployment attempts. If the first attempt fails on-error dependency becomes activated, but on the second attempt the deployment may succeed. This commit reworks error handling: * Resources which cannot be created or time out, marked with error. * Resources, that depend on failed resources also fail * Thus all graph vertexes eventually become unblocked and deployment finishes * on-error handling is done based on the final resource status/error * Deploy() now returns true if deployment succeeded. Deployment fails if any of resources (and their dependents) went into failed state except for cases, where they were skipped because all dependencies had on-error meta and parent resource didn't fail Also: * e2e tests were updated so that most of them wait for deployment to finish rather than just waiting for resource status. Thus now they also test that deployment doesn't hang * Graph vertex type (ScheduledResource) and it fields are not exported anymore. The same goes for some of dependency graph methods. * wait() method doesn't create unnecessary goroutines and channels
istalker2
force-pushed
the
unblock-deployment
branch
from
5730a94 to
06a4167
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In some cases deployment could hang forever:
(because of timeout or permanent error)
didn't
Because deployment algorithm waits for all vertexes to be created if
any one of them remained blocked, Deploy() is going to run forever
blocking AC process from handling other deployment tasks.
Also on-error processing could be triggered by intermediate resource
status. For example it could happen, if resource status was obtained
prior to resource.Create() call. Another case if resource was set to
have several deployment attempts. If the first attempt fails on-error
dependency becomes activated, but on the second attempt the deployment
may succeed.
This commit reworks error handling:
finishes
if any of resources (and their dependents) went into failed state
except for cases, where they were skipped because all dependencies
had on-error meta and parent resource didn't fail
Also:
finish rather than just waiting for resource status. Thus now they
also test that deployment doesn't hang
anymore. The same goes for some of dependency graph methods.
This change is