Skip to content

Commit b6f080b

Browse files
Add content for automatic reruns (#142)
* add new page for automatic reruns * fixes from review * add basic entry to configuration reference * link to auto rerun docs from workflows page * Update docs/guides/modules/orchestrate/pages/automatic-reruns.adoc Co-authored-by: Sasha Lysenko <[email protected]> * Update docs/guides/modules/orchestrate/pages/automatic-reruns.adoc Co-authored-by: Sasha Lysenko <[email protected]> * add screenshot and clarify UI indicators * ui section again * minor edits * fix lint error * Update docs/guides/modules/orchestrate/pages/automatic-reruns.adoc * fixes from review * clarifications for limitations * remove line about scheduled workflows --------- Co-authored-by: Sasha Lysenko <[email protected]>
1 parent 491ea3d commit b6f080b

File tree

5 files changed

+261
-4
lines changed

5 files changed

+261
-4
lines changed
191 KB
Loading

docs/guides/modules/ROOT/nav.adoc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@
3838
*** xref:orchestrate:pipelines.adoc[Pipeline and triggers overview]
3939
*** xref:orchestrate:jobs-steps.adoc[Jobs and steps overview]
4040
*** xref:orchestrate:workflows.adoc[Workflow orchestration]
41+
*** xref:orchestrate:automatic-reruns.adoc[Automatic reruns]
4142
*** xref:orchestrate:workspaces.adoc[Use workspaces to share data between jobs]
4243
*** xref:orchestrate:dynamic-config.adoc[Use dynamic configuration]
4344
*** xref:orchestrate:skip-build.adoc[Skip CI and cancel workflows]
Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
= Automatic workflow reruns
2+
:page-platform: Cloud
3+
:page-description: Configure automatic reruns for failed workflows to reduce manual intervention and improve pipeline reliability
4+
:experimental:
5+
6+
Automatic reruns help reduce the impact of temporary failures in your CI/CD pipeline. When a workflow fails due to transient issues, CircleCI can automatically restart the failed workflow without manual intervention.
7+
8+
Common use cases include:
9+
10+
* Handling flaky tests.
11+
* Managing unreliable infrastructure or network connectivity problems.
12+
* Working with spot instances that may be terminated unexpectedly.
13+
* Preventing merge queue delays caused by transient failures.
14+
15+
[#introduction]
16+
== Introduction
17+
18+
Automatic reruns provide a safety net for your CI/CD pipelines by automatically retrying failed workflows. This feature helps teams maintain productivity by reducing the need for manual intervention when workflows fail due to temporary issues.
19+
20+
The feature works by monitoring workflow completion status and automatically triggering a "rerun from failed" operation when <<rerun-criteria,specific conditions are met>>. This ensures that only failed jobs and their dependencies are retried, while successful jobs from the original workflow are not repeated.
21+
22+
Benefits of automatic workflow reruns include:
23+
24+
* Reduced manual intervention for transient failures.
25+
* Improved pipeline reliability and developer productivity.
26+
* Cost-effective retry strategy that only reruns failed jobs.
27+
* Configurable retry limits to prevent infinite loops.
28+
29+
[#quickstart]
30+
== Quickstart
31+
32+
To enable automatic reruns for a workflow, add the `max_auto_reruns` key to your workflow with a value between 1 and 5:
33+
34+
.CircleCI config to automatically retry my-workflow up to 3 times if it fails
35+
[source,yaml]
36+
----
37+
version: 2.1
38+
39+
workflows:
40+
my-workflow:
41+
max_auto_reruns: 3
42+
jobs:
43+
- build
44+
- test
45+
- deploy:
46+
requires:
47+
- build
48+
- test
49+
----
50+
51+
You can combine automatic reruns with workflow conditions. The following example shows how to configure automatic reruns for a workflow that is not triggered by a scheduled pipeline:
52+
53+
.Combining automatic reruns with workflow conditions
54+
[source,yaml]
55+
----
56+
version: 2.1
57+
58+
workflows:
59+
test-workflow:
60+
when:
61+
not:
62+
equal: [ scheduled_pipeline, << pipeline.trigger_source >> ]
63+
max_auto_reruns: 4
64+
jobs:
65+
- build
66+
- test
67+
----
68+
69+
[#how-automatic-reruns-work]
70+
== How automatic reruns work
71+
72+
Automatic workflow reruns function similarly to manually clicking "Rerun workflow from failed" in the CircleCI interface. When a workflow fails, CircleCI evaluates whether the workflow qualifies for automatic rerun based on specific criteria, as described in the following section.
73+
74+
[#rerun-criteria]
75+
=== Rerun criteria
76+
77+
For automatic reruns to trigger, all of the following conditions must be met:
78+
79+
* The workflow status is "failed".
80+
* The `max_auto_reruns` value is specified in the configuration.
81+
* The number of remaining rerun attempts is greater than zero, and less than or equal to the specified `max_auto_reruns` value.
82+
* The workflow is not a manual rerun.
83+
* The pipeline is not older than 90 days.
84+
85+
[#rerun-behavior]
86+
=== Rerun behavior
87+
88+
When an automatic rerun is triggered:
89+
90+
* Only failed jobs from the original workflow are retried. If the previous failure blocked dependent jobs from running, these jobs are also run.
91+
* Successfully completed jobs are not rerun.
92+
* The rerun uses the same actor permissions as the original workflow.
93+
94+
[#monitoring-automatic-reruns]
95+
== Monitoring automatic reruns
96+
97+
CircleCI provides several ways to monitor and track automatic rerun activity.
98+
99+
[#ui-indicators]
100+
=== UI indicators
101+
102+
Automatic reruns are indicated on the pipelines page in the CircleCI web app. In the Trigger event column you sill see *Auto-rerun* followed by the rerun attempt number, as shown in the following screenshot.
103+
104+
In this example the workflow is rerun twice out of a possible five attempts before it succeeds.
105+
106+
image::guides:ROOT:orchestrate-and-trigger/automatic-rerun.png[Automatic reruns UI]
107+
108+
=== Get details via the API
109+
110+
You can retrieve information about automatic reruns using the link:https://circleci.com/docs/api/v2/index.html#tag/Workflow/operation/getWorkflowById[CircleCI API]:
111+
112+
[source,bash]
113+
----
114+
curl -X GET "https://circleci.com/api/v2/workflow/{workflow-id}" \
115+
-H "Circle-Token: YOUR_TOKEN"
116+
----
117+
118+
The API response includes additional fields for automatic reruns:
119+
120+
* `auto_rerun_number`: The current rerun attempt number.
121+
* `max_auto_reruns`: The maximum number of reruns configured.
122+
123+
[#limitations]
124+
== Limitations
125+
126+
Be aware of these limitations when using automatic workflow reruns:
127+
128+
* Maximum rerun attempts are capped at 5 per workflow.
129+
* Only the original workflow triggers automatic reruns. Manual reruns do not trigger automatic reruns.
130+
* Automatic reruns are disabled if the pipeline is older than 90 days.
131+
* Only failed workflows trigger automatic reruns, not cancelled workflows.
132+
133+
[#troubleshooting]
134+
== Troubleshooting
135+
136+
Common issues and solutions for automatic workflow reruns.
137+
138+
[#reruns-not-triggering]
139+
=== Reruns not triggering
140+
141+
If automatic reruns are not starting, check these conditions:
142+
143+
* Verify `max_auto_reruns` is specified in your workflow configuration.
144+
* Ensure the workflow status is "failed" and not "cancelled".
145+
* Confirm the maximum rerun attempts have not been exceeded.
146+
* Check that the workflow was not manually rerun.
147+
* Verify the pipeline is less than 90 days old.
148+
149+
[source,yaml]
150+
----
151+
# Correct configuration
152+
workflows:
153+
my-workflow:
154+
max_auto_reruns: 3 # Must be present
155+
jobs:
156+
- build
157+
----
158+
159+
[#excessive-reruns]
160+
=== Excessive rerun attempts
161+
162+
To prevent unnecessary reruns and credit consumption:
163+
164+
* Set conservative `max_auto_reruns` values based on your failure patterns.
165+
* Investigate recurring failures to address root causes.
166+
* Monitor rerun patterns to optimize configuration.
167+
168+
[#configuration-errors]
169+
=== Configuration errors
170+
171+
Common configuration mistakes include:
172+
173+
* Setting `max_auto_reruns` greater than 5 (results in configuration error).
174+
* Placing `max_auto_reruns` at the job level instead of workflow level.
175+
176+
[source,yaml]
177+
----
178+
# Incorrect - job level
179+
jobs:
180+
build:
181+
max_auto_reruns: 3 # Wrong placement
182+
docker:
183+
- image: cimg/base:2021.04
184+
185+
# Correct - workflow level
186+
workflows:
187+
my-workflow:
188+
max_auto_reruns: 3 # Correct placement
189+
jobs:
190+
- build
191+
----
192+
193+
[#frequently-asked-questions]
194+
== Frequently asked questions
195+
196+
[#faq-cost]
197+
=== Do automatic reruns consume additional credits?
198+
199+
Yes, automatic reruns consume compute credits for each retry attempt. Only failed jobs are rerun, so successful jobs from the original workflow do not consume additional credits.
200+
201+
[#faq-manual-rerun]
202+
=== What happens if I manually rerun a workflow?
203+
204+
If you manually rerun a workflow and it fails, no automatic reruns will be triggered for the manually rerun workflow.
205+
206+
[#faq-approval-jobs]
207+
=== Do automatic reruns work with approval jobs?
208+
209+
Yes.
210+
211+
[#faq-contexts]
212+
=== Do automatic reruns work with restricted contexts?
213+
214+
Yes, automatic reruns use the same actor permissions as the original workflow, so they work with restricted contexts as long as the original workflow had the necessary permissions.
215+
216+
[#faq-delay]
217+
=== Can I add a delay between automatic reruns?
218+
219+
Automatic reruns start immediately after the workflow fails.
220+
221+
[#faq-step-level]
222+
=== Can I configure automatic reruns at the step level?
223+
224+
NOTE: Step-level configuration for automatic reruns will be supported soon.
225+
226+
Automatic reruns are configured at the workflow level.

docs/guides/modules/orchestrate/pages/workflows.adoc

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -559,7 +559,7 @@ This example prevents the workflow `integration_tests` from running unless the `
559559
For more examples of using conditional logic in your workflows, see the xref:orchestration-cookbook.adoc[Orchestration cookbook].
560560

561561
****
562-
If you want to conditionally run workflows based on the type of event that triggers your pipeline, you can also consider splitting your workflows into separate YAML files and leveraging the different GitHub trigger event options.
562+
If you want to conditionally run workflows based on trigger event types, consider splitting your workflows into separate YAML files and leveraging GitHub trigger event options.
563563
564564
For further details see the following guides:
565565
@@ -795,7 +795,11 @@ For further information on workspaces and their configuration see the xref:works
795795
[#rerunning-a-workflows-failed-jobs]
796796
== Rerunning a workflow's failed jobs
797797

798-
Workflows help to speed up your ability to respond to failures. One way to do this is to only rerun failed jobs rather than a whole workflow. To rerun only a workflow's _failed_ jobs, follow these steps:
798+
Workflows help to speed up your ability to respond to failures. One way to do this is to only rerun failed jobs rather than a whole workflow.
799+
800+
The steps detailed in this section show how to manually rerun only a workflow's failed jobs. You can also set up automatic reruns for failed jobs. For more information, see the xref:automatic-reruns.adoc[Automatic Reruns] page.
801+
802+
To rerun only a workflow's failed jobs, follow these steps:
799803

800804
. In the link:https://app.circleci.com/home/[CircleCI web app] select your organization.
801805
. Select **Pipelines** in the sidebar.
@@ -804,12 +808,12 @@ Workflows help to speed up your ability to respond to failures. One way to do th
804808

805809
[tabs]
806810
====
807-
Rerun_from_the_pipelines_page::
811+
Rerun from the pipelines page::
808812
+
809813
--
810814
image::guides:ROOT:orchestrate-and-trigger/rerun-from-failed-pipelines-page.png[Rerun a workflow from failed from the pipelines page]
811815
--
812-
Rerun_from_the_workflows_page::
816+
Rerun from the workflows page::
813817
+
814818
--
815819
image::guides:ROOT:orchestrate-and-trigger/rerun-from-failed-workflows-page.png[Rerun a workflow from failed from the workflows page]

docs/reference/modules/ROOT/pages/configuration-reference.adoc

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2171,6 +2171,32 @@ workflows:
21712171
21722172
'''
21732173
2174+
==== *`max_auto_reruns`*
2175+
2176+
The `max_auto_reruns` key is used to configure the maximum number of automatic reruns for a workflow.
2177+
2178+
[,yml]
2179+
----
2180+
version: 2.1
2181+
2182+
workflows:
2183+
my-workflow:
2184+
max_auto_reruns: 3
2185+
jobs:
2186+
- build
2187+
- test
2188+
- deploy:
2189+
requires:
2190+
- build
2191+
- test
2192+
----
2193+
2194+
The value of `max_auto_reruns` can be an integer between 1 and 5.
2195+
2196+
For more information, see the xref:guides:orchestrate:automatic-reruns.adoc[Automatic Reruns] page.
2197+
2198+
'''
2199+
21742200
[#triggers]
21752201
==== *`triggers`*
21762202

0 commit comments

Comments
 (0)