|
| 1 | += Automatic workflow reruns |
| 2 | +:page-platform: Cloud |
| 3 | +:page-description: Configure automatic reruns for failed workflows to reduce manual intervention and improve pipeline reliability |
| 4 | +:experimental: |
| 5 | + |
| 6 | +Automatic reruns help reduce the impact of temporary failures in your CI/CD pipeline. When a workflow fails due to transient issues, CircleCI can automatically restart the failed workflow without manual intervention. |
| 7 | + |
| 8 | +Common use cases include: |
| 9 | + |
| 10 | +* Handling flaky tests. |
| 11 | +* Managing unreliable infrastructure or network connectivity problems. |
| 12 | +* Working with spot instances that may be terminated unexpectedly. |
| 13 | +* Preventing merge queue delays caused by transient failures. |
| 14 | +
|
| 15 | +[#introduction] |
| 16 | +== Introduction |
| 17 | + |
| 18 | +Automatic reruns provide a safety net for your CI/CD pipelines by automatically retrying failed workflows. This feature helps teams maintain productivity by reducing the need for manual intervention when workflows fail due to temporary issues. |
| 19 | + |
| 20 | +The feature works by monitoring workflow completion status and automatically triggering a "rerun from failed" operation when <<rerun-criteria,specific conditions are met>>. This ensures that only failed jobs and their dependencies are retried, while successful jobs from the original workflow are not repeated. |
| 21 | + |
| 22 | +Benefits of automatic workflow reruns include: |
| 23 | + |
| 24 | +* Reduced manual intervention for transient failures. |
| 25 | +* Improved pipeline reliability and developer productivity. |
| 26 | +* Cost-effective retry strategy that only reruns failed jobs. |
| 27 | +* Configurable retry limits to prevent infinite loops. |
| 28 | +
|
| 29 | +[#quickstart] |
| 30 | +== Quickstart |
| 31 | + |
| 32 | +To enable automatic reruns for a workflow, add the `max_auto_reruns` key to your workflow with a value between 1 and 5: |
| 33 | + |
| 34 | +.CircleCI config to automatically retry my-workflow up to 3 times if it fails |
| 35 | +[source,yaml] |
| 36 | +---- |
| 37 | +version: 2.1 |
| 38 | +
|
| 39 | +workflows: |
| 40 | + my-workflow: |
| 41 | + max_auto_reruns: 3 |
| 42 | + jobs: |
| 43 | + - build |
| 44 | + - test |
| 45 | + - deploy: |
| 46 | + requires: |
| 47 | + - build |
| 48 | + - test |
| 49 | +---- |
| 50 | + |
| 51 | +You can combine automatic reruns with workflow conditions. The following example shows how to configure automatic reruns for a workflow that is not triggered by a scheduled pipeline: |
| 52 | + |
| 53 | +.Combining automatic reruns with workflow conditions |
| 54 | +[source,yaml] |
| 55 | +---- |
| 56 | +version: 2.1 |
| 57 | +
|
| 58 | +workflows: |
| 59 | + test-workflow: |
| 60 | + when: |
| 61 | + not: |
| 62 | + equal: [ scheduled_pipeline, << pipeline.trigger_source >> ] |
| 63 | + max_auto_reruns: 4 |
| 64 | + jobs: |
| 65 | + - build |
| 66 | + - test |
| 67 | +---- |
| 68 | + |
| 69 | +[#how-automatic-reruns-work] |
| 70 | +== How automatic reruns work |
| 71 | + |
| 72 | +Automatic workflow reruns function similarly to manually clicking "Rerun workflow from failed" in the CircleCI interface. When a workflow fails, CircleCI evaluates whether the workflow qualifies for automatic rerun based on specific criteria, as described in the following section. |
| 73 | + |
| 74 | +[#rerun-criteria] |
| 75 | +=== Rerun criteria |
| 76 | + |
| 77 | +For automatic reruns to trigger, all of the following conditions must be met: |
| 78 | + |
| 79 | +* The workflow status is "failed". |
| 80 | +* The `max_auto_reruns` value is specified in the configuration. |
| 81 | +* The number of remaining rerun attempts is greater than zero, and less than or equal to the specified `max_auto_reruns` value. |
| 82 | +* The workflow is not a manual rerun. |
| 83 | +* The pipeline is not older than 90 days. |
| 84 | +
|
| 85 | +[#rerun-behavior] |
| 86 | +=== Rerun behavior |
| 87 | + |
| 88 | +When an automatic rerun is triggered: |
| 89 | + |
| 90 | +* Only failed jobs from the original workflow are retried. If the previous failure blocked dependent jobs from running, these jobs are also run. |
| 91 | +* Successfully completed jobs are not rerun. |
| 92 | +* The rerun uses the same actor permissions as the original workflow. |
| 93 | +
|
| 94 | +[#monitoring-automatic-reruns] |
| 95 | +== Monitoring automatic reruns |
| 96 | + |
| 97 | +CircleCI provides several ways to monitor and track automatic rerun activity. |
| 98 | + |
| 99 | +[#ui-indicators] |
| 100 | +=== UI indicators |
| 101 | + |
| 102 | +Automatic reruns are indicated on the pipelines page in the CircleCI web app. In the Trigger event column you sill see *Auto-rerun* followed by the rerun attempt number, as shown in the following screenshot. |
| 103 | + |
| 104 | +In this example the workflow is rerun twice out of a possible five attempts before it succeeds. |
| 105 | + |
| 106 | +image::guides:ROOT:orchestrate-and-trigger/automatic-rerun.png[Automatic reruns UI] |
| 107 | + |
| 108 | +=== Get details via the API |
| 109 | + |
| 110 | +You can retrieve information about automatic reruns using the link:https://circleci.com/docs/api/v2/index.html#tag/Workflow/operation/getWorkflowById[CircleCI API]: |
| 111 | + |
| 112 | +[source,bash] |
| 113 | +---- |
| 114 | +curl -X GET "https://circleci.com/api/v2/workflow/{workflow-id}" \ |
| 115 | + -H "Circle-Token: YOUR_TOKEN" |
| 116 | +---- |
| 117 | + |
| 118 | +The API response includes additional fields for automatic reruns: |
| 119 | + |
| 120 | +* `auto_rerun_number`: The current rerun attempt number. |
| 121 | +* `max_auto_reruns`: The maximum number of reruns configured. |
| 122 | + |
| 123 | +[#limitations] |
| 124 | +== Limitations |
| 125 | + |
| 126 | +Be aware of these limitations when using automatic workflow reruns: |
| 127 | + |
| 128 | +* Maximum rerun attempts are capped at 5 per workflow. |
| 129 | +* Only the original workflow triggers automatic reruns. Manual reruns do not trigger automatic reruns. |
| 130 | +* Automatic reruns are disabled if the pipeline is older than 90 days. |
| 131 | +* Only failed workflows trigger automatic reruns, not cancelled workflows. |
| 132 | + |
| 133 | +[#troubleshooting] |
| 134 | +== Troubleshooting |
| 135 | + |
| 136 | +Common issues and solutions for automatic workflow reruns. |
| 137 | + |
| 138 | +[#reruns-not-triggering] |
| 139 | +=== Reruns not triggering |
| 140 | + |
| 141 | +If automatic reruns are not starting, check these conditions: |
| 142 | + |
| 143 | +* Verify `max_auto_reruns` is specified in your workflow configuration. |
| 144 | +* Ensure the workflow status is "failed" and not "cancelled". |
| 145 | +* Confirm the maximum rerun attempts have not been exceeded. |
| 146 | +* Check that the workflow was not manually rerun. |
| 147 | +* Verify the pipeline is less than 90 days old. |
| 148 | + |
| 149 | +[source,yaml] |
| 150 | +---- |
| 151 | +# Correct configuration |
| 152 | +workflows: |
| 153 | + my-workflow: |
| 154 | + max_auto_reruns: 3 # Must be present |
| 155 | + jobs: |
| 156 | + - build |
| 157 | +---- |
| 158 | + |
| 159 | +[#excessive-reruns] |
| 160 | +=== Excessive rerun attempts |
| 161 | + |
| 162 | +To prevent unnecessary reruns and credit consumption: |
| 163 | + |
| 164 | +* Set conservative `max_auto_reruns` values based on your failure patterns. |
| 165 | +* Investigate recurring failures to address root causes. |
| 166 | +* Monitor rerun patterns to optimize configuration. |
| 167 | + |
| 168 | +[#configuration-errors] |
| 169 | +=== Configuration errors |
| 170 | + |
| 171 | +Common configuration mistakes include: |
| 172 | + |
| 173 | +* Setting `max_auto_reruns` greater than 5 (results in configuration error). |
| 174 | +* Placing `max_auto_reruns` at the job level instead of workflow level. |
| 175 | + |
| 176 | +[source,yaml] |
| 177 | +---- |
| 178 | +# Incorrect - job level |
| 179 | +jobs: |
| 180 | + build: |
| 181 | + max_auto_reruns: 3 # Wrong placement |
| 182 | + docker: |
| 183 | + - image: cimg/base:2021.04 |
| 184 | +
|
| 185 | +# Correct - workflow level |
| 186 | +workflows: |
| 187 | + my-workflow: |
| 188 | + max_auto_reruns: 3 # Correct placement |
| 189 | + jobs: |
| 190 | + - build |
| 191 | +---- |
| 192 | + |
| 193 | +[#frequently-asked-questions] |
| 194 | +== Frequently asked questions |
| 195 | + |
| 196 | +[#faq-cost] |
| 197 | +=== Do automatic reruns consume additional credits? |
| 198 | + |
| 199 | +Yes, automatic reruns consume compute credits for each retry attempt. Only failed jobs are rerun, so successful jobs from the original workflow do not consume additional credits. |
| 200 | + |
| 201 | +[#faq-manual-rerun] |
| 202 | +=== What happens if I manually rerun a workflow? |
| 203 | + |
| 204 | +If you manually rerun a workflow and it fails, no automatic reruns will be triggered for the manually rerun workflow. |
| 205 | + |
| 206 | +[#faq-approval-jobs] |
| 207 | +=== Do automatic reruns work with approval jobs? |
| 208 | + |
| 209 | +Yes. |
| 210 | + |
| 211 | +[#faq-contexts] |
| 212 | +=== Do automatic reruns work with restricted contexts? |
| 213 | + |
| 214 | +Yes, automatic reruns use the same actor permissions as the original workflow, so they work with restricted contexts as long as the original workflow had the necessary permissions. |
| 215 | + |
| 216 | +[#faq-delay] |
| 217 | +=== Can I add a delay between automatic reruns? |
| 218 | + |
| 219 | +Automatic reruns start immediately after the workflow fails. |
| 220 | + |
| 221 | +[#faq-step-level] |
| 222 | +=== Can I configure automatic reruns at the step level? |
| 223 | + |
| 224 | +NOTE: Step-level configuration for automatic reruns will be supported soon. |
| 225 | + |
| 226 | +Automatic reruns are configured at the workflow level. |
0 commit comments