Skip to content

docs: document retry policy for network errors #150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mre wants to merge 5 commits into
base: master
Choose a base branch
from
Open

docs: document retry policy for network errors #150

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7a51218
docs: document retry policy for network errors
mre Mar 13, 2026
910d58b
refactor: replace HTML table + SVG icons with plain Markdown table
mre Mar 13, 2026
dfd44e8
refactor: restore original table from issue #149
mre Mar 13, 2026
fda8362
refactor: use SVG icons in table without text labels
mre Mar 13, 2026
161469d
Update src/content/docs/troubleshooting/network-errors.mdx
mre Mar 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 84 additions & 7 deletions src/content/docs/troubleshooting/network-errors.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,19 @@ title: Network Errors

import { Aside } from "@astrojs/starlight/components";

If you run lychee and some links fail then it could be certificate-related issue.
Certificates are used to verify the identity of a website and to establish a secure connection.
Different operating systems and tools have different ways to handle certificates.
export const Yes = () => <svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="#22c55e" strokeWidth="2.5" strokeLinecap="round" strokeLinejoin="round" aria-label="Yes"><polyline points="20 6 9 17 4 12"/></svg>;
export const No = () => <svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="#ef4444" strokeWidth="2.5" strokeLinecap="round" strokeLinejoin="round" aria-label="No"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>;

If you run lychee and some links fail, it could be a transient network issue or
a certificate-related problem. Certificates are used to verify the identity of a
website and to establish a secure connection. Different operating systems and
tools have different ways to handle certificates.

If lychee is unable to verify the certificate of a website, it will show a
network error. This could be due to an expired certificate, an invalid
certificate, or a certificate from an unknown authority.

Examples of network errors that could be certificate-related:
Examples of network errors:

```bash
Failed: Network error: error sending request for url
Expand All @@ -22,6 +26,77 @@ Failed: Network error: error sending request for url
Network error: Forbidden
```

```bash
Network error: Connection reset by server. Server forcibly closed connection
```

## Retry Policy

Lychee automatically retries failed requests before declaring a link broken.
Not every error is worth retrying &mdash; some are permanent, while others are
transient and may succeed on a second attempt.

The table below summarises which error types trigger a retry:

| Error Type | Retried? | Examples |
| ------------------------------- | -------------- | --------------------------------------------- |
| 5xx Server Errors | <Yes /> | 500, 502, 503, 504 |
| 408 Request Timeout | <Yes /> | Request took too long |
| 429 Too Many Requests | <Yes /> | Rate limit exceeded |
| Connection Timeout | <Yes /> | Server didn't respond in time |
| Connection Reset | <Yes /> | Connection dropped unexpectedly |
| Connection Aborted | <Yes /> | Connection terminated mid-request |
| Incomplete Message | <Yes /> | Response cut off before completion |
| 4xx Client Errors | <No /> | 400, 401, 403, 404 (except 408, 429) |
| 2xx Success | <No /> | 200, 201, 204 |
| 3xx Redirects | <No /> | 301, 302 |
| Initial Connection Failure | <No /> | Can't reach server at all |
Comment on lines +46 to +53
Copy link
Member

@katrinafyi katrinafyi Mar 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My networking knowledge is not so good. What's the difference between "Connection Timeout" and "Initial Connection Failure"? What would be an example of both?

Copy link
Member Author

@mre mre Mar 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The key difference is: timeout = eventually gave up waiting, connection failure = immediately told you "no."

Connection timeout means the TCP connection was attempted, and the server acknowledged it (or at least a packet was sent), but the server then stopped responding mid-way and the client gave up waiting. Example: you start a request to https://example.com, the TCP handshake begins, but the server is overloaded and never sends a response back. After a while, lychee gives up.

Initial connection failure means the connection was never established in the first place. For example when trying to connect to https://localhost:9999 where nothing is listening. The OS replies instantly with "connection refused". Or a DNS lookup that returns no result.

Does that answer your question or should I change the wording in the docs?

Copy link
Member

@katrinafyi katrinafyi Mar 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does help, thanks! Maybe just add these cases to the examples column for those rows. Calling out DNS specifically would help a lot.

| Certificate Issues | <No /> | SSL/TLS errors |
| Invalid Request Body | <No /> | Malformed data |
| Decoding Errors | <No /> | Can't parse response |
| Redirect Errors | <No /> | Too many redirects, etc. |

**As a rule of thumb:**
- **Retries:** Temporary problems (server down, network hiccup, timeout)
- **No Retry:** Permanent problems (bad request, auth failure, not found)

### Configuring retries

By default, lychee retries failed requests up to **3 times** with an
exponential backoff starting at **1 second** (i.e. 1 s, 2 s, 4 s).

You can tune this behaviour with two options:

| Option | CLI flag | Config key | Default |
|---|---|---|---|
| Maximum retries | `--max-retries <NUM>` | `max_retries` | `3` |
| Initial wait between retries | `--retry-wait-time <SECS>` | `retry_wait_time` | `1` |

For example, to retry up to 5 times with a 3-second initial wait:

```bash
lychee --max-retries 5 --retry-wait-time 3 https://example.com
```

Or in `lychee.toml`:

```toml
max_retries = 5
retry_wait_time = 3
```

To disable retries entirely, set `--max-retries 0`:

```bash
lychee --max-retries 0 https://example.com
```

<Aside>
Retries only apply to requests where lychee judges the failure to be
transient (see the table above). Permanent errors such as a `404 Not Found`
or a certificate problem are reported immediately without any retry.
</Aside>

## What now?

### Double-check with a different tool
Expand All @@ -36,7 +111,8 @@ If this works, then the issue is with lychee and not the website.
In that case, please open an issue on the lychee GitHub repository.

If it doesn't work, then the issue is with the website or your system.
It might be related to the certificate or the user agent. The site might also use a bot detection such as Cloudflare Bot Management. Read on to find out more.
It might be related to the certificate or the user agent. The site might also
use bot detection such as Cloudflare Bot Management. Read on to find out more.

### Use the `--insecure` flag

Expand Down Expand Up @@ -82,7 +158,7 @@ use a service like [BuiltWith](https://builtwith.com/). Just enter the website
URL and check if it uses any bot detection services like Cloudflare Bot Management or
Cloudflare Challenge.

If the website uses a bot detection service, which is blocking lychee, there's
If the website uses a bot detection service which is blocking lychee, there's
little you can do. You can try contacting the website administrator and ask them
to whitelist lychee.

Expand Down Expand Up @@ -113,4 +189,5 @@ Also, see this [Stack Overflow
discussion](https://stackoverflow.com/a/24618403/270334) for additional
information.

After that, try running lychee again and see if the issue is resolved. If not, please open an issue on the lychee GitHub repository.
After that, try running lychee again and see if the issue is resolved. If not,
please open an issue on the lychee GitHub repository.
Loading