Skip to content

Commit 9a3f62c

Browse files
authored
docs: add symptom-first troubleshooting hub and deep runbooks (openclaw#11196)
* docs(troubleshooting): add symptom-first troubleshooting runbooks * docs(troubleshooting): fix approvals command examples * docs(troubleshooting): wrap symptom cases in accordions * docs(automation): clarify userTimezone missing-key behavior * docs(troubleshooting): fix first-60-seconds ladder order
1 parent 1007d71 commit 9a3f62c

File tree

12 files changed

+825
-713
lines changed

12 files changed

+825
-713
lines changed

docs/automation/cron-jobs.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ the right time, and can optionally deliver output back to a chat.
1717
If you want _“run this every morning”_ or _“poke the agent in 20 minutes”_,
1818
cron is the mechanism.
1919

20+
Troubleshooting: [/automation/troubleshooting](/automation/troubleshooting)
21+
2022
## TL;DR
2123

2224
- Cron runs **inside the Gateway** (not inside the model).

docs/automation/troubleshooting.md

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
---
2+
summary: "Troubleshoot cron and heartbeat scheduling and delivery"
3+
read_when:
4+
- Cron did not run
5+
- Cron ran but no message was delivered
6+
- Heartbeat seems silent or skipped
7+
title: "Automation Troubleshooting"
8+
---
9+
10+
# Automation troubleshooting
11+
12+
Use this page for scheduler and delivery issues (`cron` + `heartbeat`).
13+
14+
## Command ladder
15+
16+
```bash
17+
openclaw status
18+
openclaw gateway status
19+
openclaw logs --follow
20+
openclaw doctor
21+
openclaw channels status --probe
22+
```
23+
24+
Then run automation checks:
25+
26+
```bash
27+
openclaw cron status
28+
openclaw cron list
29+
openclaw system heartbeat last
30+
```
31+
32+
## Cron not firing
33+
34+
```bash
35+
openclaw cron status
36+
openclaw cron list
37+
openclaw cron runs --id <jobId> --limit 20
38+
openclaw logs --follow
39+
```
40+
41+
Good output looks like:
42+
43+
- `cron status` reports enabled and a future `nextWakeAtMs`.
44+
- Job is enabled and has a valid schedule/timezone.
45+
- `cron runs` shows `ok` or explicit skip reason.
46+
47+
Common signatures:
48+
49+
- `cron: scheduler disabled; jobs will not run automatically` → cron disabled in config/env.
50+
- `cron: timer tick failed` → scheduler tick crashed; inspect surrounding stack/log context.
51+
- `reason: not-due` in run output → manual run called without `--force` and job not due yet.
52+
53+
## Cron fired but no delivery
54+
55+
```bash
56+
openclaw cron runs --id <jobId> --limit 20
57+
openclaw cron list
58+
openclaw channels status --probe
59+
openclaw logs --follow
60+
```
61+
62+
Good output looks like:
63+
64+
- Run status is `ok`.
65+
- Delivery mode/target are set for isolated jobs.
66+
- Channel probe reports target channel connected.
67+
68+
Common signatures:
69+
70+
- Run succeeded but delivery mode is `none` → no external message is expected.
71+
- Delivery target missing/invalid (`channel`/`to`) → run may succeed internally but skip outbound.
72+
- Channel auth errors (`unauthorized`, `missing_scope`, `Forbidden`) → delivery blocked by channel credentials/permissions.
73+
74+
## Heartbeat suppressed or skipped
75+
76+
```bash
77+
openclaw system heartbeat last
78+
openclaw logs --follow
79+
openclaw config get agents.defaults.heartbeat
80+
openclaw channels status --probe
81+
```
82+
83+
Good output looks like:
84+
85+
- Heartbeat enabled with non-zero interval.
86+
- Last heartbeat result is `ran` (or skip reason is understood).
87+
88+
Common signatures:
89+
90+
- `heartbeat skipped` with `reason=quiet-hours` → outside `activeHours`.
91+
- `requests-in-flight` → main lane busy; heartbeat deferred.
92+
- `empty-heartbeat-file``HEARTBEAT.md` exists but has no actionable content.
93+
- `alerts-disabled` → visibility settings suppress outbound heartbeat messages.
94+
95+
## Timezone and activeHours gotchas
96+
97+
```bash
98+
openclaw config get agents.defaults.heartbeat.activeHours
99+
openclaw config get agents.defaults.heartbeat.activeHours.timezone
100+
openclaw config get agents.defaults.userTimezone || echo "agents.defaults.userTimezone not set"
101+
openclaw cron list
102+
openclaw logs --follow
103+
```
104+
105+
Quick rules:
106+
107+
- `Config path not found: agents.defaults.userTimezone` means the key is unset; heartbeat falls back to host timezone (or `activeHours.timezone` if set).
108+
- Cron without `--tz` uses gateway host timezone.
109+
- Heartbeat `activeHours` uses configured timezone resolution (`user`, `local`, or explicit IANA tz).
110+
- ISO timestamps without timezone are treated as UTC for cron `at` schedules.
111+
112+
Common signatures:
113+
114+
- Jobs run at the wrong wall-clock time after host timezone changes.
115+
- Heartbeat always skipped during your daytime because `activeHours.timezone` is wrong.
116+
117+
Related:
118+
119+
- [/automation/cron-jobs](/automation/cron-jobs)
120+
- [/gateway/heartbeat](/gateway/heartbeat)
121+
- [/automation/cron-vs-heartbeat](/automation/cron-vs-heartbeat)
122+
- [/concepts/timezone](/concepts/timezone)

docs/channels/matrix.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,32 @@ Once verified, the bot can decrypt messages in encrypted rooms.
202202
| Location | ✅ Supported (geo URI; altitude ignored) |
203203
| Native commands | ✅ Supported |
204204

205+
## Troubleshooting
206+
207+
Run this ladder first:
208+
209+
```bash
210+
openclaw status
211+
openclaw gateway status
212+
openclaw logs --follow
213+
openclaw doctor
214+
openclaw channels status --probe
215+
```
216+
217+
Then confirm DM pairing state if needed:
218+
219+
```bash
220+
openclaw pairing list matrix
221+
```
222+
223+
Common failures:
224+
225+
- Logged in but room messages ignored: room blocked by `groupPolicy` or room allowlist.
226+
- DMs ignored: sender pending approval when `channels.matrix.dm.policy="pairing"`.
227+
- Encrypted rooms fail: crypto support or encryption settings mismatch.
228+
229+
For triage flow: [/channels/troubleshooting](/channels/troubleshooting).
230+
205231
## Configuration reference (Matrix)
206232

207233
Full configuration: [Configuration](/gateway/configuration)

docs/channels/signal.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,32 @@ Config:
168168
- Groups: `signal:group:<groupId>`.
169169
- Usernames: `username:<name>` (if supported by your Signal account).
170170

171+
## Troubleshooting
172+
173+
Run this ladder first:
174+
175+
```bash
176+
openclaw status
177+
openclaw gateway status
178+
openclaw logs --follow
179+
openclaw doctor
180+
openclaw channels status --probe
181+
```
182+
183+
Then confirm DM pairing state if needed:
184+
185+
```bash
186+
openclaw pairing list signal
187+
```
188+
189+
Common failures:
190+
191+
- Daemon reachable but no replies: verify account/daemon settings (`httpUrl`, `account`) and receive mode.
192+
- DMs ignored: sender is pending pairing approval.
193+
- Group messages ignored: group sender/mention gating blocks delivery.
194+
195+
For triage flow: [/channels/troubleshooting](/channels/troubleshooting).
196+
171197
## Configuration reference (Signal)
172198

173199
Full configuration: [Configuration](/gateway/configuration)

docs/channels/slack.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -537,6 +537,32 @@ Slack tool actions can be gated with `channels.slack.actions.*`:
537537
scopes you expect (`chat:write`, `reactions:write`, `pins:write`,
538538
`files:write`) or those operations will fail.
539539

540+
## Troubleshooting
541+
542+
Run this ladder first:
543+
544+
```bash
545+
openclaw status
546+
openclaw gateway status
547+
openclaw logs --follow
548+
openclaw doctor
549+
openclaw channels status --probe
550+
```
551+
552+
Then confirm DM pairing state if needed:
553+
554+
```bash
555+
openclaw pairing list slack
556+
```
557+
558+
Common failures:
559+
560+
- Connected but no channel replies: channel blocked by `groupPolicy` or not in `channels.slack.channels` allowlist.
561+
- DMs ignored: sender not approved when `channels.slack.dm.policy="pairing"`.
562+
- API errors (`missing_scope`, `not_in_channel`, auth failures): bot/app tokens or Slack scopes are incomplete.
563+
564+
For triage flow: [/channels/troubleshooting](/channels/troubleshooting).
565+
540566
## Notes
541567

542568
- Mention gating is controlled via `channels.slack.channels` (set `requireMention` to `true`); `agents.list[].groupChat.mentionPatterns` (or `messages.groupChat.mentionPatterns`) also count as mentions.

docs/channels/troubleshooting.md

Lines changed: 99 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,30 +1,116 @@
11
---
2-
summary: "Channel-specific troubleshooting shortcuts (Discord/Telegram/WhatsApp/iMessage)"
2+
summary: "Fast channel level troubleshooting with per channel failure signatures and fixes"
33
read_when:
4-
- A channel connects but messages don’t flow
5-
- Investigating channel misconfiguration (intents, permissions, privacy mode)
4+
- Channel transport says connected but replies fail
5+
- You need channel specific checks before deep provider docs
66
title: "Channel Troubleshooting"
77
---
88

99
# Channel troubleshooting
1010

11-
Start with:
11+
Use this page when a channel connects but behavior is wrong.
12+
13+
## Command ladder
14+
15+
Run these in order first:
1216

1317
```bash
18+
openclaw status
19+
openclaw gateway status
20+
openclaw logs --follow
1421
openclaw doctor
1522
openclaw channels status --probe
1623
```
1724

18-
`channels status --probe` prints warnings when it can detect common channel misconfigurations, and includes small live checks (credentials, some permissions/membership).
25+
Healthy baseline:
26+
27+
- `Runtime: running`
28+
- `RPC probe: ok`
29+
- Channel probe shows connected/ready
30+
31+
## WhatsApp
32+
33+
### WhatsApp failure signatures
34+
35+
| Symptom | Fastest check | Fix |
36+
| ------------------------------- | --------------------------------------------------- | ------------------------------------------------------- |
37+
| Connected but no DM replies | `openclaw pairing list whatsapp` | Approve sender or switch DM policy/allowlist. |
38+
| Group messages ignored | Check `requireMention` + mention patterns in config | Mention the bot or relax mention policy for that group. |
39+
| Random disconnect/relogin loops | `openclaw channels status --probe` + logs | Re-login and verify credentials directory is healthy. |
40+
41+
Full troubleshooting: [/channels/whatsapp#troubleshooting-quick](/channels/whatsapp#troubleshooting-quick)
42+
43+
## Telegram
44+
45+
### Telegram failure signatures
46+
47+
| Symptom | Fastest check | Fix |
48+
| --------------------------------- | ----------------------------------------------- | --------------------------------------------------------- |
49+
| `/start` but no usable reply flow | `openclaw pairing list telegram` | Approve pairing or change DM policy. |
50+
| Bot online but group stays silent | Verify mention requirement and bot privacy mode | Disable privacy mode for group visibility or mention bot. |
51+
| Send failures with network errors | Inspect logs for Telegram API call failures | Fix DNS/IPv6/proxy routing to `api.telegram.org`. |
52+
53+
Full troubleshooting: [/channels/telegram#troubleshooting](/channels/telegram#troubleshooting)
54+
55+
## Discord
56+
57+
### Discord failure signatures
58+
59+
| Symptom | Fastest check | Fix |
60+
| ------------------------------- | ----------------------------------- | --------------------------------------------------------- |
61+
| Bot online but no guild replies | `openclaw channels status --probe` | Allow guild/channel and verify message content intent. |
62+
| Group messages ignored | Check logs for mention gating drops | Mention bot or set guild/channel `requireMention: false`. |
63+
| DM replies missing | `openclaw pairing list discord` | Approve DM pairing or adjust DM policy. |
64+
65+
Full troubleshooting: [/channels/discord#troubleshooting](/channels/discord#troubleshooting)
66+
67+
## Slack
68+
69+
### Slack failure signatures
70+
71+
| Symptom | Fastest check | Fix |
72+
| -------------------------------------- | ----------------------------------------- | ------------------------------------------------- |
73+
| Socket mode connected but no responses | `openclaw channels status --probe` | Verify app token + bot token and required scopes. |
74+
| DMs blocked | `openclaw pairing list slack` | Approve pairing or relax DM policy. |
75+
| Channel message ignored | Check `groupPolicy` and channel allowlist | Allow the channel or switch policy to `open`. |
76+
77+
Full troubleshooting: [/channels/slack#troubleshooting](/channels/slack#troubleshooting)
78+
79+
## iMessage and BlueBubbles
80+
81+
### iMessage and BlueBubbles failure signatures
82+
83+
| Symptom | Fastest check | Fix |
84+
| -------------------------------- | ----------------------------------------------------------------------- | ----------------------------------------------------- |
85+
| No inbound events | Verify webhook/server reachability and app permissions | Fix webhook URL or BlueBubbles server state. |
86+
| Can send but no receive on macOS | Check macOS privacy permissions for Messages automation | Re-grant TCC permissions and restart channel process. |
87+
| DM sender blocked | `openclaw pairing list imessage` or `openclaw pairing list bluebubbles` | Approve pairing or update allowlist. |
88+
89+
Full troubleshooting:
90+
91+
- [/channels/imessage#troubleshooting-macos-privacy-and-security-tcc](/channels/imessage#troubleshooting-macos-privacy-and-security-tcc)
92+
- [/channels/bluebubbles#troubleshooting](/channels/bluebubbles#troubleshooting)
93+
94+
## Signal
95+
96+
### Signal failure signatures
97+
98+
| Symptom | Fastest check | Fix |
99+
| ------------------------------- | ------------------------------------------ | -------------------------------------------------------- |
100+
| Daemon reachable but bot silent | `openclaw channels status --probe` | Verify `signal-cli` daemon URL/account and receive mode. |
101+
| DM blocked | `openclaw pairing list signal` | Approve sender or adjust DM policy. |
102+
| Group replies do not trigger | Check group allowlist and mention patterns | Add sender/group or loosen gating. |
103+
104+
Full troubleshooting: [/channels/signal#troubleshooting](/channels/signal#troubleshooting)
19105

20-
## Channels
106+
## Matrix
21107

22-
- Discord: [/channels/discord#troubleshooting](/channels/discord#troubleshooting)
23-
- Telegram: [/channels/telegram#troubleshooting](/channels/telegram#troubleshooting)
24-
- WhatsApp: [/channels/whatsapp#troubleshooting-quick](/channels/whatsapp#troubleshooting-quick)
25-
- iMessage (legacy): [/channels/imessage#troubleshooting-macos-privacy-and-security-tcc](/channels/imessage#troubleshooting-macos-privacy-and-security-tcc)
108+
### Matrix failure signatures
26109

27-
## Telegram quick fixes
110+
| Symptom | Fastest check | Fix |
111+
| ----------------------------------- | -------------------------------------------- | ----------------------------------------------- |
112+
| Logged in but ignores room messages | `openclaw channels status --probe` | Check `groupPolicy` and room allowlist. |
113+
| DMs do not process | `openclaw pairing list matrix` | Approve sender or adjust DM policy. |
114+
| Encrypted rooms fail | Verify crypto module and encryption settings | Enable encryption support and rejoin/sync room. |
28115

29-
- Logs show `HttpError: Network request for 'sendMessage' failed` or `sendChatAction` → check IPv6 DNS. If `api.telegram.org` resolves to IPv6 first and the host lacks IPv6 egress, force IPv4 or enable IPv6. See [/channels/telegram#troubleshooting](/channels/telegram#troubleshooting).
30-
- Logs show `setMyCommands failed` → check outbound HTTPS and DNS reachability to `api.telegram.org` (common on locked-down VPS or proxies).
116+
Full troubleshooting: [/channels/matrix#troubleshooting](/channels/matrix#troubleshooting)

docs/docs.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -664,7 +664,7 @@
664664
},
665665
{
666666
"source": "/troubleshooting",
667-
"destination": "/gateway/troubleshooting"
667+
"destination": "/help/troubleshooting"
668668
},
669669
{
670670
"source": "/web/tui",
@@ -966,6 +966,7 @@
966966
"hooks/soul-evil",
967967
"automation/cron-jobs",
968968
"automation/cron-vs-heartbeat",
969+
"automation/troubleshooting",
969970
"automation/webhook",
970971
"automation/gmail-pubsub",
971972
"automation/poll",
@@ -976,6 +977,7 @@
976977
"group": "Media and devices",
977978
"pages": [
978979
"nodes/index",
980+
"nodes/troubleshooting",
979981
"nodes/images",
980982
"nodes/audio",
981983
"nodes/camera",

docs/gateway/heartbeat.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@ title: "Heartbeat"
1313
Heartbeat runs **periodic agent turns** in the main session so the model can
1414
surface anything that needs attention without spamming you.
1515

16+
Troubleshooting: [/automation/troubleshooting](/automation/troubleshooting)
17+
1618
## Quick start (beginner)
1719

1820
1. Leave heartbeats enabled (default is `30m`, or `1h` for Anthropic OAuth/setup-token) or set your own cadence.

0 commit comments

Comments
 (0)