Skip to content

Commit 618760f

Browse files
marwen-abidmarwen-abid
authored
SDP-1926: add Troubleshooting page (#2151)
* SDP-1926 add troubleshooting page * SDP-1926 add troubleshooting page - payment error * SDP-1926 add troubleshooting page - receiver not getting the invitation * SDP-1926 add troubleshooting page - receiver not getting otp * SDP-1926 add troubleshooting to routes.txt * SDP-1926 address comments * SDP-1926 add Slow Payments Section
1 parent 297ef57 commit 618760f

File tree

3 files changed

+300
-0
lines changed

3 files changed

+300
-0
lines changed

‎docs/platforms/stellar-disbursement-platform/admin-guide/troubleshooting.mdx‎

Lines changed: 299 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,299 @@
1+
---
2+
title: Troubleshooting
3+
sidebar_position: 90
4+
---
5+
6+
import Tabs from "@theme/Tabs";
7+
import TabItem from "@theme/TabItem";
8+
9+
This guide helps you diagnose and resolve common issues with the Stellar Disbursement Platform (SDP).
10+
11+
## Quick Reference
12+
13+
| Symptom | Likely Cause | Jump to |
14+
| --- | --- | --- |
15+
| Payment stuck in "Pending" | TSS issue, missing accounts, or insufficient funds | [Pending Payments](#payment-stuck-in-pending) |
16+
| "Resource Missing" in logs | Account doesn't exist on network | [Pending Payments](#payment-stuck-in-pending) |
17+
| Payment failed with `op_no_trust` | Receiver missing trustline for asset | [Operation Errors](#payment-failed-operation-error) |
18+
| Payment failed with `op_underfunded` | Distribution account low on funds | [Operation Errors](#payment-failed-operation-error) |
19+
| Payment failed with `op_no_destination` | Receiver account doesn't exist | [Operation Errors](#payment-failed-operation-error) |
20+
| Receiver didn't get invitation | Scheduler config or messaging provider issue | [Invitation Issues](#receiver-not-receiving-invitation) |
21+
| Receiver didn't get OTP | Mismatched contact info or provider issue | [OTP Issues](#receiver-not-receiving-otp) |
22+
| Channel account errors after testnet reset | Channel accounts were wiped | [Recreating Channel Accounts](#recreating-channel-accounts) |
23+
| Payments processing slowly | Not enough channel accounts | [Slow Payments](#slow-payments-due-to-insufficient-channel-accounts) |
24+
25+
---
26+
27+
## Payments
28+
29+
### Payment Stuck in "Pending" {#payment-stuck-in-pending}
30+
31+
Payments can get stuck in "Pending" status for several reasons. Work through these checks in order.
32+
33+
#### 1. Check TSS Service Health
34+
35+
The Transaction Submission Service (TSS) must be running and reachable.
36+
37+
```bash
38+
# Check TSS container status
39+
docker ps | grep tss
40+
41+
# View recent TSS logs
42+
docker logs --tail 100 <tss-container-name>
43+
```
44+
45+
If TSS is down or unreachable, restart it and monitor the logs for errors.
46+
47+
#### 2. Verify Distribution Account Funds
48+
49+
The distribution account must have sufficient XLM to cover the payment amount plus transaction fees.
50+
51+
Use [Stellar Expert](https://stellar.expert/explorer/public/) or the Stellar CLI to check the balance.
52+
53+
#### 3. Validate Channel Accounts
54+
55+
Channel accounts may become invalid after testnet resets. Look for errors like this in your logs:
56+
57+
<details>
58+
<summary>Example error: "Resource Missing"</summary>
59+
60+
```
61+
time="2025-12-19T18:43:37.017Z" level=error msg="[DRY_RUN Crash Reporter] unexpected TSS error: preparing bundle for processing: building transaction: horizon response error: getting account detail: horizon error: \"Resource Missing\" - check horizon.Error.Problem for more information" app_version=6.0.1 asset=XLM channel_account=GBKEVxxxx ...
62+
```
63+
64+
</details>
65+
66+
**Diagnosis:** This error means either the **destination account** or the **channel account** doesn't exist on the network. Check both using [Stellar Expert](https://stellar.expert/explorer/public/).
67+
68+
**Solution:**
69+
70+
<Tabs>
71+
<TabItem value="destination" label="Destination Account Missing">
72+
73+
The receiver's account hasn't been created on the Stellar network. The account must be funded with the minimum balance (currently 1 XLM on mainnet) before it can receive payments.
74+
75+
</TabItem>
76+
<TabItem value="channel" label="Channel Accounts Missing">
77+
78+
Channel accounts may disappear after testnet resets. See [Recreating Channel Accounts](#recreating-channel-accounts) below.
79+
80+
</TabItem>
81+
</Tabs>
82+
83+
---
84+
85+
### Payment Failed with Operation Error {#payment-failed-operation-error}
86+
87+
When a payment fails, the Status History shows a Horizon error with operation codes that explain why the transaction was rejected.
88+
89+
![Payment Failed Error](/assets/SDP/SDP45.png)
90+
91+
#### Reading the Error
92+
93+
Look for the `operation codes` at the end of the error message:
94+
95+
```
96+
Extras=transaction: tx_fee_bump_inner_failed - inner transaction: tx_failed - operation codes: [ op_no_trust ]
97+
```
98+
99+
The operation code (e.g., `op_no_trust`) tells you exactly what went wrong.
100+
101+
#### Common Operation Codes
102+
103+
| Code | Meaning | Solution |
104+
| --- | --- | --- |
105+
| `op_no_trust` | Receiver hasn't established a trustline for this asset | Receiver must add a trustline for the asset (e.g., EURC) before they can receive it |
106+
| `op_underfunded` | Source account doesn't have enough of the asset | Fund the distribution account with more of the asset |
107+
| `op_no_destination` | Destination account doesn't exist | Receiver must create and fund their Stellar account first |
108+
109+
:::info Trustlines explained
110+
111+
On Stellar, accounts must explicitly "trust" an asset before receiving it. This is a security feature—it prevents spam tokens. The receiver needs to add a trustline for the specific asset (like EURC) using their wallet or a Stellar tool.
112+
113+
:::
114+
115+
#### Example: `op_no_trust`
116+
117+
<details>
118+
<summary>Full error message</summary>
119+
120+
```
121+
horizon response error: StatusCode=400, Type=https://stellar.org/horizon-errors/transaction_failed,
122+
Title=Transaction Failed, Detail=The transaction failed when submitted to the stellar network.
123+
The `extras.result_codes` field on this response contains further details.
124+
Descriptions of each code can be found at: https://developers.stellar.org/docs/data/apis/horizon/api-reference/errors/http-status-codes/horizon-specific/transaction-failed/,
125+
Extras=transaction: tx_fee_bump_inner_failed - inner transaction: tx_failed - operation codes: [ op_no_trust ]
126+
```
127+
128+
</details>
129+
130+
**Diagnosis:** The receiver account exists but hasn't added a trustline for the asset you're trying to send (in this case, EURC).
131+
132+
**Solution:** The receiver must add a trustline for the asset before the payment can succeed. Once they've done so, use the **Retry** button in the dashboard to resubmit the payment.
133+
134+
For a complete list of operation result codes, see the [Stellar documentation](https://developers.stellar.org/docs/data/apis/horizon/api-reference/errors/result-codes/operation-specific/payment).
135+
136+
---
137+
138+
### Slow Payments Due to Insufficient Channel Accounts {#slow-payments-due-to-insufficient-channel-accounts}
139+
140+
If payments are processing slower than expected, you may not have enough channel accounts. Channel accounts allow the SDP to submit multiple transactions in parallel. Without enough of them, transactions queue up and process sequentially.
141+
142+
#### Symptoms
143+
144+
- Large disbursements take longer than expected to complete
145+
- Payments sit in "Pending" status longer than usual before being submitted
146+
147+
#### 1. Check Current Channel Account Count
148+
149+
View how many channel accounts are currently configured:
150+
151+
```bash
152+
./stellar-disbursement-platform channel-accounts view
153+
```
154+
155+
#### 2. Add More Channel Accounts
156+
157+
Use the `ensure` command to increase the number of channel accounts. This command is idempotent—it only creates new accounts if you have fewer than the specified number:
158+
159+
```bash
160+
# Ensure you have at least 10 channel accounts
161+
./stellar-disbursement-platform channel-accounts ensure 10
162+
```
163+
164+
:::tip How many channel accounts do you need?
165+
166+
The optimal number depends on your disbursement volume:
167+
168+
- **Low volume** (< 100 payments/day): 2–5 accounts
169+
- **Normal volume** (> 100 payments/day): 5–10 accounts
170+
171+
Start with a conservative number and increase if you notice slow processing times.
172+
173+
:::
174+
175+
#### 3. Verify TSS Configuration
176+
177+
The TSS service also has a configuration for how many channel accounts it should utilize. Check that your `--num-channel-accounts` flag (or `NUM_CHANNEL_ACCOUNTS` environment variable) matches or is less than the number of accounts you created:
178+
179+
```bash
180+
# In your TSS configuration
181+
--num-channel-accounts=10
182+
```
183+
184+
If this value is higher than the actual number of channel accounts available, TSS will only use what exists.
185+
186+
---
187+
188+
## Receiver Communications
189+
190+
Issues with invitations, OTPs, and other messages sent to receivers.
191+
192+
### Receiver Not Receiving the Invitation {#receiver-not-receiving-invitation}
193+
194+
When you trigger a disbursement targeting an unregistered receiver (via email, SMS, or WhatsApp), they should receive an invitation link to register. If they haven't received it, work through these checks.
195+
196+
#### 1. Verify Scheduler Configuration
197+
198+
The invitation job runs on a schedule controlled by an environment variable:
199+
200+
```bash
201+
SCHEDULER_RECEIVER_INVITATION_JOB_SECONDS=30
202+
```
203+
204+
**Check:** Is this set to a reasonable interval (10–60 seconds)? If it's set too high or missing, invitations may be significantly delayed.
205+
206+
#### 2. Check SDP Logs for Submission Failures {#check-logs-submission-failures}
207+
208+
The SDP logs will show whether the message was sent and if the messaging provider accepted or rejected it.
209+
210+
```bash
211+
# Look for messaging-related entries
212+
docker logs <sdp-container-name> 2>&1 | grep -iE "invitation|otp|message"
213+
```
214+
215+
**Common provider issues:**
216+
217+
| Provider | Typical Failure | What to Check |
218+
| --- | --- | --- |
219+
| AWS SES | Rate limiting, sandbox mode | Are you in production mode? Check sending limits in AWS console |
220+
| Twilio (SMS) | Geofencing, unverified numbers | Is the destination country enabled? Is your sender ID verified? |
221+
| Twilio (WhatsApp) | Template not approved, 24h window | Is your message template approved? Are you outside the 24h conversation window? |
222+
223+
:::tip
224+
225+
If you're testing, check spam/junk folders first—especially for email invitations.
226+
227+
:::
228+
229+
#### 3. Verify Receiver Contact Info
230+
231+
Double-check that the receiver's contact information (email, phone number) in the disbursement file is:
232+
233+
- Correctly formatted (e.g., phone numbers include country code)
234+
- Valid and reachable
235+
- Not a duplicate that was already processed
236+
237+
---
238+
239+
### Receiver Not Receiving OTP During Registration {#receiver-not-receiving-otp}
240+
241+
During registration, receivers enter the contact details (phone number or email) that the payer used when submitting the disbursement. The SDP sends an OTP to verify ownership of that contact method.
242+
243+
#### 1. Check for Mismatched Contact Info (Most Common)
244+
245+
The most frequent cause is the receiver entering a different email or phone number than what the payer submitted—often without realizing it.
246+
247+
**How to verify:** Check the `receiver_registration_attempts` table, which logs attempts from contacts that couldn't be matched to any receiver in the system.
248+
249+
```sql
250+
SELECT * FROM sdp_<tenant_name>.receiver_registration_attempts
251+
ORDER BY created_at DESC
252+
LIMIT 20;
253+
```
254+
255+
If you see the receiver's attempted contact info here, it means:
256+
257+
- They entered something different from what's on file
258+
- You may need to coordinate with the receiver to confirm which contact info is correct
259+
- If the payer made an error, you may need to update the receiver's contact info or create a new disbursement
260+
261+
#### 2. Check for Provider Issues
262+
263+
If the contact info matches but the OTP still isn't arriving, the issue is likely with the messaging provider. See [Check SDP Logs for Submission Failures](#check-logs-submission-failures) above for common provider issues and how to diagnose them.
264+
265+
---
266+
267+
## Channel Accounts
268+
269+
### Recreating Channel Accounts {#recreating-channel-accounts}
270+
271+
After a testnet reset, your channel accounts no longer exist on-chain but are still referenced in the database. You need to clean up invalid accounts and create new ones.
272+
273+
Run these commands inside the TSS container:
274+
275+
```bash
276+
# Step 1: Remove invalid accounts from the database
277+
./stellar-disbursement-platform channel-account verify --delete-invalid-accounts
278+
279+
# Step 2: Create new channel accounts (adjust the count as needed)
280+
./stellar-disbursement-platform channel-account ensure 10
281+
```
282+
283+
:::tip
284+
285+
The `ensure` command is idempotent—it only creates accounts if you have fewer than the specified number. Running `ensure 10` when you already have 10 valid accounts does nothing.
286+
287+
:::
288+
289+
---
290+
291+
## Still Stuck?
292+
293+
If you've worked through the relevant sections and the issue persists:
294+
295+
1. **Collect logs** from all relevant services (SDP, TSS, Anchor Platform)
296+
2. **Note the exact error message** and when it started occurring
297+
3. **Check for recent changes** to configuration, environment, or network (e.g., testnet reset)
298+
299+
Contact us either by opening an issue on our [Backend GitHub repository](https://github.com/stellar/stellar-disbursement-platform-backend/issues) or [Frontend GitHub repository](https://github.com/stellar/stellar-disbursement-platform-frontend/issues) with the details above and the version of SDP you're running. We'll help you troubleshoot further! We're also available on Discord in the [#bulk-disbursements](https://discord.com/channels/897514728459468821/1310800776331006002) channel.

‎routes.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -608,6 +608,7 @@
608608
/docs/platforms/stellar-disbursement-platform/admin-guide/monitoring
609609
/docs/platforms/stellar-disbursement-platform/admin-guide/overview
610610
/docs/platforms/stellar-disbursement-platform/admin-guide/security
611+
/docs/platforms/stellar-disbursement-platform/admin-guide/troubleshooting
611612
/docs/platforms/stellar-disbursement-platform/admin-guide/user-interface
612613
/docs/platforms/stellar-disbursement-platform/admin-guide/user-interface/analytics
613614
/docs/platforms/stellar-disbursement-platform/admin-guide/user-interface/circle-configuration

‎static/assets/SDP/SDP45.png‎

350 KB
Loading

0 commit comments

Comments
 (0)