Skip to content

Commit 6d69408

Browse files
authored
Add detailed documentation for Transfers Poller/Receiver daemons (#552)
* Add detailed documentation for Transfers Poller/Receiver daemons #397 * some edits to make it clear at the top #397 * fts messaging link edit #397 * add activeMQ note, remove bold and change topic name #397
1 parent 376df36 commit 6d69408

File tree

2 files changed

+64
-1
lines changed

2 files changed

+64
-1
lines changed

docs/operator_transfers/transfers_overview.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,7 +126,8 @@ documentation.
126126
The path will be then submitted to the transfertool either in its integrity,
127127
if transfertool supports multi-hopping, or in multiple iterations.
128128

129-
The final steps are for the `receiver` or `poller` to monitor the transfer's
129+
The final steps are for the `receiver` or `poller` (refer to [Poller/Receiver](transfers_poller_receiver.md))
130+
to monitor the transfer's
130131
completion in transfertool and `finisher` to mark the transfers as completed.
131132
We only described here a simple case, when the transfer is successful on the
132133
first try. In case of errors, multiple transitions are possible between
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
---
2+
id: transfers_poller_receiver
3+
title: Transfers Poller/Receiver
4+
---
5+
6+
`conveyor-poller` and `conveyor-receiver` are daemons responsible for tracking transfer status and updating it in the system.
7+
**You need to run one or both of them** depending on the transfer tool you are using:
8+
9+
- If you are using only FTS3, it is **recommended to use the receiver** for scalability. Although poller also works.
10+
- If you are using only Globus and/or BitTorrent, you only need to run the poller.
11+
- If both FTS3 and Globus and/or BitTorrent, you need both receiver and poller.
12+
13+
## Poller
14+
15+
The `conveyor-poller` daemon periodically queries the transfer tools (such as FTS, Globus, or BitTorrent) directly to retrieve the status of ongoing transfers.
16+
Based on the results, it updates the state of each transfer in the system and/or refreshes the last access time.
17+
The transfer tool to query is taken from the transfer information internally and **no additional configuration is needed**.
18+
19+
**If using Globus or BitTorrent as the transfer tool, you need to use the poller.**
20+
21+
## Receiver
22+
23+
The `conveyor-receiver` daemon subscribes to an ActiveMQ message queue and continuously listens for transfer status updates.
24+
When it receives a message, it processes the update, changes the transfer's state accordingly, and/or updates the last access time.
25+
26+
To set up FTS to send transfer status updates to ActiveMQ broker, see the instructions [FTS3 Messaging Guide](https://fts3-docs.web.cern.ch/fts3-docs/docs/messaging.html).
27+
28+
> **Note:** The receiver connects to ActiveMQ using the **STOMP protocol**, so ensure your ActiveMQ instance has STOMP support enabled. As of this writing (May 2025), CERN FTS uses [ActiveMQ-Classic](https://activemq.apache.org/components/classic/) and
29+
not tested with [ActiveMQ-Artemis](https://activemq.apache.org/components/artemis/).
30+
31+
To configure `conveyor-receiver`, your rucio server config i.e. `rucio.cfg`, include the following section with described options:
32+
33+
```cfg
34+
[messaging-fts3]
35+
# Set to True to use SSL with certificate/key authentication, or False to use username/password
36+
use_ssl = True
37+
38+
# A comma-separated list of ActiveMQ broker DNS hostnames or aliases
39+
brokers = activemq1.example.org,activemq2.example.org
40+
41+
# Only required if use_ssl = False
42+
username = your-username
43+
password = your-password
44+
45+
# Port to use when NOT using SSL (i.e., use_ssl = False)
46+
nonssl_port = 61613
47+
48+
# Port to use when using SSL (i.e., use_ssl = True)
49+
port = 61617
50+
51+
# Required if use_ssl = True
52+
ssl_key_file = /path/to/hostkey.pem
53+
ssl_cert_file = /path/to/hostcert.pem
54+
55+
# The message topic or queue where FTS publishes transfer status (complete) updates.
56+
destination = /topic/transfer.fts_monitoring_complete
57+
58+
# Optional: virtual host name used to connect to the broker
59+
# This is only needed if your ActiveMQ setup requires a virtual host (e.g., in multi-tenant environments)
60+
broker_virtual_host = /atlas
61+
```
62+

0 commit comments

Comments
 (0)