Skip to content

Commit 6c4dd5f

Browse files
michaelklishinmichaelklishin
authored
Merge pull request #2336 from rabbitmq/local-shovels
Document local shovels protocol
2 parents f30e243 + 8b7c636 commit 6c4dd5f

File tree

5 files changed

+345
-12
lines changed

5 files changed

+345
-12
lines changed

‎docs/shovel-dynamic.md‎

Lines changed: 153 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -151,7 +151,7 @@ The body in this example includes a few keys:
151151
<td>src-protocol</td>
152152
<td>
153153
Protocol to use when connecting to the source.
154-
Either <code>amqp091</code> or <code>amqp10</code>. If omitted it will default to <code>amqp091</code>.
154+
Either <code>amqp091</code>, <code>amqp10</code> or <code>local</code>. If omitted it will default to <code>amqp091</code>.
155155
See protocol specific properties below.
156156
</td>
157157
</tr>
@@ -794,6 +794,158 @@ counterparts.
794794
</tbody>
795795
</table>
796796

797+
798+
## Local Shovel Definition Reference {#local-reference}
799+
800+
There are several Shovel properties that haven't been covered in the above example.
801+
They don't change how dynamic shovels work fundamentally, and do not change
802+
the declaration process.
803+
804+
<table>
805+
<caption>Optional Dynamic Shovel Definition Settings (Local)</caption>
806+
807+
<thead>
808+
<tr>
809+
<td><strong>Key</strong></td>
810+
<td><strong>Description</strong></td>
811+
</tr>
812+
</thead>
813+
814+
<tbody>
815+
<tr>
816+
<td>reconnect-delay</td>
817+
<td>
818+
The duration (in seconds) to wait before reconnecting to the
819+
brokers after being disconnected at either end. Default is 1.
820+
</td>
821+
</tr>
822+
<tr>
823+
<td>ack-mode</td>
824+
<td>
825+
<p>
826+
Determines how the shovel should <a href="./confirms">acknowledge</a> consumed messages.
827+
Valid values are <code>on-confirm</code>, <code>on-publish</code>, and <code>no-ack</code>.
828+
<code>on-confirm</code> is used by default.
829+
</p>
830+
<p>
831+
If set to <code>on-confirm</code> (the default), messages are
832+
<a href="./confirms">acknowledged</a> to the source broker after they have been confirmed
833+
by the destination. This handles network errors and broker
834+
failures without losing messages, and is the slowest option.
835+
</p>
836+
<p>
837+
If set to <code>on-publish</code>, messages are <a href="./confirms">acknowledged</a> to
838+
the source broker after they have been published at the
839+
destination (but not yet confirmed). Messages may be lost in the event of network or broker failures.
840+
</p>
841+
<p>
842+
If set to <code>no-ack</code>, <a href="./confirms">automatic message acknowledgements</a> will be used.
843+
This option will offer the highest throughput but is not safe (will lose messages in the event of network or broker failures).
844+
</p>
845+
</td>
846+
</tr>
847+
<tr>
848+
<td>src-delete-after</td>
849+
<td>
850+
<p>
851+
Determines when (if ever) the shovel should delete
852+
itself. This can be useful if the shovel is being treated
853+
as more of a move operation - i.e. being used to move
854+
messages from one queue to another on an ad hoc basis.
855+
</p>
856+
<p>
857+
The default is <code>never</code>, meaning the
858+
shovel should never delete itself.
859+
</p>
860+
<p>
861+
If set to <code>queue-length</code> then the shovel will
862+
measure the length of the source queue when starting up,
863+
and delete itself after it has transferred that many
864+
messages.
865+
</p>
866+
<p>
867+
If set to an integer, then the shovel will transfer that
868+
number of messages before deleting itself.
869+
</p>
870+
</td>
871+
</tr>
872+
<tr>
873+
<td>src-prefetch-count</td>
874+
<td>
875+
The maximum number of unacknowledged messages copied over a shovel at
876+
any one time. Default is <code>1000</code>.
877+
</td>
878+
</tr>
879+
<tr>
880+
<td>src-exchange</td>
881+
<td>
882+
<p>
883+
The exchange from which to consume. Either this
884+
or <code>src-queue</code> (but not both) must be set.
885+
</p>
886+
<p>
887+
The shovel will declare an exclusive queue and bind it to the
888+
named exchange with <code>src-exchange-key</code> before consuming
889+
from the queue.
890+
</p>
891+
<p>
892+
If the source exchange does not exist on the source broker, it
893+
will be not declared; the shovel will fail to start.
894+
</p>
895+
</td>
896+
</tr>
897+
<tr>
898+
<td>src-exchange-key</td>
899+
<td>
900+
Routing key when using <code>src-exchange</code>.
901+
</td>
902+
</tr>
903+
<tr>
904+
<td>src-consumer-args</td>
905+
<td>
906+
Consumer arguments, such as `x-single-active-consumer` or `x-stream-offset`.
907+
</td>
908+
</tr>
909+
<tr>
910+
<td>dest-exchange</td>
911+
<td>
912+
<p>
913+
The exchange to which messages should be published. Either this
914+
or <code>dest-queue</code> (but not both) may be set.
915+
</p>
916+
<p>
917+
If the destination exchange does not exist on the destination broker,
918+
it will be not declared; the shovel will fail to start.
919+
</p>
920+
</td>
921+
</tr>
922+
<tr>
923+
<td>dest-exchange-key</td>
924+
<td>
925+
Routing key when using <code>dest-exchange</code>. If this is not
926+
set, the original message's routing key will be used.
927+
</td>
928+
</tr>
929+
<tr>
930+
<td>dest-add-forward-headers</td>
931+
<td>
932+
Whether to add <code>x-opt-shovelled</code> headers to the
933+
shovelled messages indicating where they have been shovelled
934+
from and to. Default is false.
935+
</td>
936+
</tr>
937+
<tr>
938+
<td>dest-add-timestamp-header</td>
939+
<td>
940+
Whether to add <code>x-opt-shovelled-timestamp</code> headers to the
941+
shovelled messages containing timestamp (in seconds since epoch)
942+
when message had been shovelled. Default is false.
943+
</td>
944+
</tr>
945+
</tbody>
946+
</table>
947+
948+
797949
## Monitoring Shovels {#status}
798950

799951
See [Monitoring Shovels](./shovel#status) in the overview Shovel plugin guide.

‎docs/shovel-static.md‎

Lines changed: 180 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -107,8 +107,8 @@ All the other properties are optional.
107107
`source` is a mandatory key and has different keys properties
108108
for different protocols. Two properties are common across all
109109
protocols: `protocol` and `uris`.
110-
`protocol` supports two values: `amqp091` and `amqp10`,
111-
for AMQP 0-9-1 and AMQP 1.0, respectively:
110+
`protocol` supports three values: `amqp091`, `amqp10` and `local`,
111+
for AMQP 0-9-1, AMQP 1.0 and local shovels respectively:
112112

113113
```erlang
114114
%% for AMQP 0-9-1
@@ -131,7 +131,7 @@ are available to static shovels, such as TLS certificate and private key.
131131

132132
### General Source Keys
133133

134-
Some keys are supported by both AMQP 0-9-1 and AMQP 1.0 sources.
134+
Some keys are supported by AMQP 0-9-1, AMQP 1.0 and local sources.
135135
They are described in the table below.
136136

137137
<table>
@@ -290,7 +290,7 @@ AMQP 0-9-1-specific source keys are covered in a separate table:
290290
<em>most recently declared queue</em> in <code>declarations</code> is used.
291291
This allows anonymous queues to be declared and used.
292292

293-
See also [Predeclared topology section](#predeclared-topology) below.
293+
See also [Predeclared Topology section](#predeclared-topology) below.
294294
</p>
295295
</td>
296296
</tr>
@@ -363,13 +363,146 @@ AMQP 1.0 source settings are different from those of AMQP 0-9-1 sources.
363363
</tbody>
364364
</table>
365365

366+
### Local Shovel Source Keys
367+
368+
Local shovel's specific source keys are covered in a separate table:
369+
370+
<table>
371+
<caption>Local Shovel Source Keys (Properties)</caption>
372+
373+
<thead>
374+
<tr>
375+
<td><strong>Key</strong></td>
376+
<td><strong>Description</strong></td>
377+
</tr>
378+
</thead>
379+
380+
<tbody>
381+
<tr>
382+
<td>declarations</td>
383+
<td>
384+
<p>
385+
An optional list of AMQP 0-9-1 operations to be executed by the Shovel
386+
before it starts transferring messages. They are typically used to set
387+
up the topology.
388+
</p>
389+
```erlang
390+
{declarations, [
391+
%% declaration list
392+
]}
393+
```
394+
<p>
395+
The declarations follow method and property names used by the <a href="/client-libraries/erlang-client-user-guide">RabbitMQ Erlang Client</a>.
396+
</p>
397+
<p>
398+
A minimalistic declaration example:
399+
</p>
400+
```erlang
401+
{declarations, [
402+
'queue.declare',
403+
{'queue.bind', [
404+
{exchange, <<"my_exchange">>},
405+
{queue, <<>>}
406+
]}
407+
]}
408+
```
409+
<p>
410+
will first declare an anonymous queue, and then bind it
411+
to the exchange called <code>"my_exchange"</code>. The
412+
queue name of <code>&lt;&lt;>></code> on method <code>queue.bind</code>
413+
means "use the queue last declared on this channel".
414+
</p>
415+
<p>
416+
Each element of the declaration list is either an AMQP 0-9-1 method
417+
given as single quoted atom such as <code>'queue.declare'</code>,
418+
or a tuple with first element the method atom, and second element
419+
a property list of parameters.
420+
</p>
421+
<p>
422+
If just the method name is used all the
423+
parameters take their defaults (as illustrated with
424+
<code>'queue.declare'</code> above).
425+
</p>
426+
<p>
427+
If a tuple and property-list is supplied, then the
428+
properties in the list specify some or all of the
429+
parameters explicitly.
430+
</p>
431+
<p>
432+
Here is another example:
433+
</p>
434+
```erlang
435+
{'exchange.declare', [
436+
{exchange, <<"my_exchange">>},
437+
{type, <<"direct">>},
438+
durable
439+
]}
440+
```
441+
<p>
442+
will declare a durable, direct exchange called
443+
"<code>my_exchange</code>".
444+
</p>
445+
</td>
446+
</tr>
447+
448+
<tr>
449+
<td>queue</td>
450+
<td>
451+
<p>
452+
The name of the source queue as an Erlang binary value. This property is mandatory:
453+
454+
```erlang
455+
{queue, <<"queue.1">>}
456+
```
457+
</p>
458+
<p>
459+
<code>queue.1</code> is the name of the queue
460+
to shovel messages from, as a binary string.
461+
</p>
462+
<p>
463+
This queue must exist. Use the resource <code>declarations</code>
464+
covered above to declare the queue or ensure it exists. If
465+
the value is <code>&lt;&lt;>></code> (the empty binary string) then the
466+
<em>most recently declared queue</em> in <code>declarations</code> is used.
467+
This allows anonymous queues to be declared and used.
468+
469+
See also [Predeclared topology section](#predeclared-topology) below.
470+
</p>
471+
</td>
472+
</tr>
473+
474+
<tr>
475+
<td>prefetch-count</td>
476+
<td>
477+
The maximum number of unacknowledged messages copied over a shovel at
478+
any one time. Default is <code>1000</code>:
479+
480+
```erlang
481+
{prefetch_count, 1000}
482+
```
483+
</td>
484+
</tr>
485+
</tbody>
486+
</table>
487+
488+
#### Predeclared Topology {#predeclared-topology}
489+
490+
The `declarations` attribute is typically used to set up the topology. At the very least, it must set up the source queue.
491+
492+
There are deployment scenarios where the topology is automatically [imported from a definitions file at boot time](./definitions#import-on-boot). In these scenarios, we can configure the plugin to wait until the queue is available by adding the following line to the `rabbitmq.conf` file:
493+
```ini
494+
shovel.topology.predeclared = true
495+
```
496+
497+
With the above configuration, if a static shovel has no `declarations` attribute or it is empty, the piugin will wait until the source's `queue` is eventually declared.
498+
366499
## Destination
367500

368501
`destination` is a mandatory key and has different keys properties
369502
for different protocols. Two properties are common across all
370503
protocols: `protocol` and `uris`.
371-
`protocol` supports two values: `amqp091` and `amqp10`,
372-
for AMQP 0-9-1 and AMQP 1.0, respectively:
504+
`protocol` supports three values: `amqp091`, `amqp10` and `local`
505+
for AMQP 0-9-1, AMQP 1.0 and local shovels, respectively:
373506

374507
```erlang
375508
%% for AMQP 0-9-1
@@ -639,6 +772,47 @@ are available to static shovels, such as TLS certificate and private key.
639772
</tbody>
640773
</table>
641774

775+
### Local Shovel Destination Keys
776+
777+
<table>
778+
<caption>Local Shovel Destination Keys (Properties)</caption>
779+
780+
<thead>
781+
<tr>
782+
<td><strong>Key</strong></td>
783+
<td><strong>Description</strong></td>
784+
</tr>
785+
</thead>
786+
787+
<tbody>
788+
<tr>
789+
<td>add_timestamp_header</td>
790+
<td>
791+
This boolean key controls whether a custom header, <code>x-opt-shovelled-timestamp</code>,
792+
will be added to the message before it is re-published:
793+
794+
```erlang
795+
{add_timestamp_header, true}
796+
```
797+
798+
This header value is timestamp (in seconds since epoch) when message had been shovelled.
799+
By default the header is not added.
800+
</td>
801+
</tr>
802+
803+
<tr>
804+
<td>add_forward_headers</td>
805+
<td>
806+
When set to true the shovel will add a number of custom message headers: <code>x-opt-shovelled-by</code>, <code>x-opt-shovel-type</code>, <code>x-opt-shovel-name</code>,
807+
to provide some additional metadata about the transfer.
808+
809+
```erlang
810+
{add_forward_headers, true}
811+
```
812+
</td>
813+
</tr>
814+
</tbody>
815+
</table>
642816

643817
## Example Configuration {#example-config}
644818

0 commit comments

Comments
 (0)