Skip to content

Document local shovels protocol #2336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 24, 2025
Merged

Document local shovels protocol #2336

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
100991a
Document local shovels protocol
dcorbacho Aug 22, 2025
8b7c636
Cosmetics #2336
michaelklishin Aug 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 153 additions & 1 deletion docs/shovel-dynamic.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ The body in this example includes a few keys:
<td>src-protocol</td>
<td>
Protocol to use when connecting to the source.
Either <code>amqp091</code> or <code>amqp10</code>. If omitted it will default to <code>amqp091</code>.
Either <code>amqp091</code>, <code>amqp10</code> or <code>local</code>. If omitted it will default to <code>amqp091</code>.
See protocol specific properties below.
</td>
</tr>
Expand Down Expand Up @@ -794,6 +794,158 @@ counterparts.
</tbody>
</table>


## Local Shovel Definition Reference {#local-reference}

There are several Shovel properties that haven't been covered in the above example.
They don't change how dynamic shovels work fundamentally, and do not change
the declaration process.

<table>
<caption>Optional Dynamic Shovel Definition Settings (Local)</caption>

<thead>
<tr>
<td><strong>Key</strong></td>
<td><strong>Description</strong></td>
</tr>
</thead>

<tbody>
<tr>
<td>reconnect-delay</td>
<td>
The duration (in seconds) to wait before reconnecting to the
brokers after being disconnected at either end. Default is 1.
</td>
</tr>
<tr>
<td>ack-mode</td>
<td>
<p>
Determines how the shovel should <a href="./confirms">acknowledge</a> consumed messages.
Valid values are <code>on-confirm</code>, <code>on-publish</code>, and <code>no-ack</code>.
<code>on-confirm</code> is used by default.
</p>
<p>
If set to <code>on-confirm</code> (the default), messages are
<a href="./confirms">acknowledged</a> to the source broker after they have been confirmed
by the destination. This handles network errors and broker
failures without losing messages, and is the slowest option.
</p>
<p>
If set to <code>on-publish</code>, messages are <a href="./confirms">acknowledged</a> to
the source broker after they have been published at the
destination (but not yet confirmed). Messages may be lost in the event of network or broker failures.
</p>
<p>
If set to <code>no-ack</code>, <a href="./confirms">automatic message acknowledgements</a> will be used.
This option will offer the highest throughput but is not safe (will lose messages in the event of network or broker failures).
</p>
</td>
</tr>
<tr>
<td>src-delete-after</td>
<td>
<p>
Determines when (if ever) the shovel should delete
itself. This can be useful if the shovel is being treated
as more of a move operation - i.e. being used to move
messages from one queue to another on an ad hoc basis.
</p>
<p>
The default is <code>never</code>, meaning the
shovel should never delete itself.
</p>
<p>
If set to <code>queue-length</code> then the shovel will
measure the length of the source queue when starting up,
and delete itself after it has transferred that many
messages.
</p>
<p>
If set to an integer, then the shovel will transfer that
number of messages before deleting itself.
</p>
</td>
</tr>
<tr>
<td>src-prefetch-count</td>
<td>
The maximum number of unacknowledged messages copied over a shovel at
any one time. Default is <code>1000</code>.
</td>
</tr>
<tr>
<td>src-exchange</td>
<td>
<p>
The exchange from which to consume. Either this
or <code>src-queue</code> (but not both) must be set.
</p>
<p>
The shovel will declare an exclusive queue and bind it to the
named exchange with <code>src-exchange-key</code> before consuming
from the queue.
</p>
<p>
If the source exchange does not exist on the source broker, it
will be not declared; the shovel will fail to start.
</p>
</td>
</tr>
<tr>
<td>src-exchange-key</td>
<td>
Routing key when using <code>src-exchange</code>.
</td>
</tr>
<tr>
<td>src-consumer-args</td>
<td>
Consumer arguments, such as `x-single-active-consumer` or `x-stream-offset`.
</td>
</tr>
<tr>
<td>dest-exchange</td>
<td>
<p>
The exchange to which messages should be published. Either this
or <code>dest-queue</code> (but not both) may be set.
</p>
<p>
If the destination exchange does not exist on the destination broker,
it will be not declared; the shovel will fail to start.
</p>
</td>
</tr>
<tr>
<td>dest-exchange-key</td>
<td>
Routing key when using <code>dest-exchange</code>. If this is not
set, the original message's routing key will be used.
</td>
</tr>
<tr>
<td>dest-add-forward-headers</td>
<td>
Whether to add <code>x-opt-shovelled</code> headers to the
shovelled messages indicating where they have been shovelled
from and to. Default is false.
</td>
</tr>
<tr>
<td>dest-add-timestamp-header</td>
<td>
Whether to add <code>x-opt-shovelled-timestamp</code> headers to the
shovelled messages containing timestamp (in seconds since epoch)
when message had been shovelled. Default is false.
</td>
</tr>
</tbody>
</table>


## Monitoring Shovels {#status}

See [Monitoring Shovels](./shovel#status) in the overview Shovel plugin guide.
186 changes: 180 additions & 6 deletions docs/shovel-static.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,8 @@ All the other properties are optional.
`source` is a mandatory key and has different keys properties
for different protocols. Two properties are common across all
protocols: `protocol` and `uris`.
`protocol` supports two values: `amqp091` and `amqp10`,
for AMQP 0-9-1 and AMQP 1.0, respectively:
`protocol` supports three values: `amqp091`, `amqp10` and `local`,
for AMQP 0-9-1, AMQP 1.0 and local shovels respectively:

```erlang
%% for AMQP 0-9-1
Expand All @@ -131,7 +131,7 @@ are available to static shovels, such as TLS certificate and private key.

### General Source Keys

Some keys are supported by both AMQP 0-9-1 and AMQP 1.0 sources.
Some keys are supported by AMQP 0-9-1, AMQP 1.0 and local sources.
They are described in the table below.

<table>
Expand Down Expand Up @@ -290,7 +290,7 @@ AMQP 0-9-1-specific source keys are covered in a separate table:
<em>most recently declared queue</em> in <code>declarations</code> is used.
This allows anonymous queues to be declared and used.

See also [Predeclared topology section](#predeclared-topology) below.
See also [Predeclared Topology section](#predeclared-topology) below.
</p>
</td>
</tr>
Expand Down Expand Up @@ -363,13 +363,146 @@ AMQP 1.0 source settings are different from those of AMQP 0-9-1 sources.
</tbody>
</table>

### Local Shovel Source Keys

Local shovel's specific source keys are covered in a separate table:

<table>
<caption>Local Shovel Source Keys (Properties)</caption>

<thead>
<tr>
<td><strong>Key</strong></td>
<td><strong>Description</strong></td>
</tr>
</thead>

<tbody>
<tr>
<td>declarations</td>
<td>
<p>
An optional list of AMQP 0-9-1 operations to be executed by the Shovel
before it starts transferring messages. They are typically used to set
up the topology.
</p>
```erlang
{declarations, [
%% declaration list
]}
```
<p>
The declarations follow method and property names used by the <a href="/client-libraries/erlang-client-user-guide">RabbitMQ Erlang Client</a>.
</p>
<p>
A minimalistic declaration example:
</p>
```erlang
{declarations, [
'queue.declare',
{'queue.bind', [
{exchange, <<"my_exchange">>},
{queue, <<>>}
]}
]}
```
<p>
will first declare an anonymous queue, and then bind it
to the exchange called <code>"my_exchange"</code>. The
queue name of <code>&lt;&lt;>></code> on method <code>queue.bind</code>
means "use the queue last declared on this channel".
</p>
<p>
Each element of the declaration list is either an AMQP 0-9-1 method
given as single quoted atom such as <code>'queue.declare'</code>,
or a tuple with first element the method atom, and second element
a property list of parameters.
</p>
<p>
If just the method name is used all the
parameters take their defaults (as illustrated with
<code>'queue.declare'</code> above).
</p>
<p>
If a tuple and property-list is supplied, then the
properties in the list specify some or all of the
parameters explicitly.
</p>
<p>
Here is another example:
</p>
```erlang
{'exchange.declare', [
{exchange, <<"my_exchange">>},
{type, <<"direct">>},
durable
]}
```
<p>
will declare a durable, direct exchange called
"<code>my_exchange</code>".
</p>
</td>
</tr>

<tr>
<td>queue</td>
<td>
<p>
The name of the source queue as an Erlang binary value. This property is mandatory:

```erlang
{queue, <<"queue.1">>}
```
</p>
<p>
<code>queue.1</code> is the name of the queue
to shovel messages from, as a binary string.
</p>
<p>
This queue must exist. Use the resource <code>declarations</code>
covered above to declare the queue or ensure it exists. If
the value is <code>&lt;&lt;>></code> (the empty binary string) then the
<em>most recently declared queue</em> in <code>declarations</code> is used.
This allows anonymous queues to be declared and used.

See also [Predeclared topology section](#predeclared-topology) below.
</p>
</td>
</tr>

<tr>
<td>prefetch-count</td>
<td>
The maximum number of unacknowledged messages copied over a shovel at
any one time. Default is <code>1000</code>:

```erlang
{prefetch_count, 1000}
```
</td>
</tr>
</tbody>
</table>

#### Predeclared Topology {#predeclared-topology}

The `declarations` attribute is typically used to set up the topology. At the very least, it must set up the source queue.

There are deployment scenarios where the topology is automatically [imported from a definitions file at boot time](./definitions#import-on-boot). In these scenarios, we can configure the plugin to wait until the queue is available by adding the following line to the `rabbitmq.conf` file:
```ini
shovel.topology.predeclared = true
```

With the above configuration, if a static shovel has no `declarations` attribute or it is empty, the piugin will wait until the source's `queue` is eventually declared.

## Destination

`destination` is a mandatory key and has different keys properties
for different protocols. Two properties are common across all
protocols: `protocol` and `uris`.
`protocol` supports two values: `amqp091` and `amqp10`,
for AMQP 0-9-1 and AMQP 1.0, respectively:
`protocol` supports three values: `amqp091`, `amqp10` and `local`
for AMQP 0-9-1, AMQP 1.0 and local shovels, respectively:

```erlang
%% for AMQP 0-9-1
Expand Down Expand Up @@ -639,6 +772,47 @@ are available to static shovels, such as TLS certificate and private key.
</tbody>
</table>

### Local Shovel Destination Keys

<table>
<caption>Local Shovel Destination Keys (Properties)</caption>

<thead>
<tr>
<td><strong>Key</strong></td>
<td><strong>Description</strong></td>
</tr>
</thead>

<tbody>
<tr>
<td>add_timestamp_header</td>
<td>
This boolean key controls whether a custom header, <code>x-opt-shovelled-timestamp</code>,
will be added to the message before it is re-published:

```erlang
{add_timestamp_header, true}
```

This header value is timestamp (in seconds since epoch) when message had been shovelled.
By default the header is not added.
</td>
</tr>

<tr>
<td>add_forward_headers</td>
<td>
When set to true the shovel will add a number of custom message headers: <code>x-opt-shovelled-by</code>, <code>x-opt-shovel-type</code>, <code>x-opt-shovel-name</code>,
to provide some additional metadata about the transfer.

```erlang
{add_forward_headers, true}
```
</td>
</tr>
</tbody>
</table>

## Example Configuration {#example-config}

Expand Down
Loading