diff --git a/docs/shovel-dynamic.md b/docs/shovel-dynamic.md index 0f3e3f11b..c32e596ac 100644 --- a/docs/shovel-dynamic.md +++ b/docs/shovel-dynamic.md @@ -151,7 +151,7 @@ The body in this example includes a few keys: src-protocol Protocol to use when connecting to the source. - Either amqp091 or amqp10. If omitted it will default to amqp091. + Either amqp091, amqp10 or local. If omitted it will default to amqp091. See protocol specific properties below. @@ -794,6 +794,158 @@ counterparts. + +## Local Shovel Definition Reference {#local-reference} + +There are several Shovel properties that haven't been covered in the above example. +They don't change how dynamic shovels work fundamentally, and do not change +the declaration process. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Optional Dynamic Shovel Definition Settings (Local)
KeyDescription
reconnect-delay + The duration (in seconds) to wait before reconnecting to the + brokers after being disconnected at either end. Default is 1. +
ack-mode +

+ Determines how the shovel should acknowledge consumed messages. + Valid values are on-confirm, on-publish, and no-ack. + on-confirm is used by default. +

+

+ If set to on-confirm (the default), messages are + acknowledged to the source broker after they have been confirmed + by the destination. This handles network errors and broker + failures without losing messages, and is the slowest option. +

+

+ If set to on-publish, messages are acknowledged to + the source broker after they have been published at the + destination (but not yet confirmed). Messages may be lost in the event of network or broker failures. +

+

+ If set to no-ack, automatic message acknowledgements will be used. + This option will offer the highest throughput but is not safe (will lose messages in the event of network or broker failures). +

+
src-delete-after +

+ Determines when (if ever) the shovel should delete + itself. This can be useful if the shovel is being treated + as more of a move operation - i.e. being used to move + messages from one queue to another on an ad hoc basis. +

+

+ The default is never, meaning the + shovel should never delete itself. +

+

+ If set to queue-length then the shovel will + measure the length of the source queue when starting up, + and delete itself after it has transferred that many + messages. +

+

+ If set to an integer, then the shovel will transfer that + number of messages before deleting itself. +

+
src-prefetch-count + The maximum number of unacknowledged messages copied over a shovel at + any one time. Default is 1000. +
src-exchange +

+ The exchange from which to consume. Either this + or src-queue (but not both) must be set. +

+

+ The shovel will declare an exclusive queue and bind it to the + named exchange with src-exchange-key before consuming + from the queue. +

+

+ If the source exchange does not exist on the source broker, it + will be not declared; the shovel will fail to start. +

+
src-exchange-key + Routing key when using src-exchange. +
src-consumer-args + Consumer arguments, such as `x-single-active-consumer` or `x-stream-offset`. +
dest-exchange +

+ The exchange to which messages should be published. Either this + or dest-queue (but not both) may be set. +

+

+ If the destination exchange does not exist on the destination broker, + it will be not declared; the shovel will fail to start. +

+
dest-exchange-key + Routing key when using dest-exchange. If this is not + set, the original message's routing key will be used. +
dest-add-forward-headers + Whether to add x-opt-shovelled headers to the + shovelled messages indicating where they have been shovelled + from and to. Default is false. +
dest-add-timestamp-header + Whether to add x-opt-shovelled-timestamp headers to the + shovelled messages containing timestamp (in seconds since epoch) + when message had been shovelled. Default is false. +
+ + ## Monitoring Shovels {#status} See [Monitoring Shovels](./shovel#status) in the overview Shovel plugin guide. diff --git a/docs/shovel-static.md b/docs/shovel-static.md index 408929f73..33ab745e8 100644 --- a/docs/shovel-static.md +++ b/docs/shovel-static.md @@ -107,8 +107,8 @@ All the other properties are optional. `source` is a mandatory key and has different keys properties for different protocols. Two properties are common across all protocols: `protocol` and `uris`. -`protocol` supports two values: `amqp091` and `amqp10`, -for AMQP 0-9-1 and AMQP 1.0, respectively: +`protocol` supports three values: `amqp091`, `amqp10` and `local`, +for AMQP 0-9-1, AMQP 1.0 and local shovels respectively: ```erlang %% for AMQP 0-9-1 @@ -131,7 +131,7 @@ are available to static shovels, such as TLS certificate and private key. ### General Source Keys -Some keys are supported by both AMQP 0-9-1 and AMQP 1.0 sources. +Some keys are supported by AMQP 0-9-1, AMQP 1.0 and local sources. They are described in the table below. @@ -290,7 +290,7 @@ AMQP 0-9-1-specific source keys are covered in a separate table: most recently declared queue in declarations is used. This allows anonymous queues to be declared and used. - See also [Predeclared topology section](#predeclared-topology) below. + See also [Predeclared Topology section](#predeclared-topology) below.

@@ -363,13 +363,146 @@ AMQP 1.0 source settings are different from those of AMQP 0-9-1 sources.
+### Local Shovel Source Keys + +Local shovel's specific source keys are covered in a separate table: + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Local Shovel Source Keys (Properties)
KeyDescription
declarations +

+ An optional list of AMQP 0-9-1 operations to be executed by the Shovel + before it starts transferring messages. They are typically used to set + up the topology. +

+```erlang + {declarations, [ + %% declaration list + ]} +``` +

+ The declarations follow method and property names used by the RabbitMQ Erlang Client. +

+

+ A minimalistic declaration example: +

+```erlang + {declarations, [ + 'queue.declare', + {'queue.bind', [ + {exchange, <<"my_exchange">>}, + {queue, <<>>} + ]} + ]} +``` +

+ will first declare an anonymous queue, and then bind it + to the exchange called "my_exchange". The + queue name of <<>> on method queue.bind + means "use the queue last declared on this channel". +

+

+ Each element of the declaration list is either an AMQP 0-9-1 method + given as single quoted atom such as 'queue.declare', + or a tuple with first element the method atom, and second element + a property list of parameters. +

+

+ If just the method name is used all the + parameters take their defaults (as illustrated with + 'queue.declare' above). +

+

+ If a tuple and property-list is supplied, then the + properties in the list specify some or all of the + parameters explicitly. +

+

+ Here is another example: +

+```erlang +{'exchange.declare', [ + {exchange, <<"my_exchange">>}, + {type, <<"direct">>}, + durable + ]} +``` +

+ will declare a durable, direct exchange called + "my_exchange". +

+
queue +

+ The name of the source queue as an Erlang binary value. This property is mandatory: + +```erlang +{queue, <<"queue.1">>} +``` +

+

+ queue.1 is the name of the queue + to shovel messages from, as a binary string. +

+

+ This queue must exist. Use the resource declarations + covered above to declare the queue or ensure it exists. If + the value is <<>> (the empty binary string) then the + most recently declared queue in declarations is used. + This allows anonymous queues to be declared and used. + + See also [Predeclared topology section](#predeclared-topology) below. +

+
prefetch-count + The maximum number of unacknowledged messages copied over a shovel at + any one time. Default is 1000: + +```erlang +{prefetch_count, 1000} +``` +
+ +#### Predeclared Topology {#predeclared-topology} + +The `declarations` attribute is typically used to set up the topology. At the very least, it must set up the source queue. + +There are deployment scenarios where the topology is automatically [imported from a definitions file at boot time](./definitions#import-on-boot). In these scenarios, we can configure the plugin to wait until the queue is available by adding the following line to the `rabbitmq.conf` file: +```ini +shovel.topology.predeclared = true +``` + +With the above configuration, if a static shovel has no `declarations` attribute or it is empty, the piugin will wait until the source's `queue` is eventually declared. + ## Destination `destination` is a mandatory key and has different keys properties for different protocols. Two properties are common across all protocols: `protocol` and `uris`. -`protocol` supports two values: `amqp091` and `amqp10`, -for AMQP 0-9-1 and AMQP 1.0, respectively: +`protocol` supports three values: `amqp091`, `amqp10` and `local` +for AMQP 0-9-1, AMQP 1.0 and local shovels, respectively: ```erlang %% for AMQP 0-9-1 @@ -639,6 +772,47 @@ are available to static shovels, such as TLS certificate and private key. +### Local Shovel Destination Keys + + + + + + + + + + + + + + + + + + + + + + +
Local Shovel Destination Keys (Properties)
KeyDescription
add_timestamp_header + This boolean key controls whether a custom header, x-opt-shovelled-timestamp, + will be added to the message before it is re-published: + +```erlang +{add_timestamp_header, true} +``` + + This header value is timestamp (in seconds since epoch) when message had been shovelled. + By default the header is not added. +
add_forward_headers + When set to true the shovel will add a number of custom message headers: x-opt-shovelled-by, x-opt-shovel-type, x-opt-shovel-name, + to provide some additional metadata about the transfer. + +```erlang +{add_forward_headers, true} +``` +
## Example Configuration {#example-config} diff --git a/docs/shovel.md b/docs/shovel.md index 72ef89969..0e8cbe15e 100644 --- a/docs/shovel.md +++ b/docs/shovel.md @@ -56,6 +56,12 @@ and uses [acknowledgements](./confirms) on both ends to cope with failures. A Shovel uses [Erlang AMQP 0-9-1](/client-libraries/erlang-client-user-guide) and [Erlang AMQP 1.0](https://github.com/rabbitmq/rabbitmq-server/tree/main/deps/rabbitmq_amqp_client) clients under the hood. +RabbitMQ 4.2 introduced a new protocol type, the `local` shovels. Local shovels do not +use any of the existing protocols to connect to the source or destination, but +use an internal API to consume and/or publish directly from the queues in the local +cluster. They can only be used on the cluster the shovel is being declared. +A local shovel shares most features and configuration with AMQP 0-9-1 shovels. + ## Why Use Shovel {#benefits} Shovel is a minimalistic yet flexible tool in the distributed messaging toolkit @@ -207,7 +213,6 @@ what virtual host to connect to (assuming the target AMQP 1.0 broker is RabbitMQ See the [dynamic AMQP 1.0 Shovel reference](./shovel-dynamic#amqp10-reference) to learn more. - ## Authentication and authorisation for Shovels {#authn-authz-for-shovels} The plugin uses [Erlang AMQP 0-9-1](/client-libraries/erlang-client-user-guide) and [Erlang AMQP 1.0](https://github.com/rabbitmq/rabbitmq-server/tree/main/deps/rabbitmq_amqp_client) @@ -219,6 +224,7 @@ This is true for both sources and destinations. Authentication and authorisation failures of shovel connections will be [logged](./logging) by the node that's running the shovel. +Local shovels are also subject to the same authentication checks. ## Shovel Failure Handling in Clusters {#clustering} @@ -296,6 +302,7 @@ These examples use a URI with four additional [URI query parameters](./uri-query Just like with "regular" client connections, if TLS-enabled shovels need to perform peer verification then server's CA must be trusted on the node where shovels runs, and vice versa. +Local shovels do not use any connection, so no TLS is used. ## Monitoring Shovels {#status} diff --git a/versioned_docs/version-4.0/shovel-static.md b/versioned_docs/version-4.0/shovel-static.md index 408929f73..e776915bd 100644 --- a/versioned_docs/version-4.0/shovel-static.md +++ b/versioned_docs/version-4.0/shovel-static.md @@ -290,7 +290,7 @@ AMQP 0-9-1-specific source keys are covered in a separate table: most recently declared queue in declarations is used. This allows anonymous queues to be declared and used. - See also [Predeclared topology section](#predeclared-topology) below. + See also [Predeclared Topology section](#predeclared-topology) below.

@@ -309,7 +309,7 @@ AMQP 0-9-1-specific source keys are covered in a separate table: -#### Predeclared topology {#predeclared-topology} +#### Predeclared Topology {#predeclared-topology} The `declarations` attribute is typically used to set up the topology. At the very least, it must set up the source queue. diff --git a/versioned_docs/version-4.1/shovel-static.md b/versioned_docs/version-4.1/shovel-static.md index 408929f73..e776915bd 100644 --- a/versioned_docs/version-4.1/shovel-static.md +++ b/versioned_docs/version-4.1/shovel-static.md @@ -290,7 +290,7 @@ AMQP 0-9-1-specific source keys are covered in a separate table: most recently declared queue in declarations is used. This allows anonymous queues to be declared and used. - See also [Predeclared topology section](#predeclared-topology) below. + See also [Predeclared Topology section](#predeclared-topology) below.

@@ -309,7 +309,7 @@ AMQP 0-9-1-specific source keys are covered in a separate table: -#### Predeclared topology {#predeclared-topology} +#### Predeclared Topology {#predeclared-topology} The `declarations` attribute is typically used to set up the topology. At the very least, it must set up the source queue.