Replies: 1 comment 22 replies
-
|
According to the logs, this Shovel's TCP connection was closed. We do not have any RabbitMQ version information but for 3.12, I recall only one similar report that provided no details and was closed for inactivity. We could not reproduce this behavior with Toxiproxy several times, for example, in #3106. Please help us help you and provide:
|
Beta Was this translation helpful? Give feedback.
-
|
Again sorry for late reply here are the yaml files
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: rabbitmq-source
spec:
replicas: 1
override:
service:
spec:
ports:
- name: stream
protocol: TCP
port: 5552
targetPort: 5552
- name: mqtt
protocol: TCP
port: 1883
targetPort: 1883
statefulSet:
spec:
template:
spec:
containers:
- name: rabbitmq
ports:
- containerPort: 5552
name: stream
protocol: TCP
- containerPort: 1883
name: mqtt
protocol: TCP
rabbitmq:
additionalPlugins:
- rabbitmq_consistent_hash_exchange
- rabbitmq_management
- rabbitmq_management_agent
- rabbitmq_stream
- rabbitmq_stream_management
- rabbitmq_federation
- rabbitmq_federation_management
- rabbitmq_mqtt
- rabbitmq_web_mqtt
- rabbitmq_web_dispatch
- rabbitmq_peer_discovery_k8s
- rabbitmq_peer_discovery_common
- rabbitmq_prometheus
- rabbitmq_shovel
- rabbitmq_shovel_management
- rabbitmq_random_exchange
# persistence:
# storage: 40Gi
# storageClassName: local-path
resources:
limits:
cpu: 2000m
memory: 2000Mi
requests:
cpu: 1000m
memory: 500Mi
---
apiVersion: rabbitmq.com/v1beta1
kind: Exchange
metadata:
name: source.fanout
spec:
name: source.fanout # name of the exchange
vhost: test # default to '/' if not provided
type: fanout # default to 'direct' if not provided
autoDelete: false
durable: false
rabbitmqClusterReference:
name: rabbitmq-source # rabbitmqCluster must exist in the same namespace as this resource
---
apiVersion: v1
kind: Secret
metadata:
name: test-rabbitmq-source-creds
type: Opaque
stringData:
username: test # Note that Messaging Topology Operator does not watch this secret. Updating this secret object won't update actual user credentials.
password: test # As a workaround, you can add a label or annotation to the User object to trigger a Reconile loop and credentials will be updated.
---
apiVersion: rabbitmq.com/v1beta1
kind: User
metadata:
name: test-source
spec:
tags:
- administrator
rabbitmqClusterReference:
name: rabbitmq-source
importCredentialsSecret:
name: test-rabbitmq-source-creds
---
apiVersion: rabbitmq.com/v1beta1
kind: Permission
metadata:
name: test-source
spec:
vhost: test # name of a vhost
user: test # name of a RabbitMQ user
permissions:
write: .*
configure: .*
read: .*
rabbitmqClusterReference:
name: rabbitmq-source # rabbitmqCluster must exist in the same namespace as this resource
---
apiVersion: rabbitmq.com/v1beta1
kind: Vhost
metadata:
name: test-source
spec:
name: test # vhost name; required and cannot be updated
rabbitmqClusterReference:
name: rabbitmq-source # rabbitmqCluster must exist in the same namespace as this resource
---
apiVersion: v1
kind: Secret
metadata:
name: test-shovel-secret
type: Opaque
stringData:
srcUri: amqp://test@/test
destUri: amqp://test:test@rabbitmq-destination:5672/test
---
apiVersion: rabbitmq.com/v1beta1
kind: Shovel
metadata:
name: test.shovel
spec:
name: test.shovel
vhost: test
ackMode: no-ack
addForwardHeaders: true
uriSecret:
name: test-shovel-secret
srcExchange: source.fanout
srcProtocol: amqp091
destExchange: destination.fanout
destProtocol: amqp091
rabbitmqClusterReference:
name: rabbitmq-source
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
name: rabbitmq-destination
spec:
replicas: 1
override:
service:
spec:
ports:
- name: stream
protocol: TCP
port: 5552
targetPort: 5552
- name: mqtt
protocol: TCP
port: 1883
targetPort: 1883
statefulSet:
spec:
template:
spec:
containers:
- name: rabbitmq
ports:
- containerPort: 5552
name: stream
protocol: TCP
- containerPort: 1883
name: mqtt
protocol: TCP
rabbitmq:
additionalPlugins:
- rabbitmq_consistent_hash_exchange
- rabbitmq_management
- rabbitmq_management_agent
- rabbitmq_stream
- rabbitmq_stream_management
- rabbitmq_federation
- rabbitmq_federation_management
- rabbitmq_mqtt
- rabbitmq_web_mqtt
- rabbitmq_web_dispatch
- rabbitmq_peer_discovery_k8s
- rabbitmq_peer_discovery_common
- rabbitmq_prometheus
- rabbitmq_shovel
- rabbitmq_shovel_management
- rabbitmq_random_exchange
# persistence:
# storage: 40Gi
# storageClassName: local-path
resources:
limits:
cpu: 2000m
memory: 2000Mi
requests:
cpu: 1000m
memory: 500Mi
---
apiVersion: rabbitmq.com/v1beta1
kind: Exchange
metadata:
name: destination.fanout
spec:
name: destination.fanout # name of the exchange
vhost: test # default to '/' if not provided
type: fanout # default to 'direct' if not provided
autoDelete: false
durable: false
rabbitmqClusterReference:
name: rabbitmq-destination # rabbitmqCluster must exist in the same namespace as this resource
---
apiVersion: v1
kind: Secret
metadata:
name: test-rabbitmq-destination-creds
type: Opaque
stringData:
username: test # Note that Messaging Topology Operator does not watch this secret. Updating this secret object won't update actual user credentials.
password: test # As a workaround, you can add a label or annotation to the User object to trigger a Reconile loop and credentials will be updated.
---
apiVersion: rabbitmq.com/v1beta1
kind: User
metadata:
name: test-destination
spec:
tags:
- administrator
rabbitmqClusterReference:
name: rabbitmq-destination
importCredentialsSecret:
name: test-rabbitmq-destination-creds
---
apiVersion: rabbitmq.com/v1beta1
kind: Permission
metadata:
name: test-destination
spec:
vhost: test # name of a vhost
user: test # name of a RabbitMQ user
permissions:
write: .*
configure: .*
read: .*
rabbitmqClusterReference:
name: rabbitmq-destination # rabbitmqCluster must exist in the same namespace as this resource
---
apiVersion: rabbitmq.com/v1beta1
kind: Vhost
metadata:
name: test-destination
spec:
name: test # vhost name; required and cannot be updated
rabbitmqClusterReference:
name: rabbitmq-destination # rabbitmqCluster must exist in the same namespace as this resource |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, I will have a look at that. Just to confirm - you are indeed using single-node deployments on both side and the issue occurs in such a setup? (I was almost certain that clustering would have to be involved here...) By the way, there's a lot of unneeded stuff in there: The Operator sets up app the ports automatically when you enable the plugins |
Beta Was this translation helpful? Give feedback.
-
Yes, and also both rabbitmqs are on the same node so it is weird indeed.
Oh, I did not know that! Thanks for the heads-up |
Beta Was this translation helpful? Give feedback.
-
|
I've spent quite some time looking into the code and trying to reproduce this but I can't and I honestly don't see how it's possible that the queue exists without a binding. The operations to declare the queue and binding are next to each other What I'd like to suggest is to use Lastly, this led me to think about |
Beta Was this translation helpful? Give feedback.
-
IMHO it happens only when connection is lost. After the connection lost, maybe on initialization phase, things are not in a proper state. What those things could be I am not sure.
I can not think any reason for to use I will try |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the bug
We have multiple shovels that send data between two rabbitmqs. Periodically these shovels hang, meaning we don't see any publish on the receiver side, and they stay like that forever. When I check the statuses from the management UI, they are in the running state. When I manually restart shovels everything starts to work again.
I checked the logs of rabbitmq, you can find the latest logs below.
Reproduction steps
Expected behavior
Shovels should not hang
Additional context
rabbitmq latest logs
Beta Was this translation helpful? Give feedback.
All reactions