Ordering across all nodes

[RollsChris]

Hi we could do with some advice on message ordering.

Our system requires that we output all messages in order, we have noticed recently that messages can be out of order across batches coming from the publisher.

I'm writing this while we try to understand it ourselves as we are in a time critical period.

So What is the general way people handle this? is there a way to force ordering from the publisher or what strategy should we employ on the consumer to re order the messages as they come through?

We could really do with some advice on this asap while we look into the issue.

[marcschier]

Hi @RollsChris, are you looking for ordering inside a writer or across writers? Inside a writer you should be able to enable SequentialPublishing to get publishing from the stack in sequence number order. But I am curious as to what you observed with the default behavior. We log sequence numbers arriving out of order, so something should be in your logs. If you use different subscriptions (e.g. publishing interval configuration) then no ordering can be guaranteed outside of the subscription context.

[RollsChris]

Hi, Thanks for getting back :)

I think we have one writer group and and writer. I have attached our publishednodes.json which doesn't specifically set the publish interval on any nodes.

publishednodes.json

Here are the settings we use to configure the publisher:

        "--autoaccept",
        "--trustmyself=true",
        "--diagnosticsinterval=60",
        "--publishfile=/publishervolume/publishednodes.json",
        "--createifnotexist",
        "--strict",
        "--messageencoding=Json",
        "--qos=AtLeastOnce",
        "--messagetimestamp=CurrentTimeUtc",
        "--sendkeepalives",
        "--defaultmessagetransport=IoTHub",
        "--iothubprotocol=Mqtt",
        "--enableprometheusendpoint=true",
        "--heartbeatbehavior=WatchdogLKVWithUpdatedTimestamps",
        "--opcsamplinginterval=1000",
        "--heartbeatinterval=20",
        "--dr"

I think with this setup we have one writer group and DateSetWrtiter. i.e. all on the same publishing interval?

So within one batch of messages they are ordered but if it spans multiple batch's they can go out of order.
I think you can get multiple batches even when the number of nodes exceeds the maximum for a group?

so I'm really not sure

Thanks

[marcschier]

Yes there should only be one writer, session, subscription etc. In your case.

If you use heartbeat, you can get heartbeats and messages delivered out of order, just because heartbeat uses lots of timers which are independent of the publishing loop. That is the current behavior, but it is a source of confusion and bugs so might need to address at some point)

Batches should be in order meaning you should not get earlier notifications in a later batch (with above exception, so your first test would be to disable heartbeat, which I in general recommend to do).

The stack orders notifications (--sequentialpublishing switch) which is true by default, although there could be something wrong there. Will look into it.

[RollsChris]

Hi thanks for responding,

i started performing some tests but got pulled onto something else. We do use the heart beat and it seems to be source of confusion.

For example we had it so that when the OPC server was down the publisher was still publishing messages but without any payload which caused us much confusion.

I will have to get back to you when i have more information.

Thanks