v20.0.0

[ghost]

Massively simplified & improved pub/sub

Pub/sub is now a lot more predictable and always guarantees strict ordering inside topics, across topics and between WebSocket::send and WebSocket::publish / App::publish. Subscription and unsubscription is always guaranteed to be strictly ordered with respect to publish. Support for MQTT wildcards has been removed.

Backwards compatible

Unless you're relying on MQTT wildcards ('#', '+'), v20.0.0 is entirely backwards compatible with v19.9.0.

Motivation

When pub/sub was introduced in 2018, it was designed based on a few assumptions:

• SHARED_COMPRESSOR being preferred by customers and customers being okay with supporting many application-level messages wrapped in and delivered as one bigger WebSocket message.
  • Expectation: Customers being willing to optimize the application protocol to minimize compression overhead.
  • Reality: This is way too complicated for customers and not even backwards compatible with their existing protocols. Customers simply want things to work, and they heavily prefer DEDICATED_COMPRESSOR.
• Subscription and unsubscription to/from topics being "malleable" and not strictly ordered.
  • Expectation: Customers being fine with subscriptions and unsubscriptions being executed in as big of a batch as possible, with minor re-ordering of messages not being a problem.
  • Reality: Many apps depend on a strict ordering guarantee and break down if order is not guaranteed. This is why the nonStrict hack was added, which performs really bad in worst-case and is badly understood.
• WebSocket::send and WebSocket::publish being two different streams out of order with each other.
  • Expectation: There being a separate use for pub/sub, not related to send.
  • Reality: Customers expect WebSocket::publish followed by WebSocket::send to be delivered in order, and the opposite likewise.
• MQTT syntax being useful
  • Expectation: Efficient use of a logarithmic tree of topics and wildcards
  • Reality: Customers don't care

---

This discussion was created from the release v20.0.0.

[toluca70]

We rely on the MQTT wildcards. Is the expectation that instead we need to subscribe on every possible topic that might match? The issue is the topic used to publish is dynamic and all possible topics are not known at time of subscription.

[e3dio]

I just updated some code, if you have sockets subscribed like this:

ws1.subscribe('topic/#')
ws2.subscribe('topic/123')

app.publish('topic/123', msg)

You can change that to:

ws1.subscribe('topic/#')
ws2.subscribe('topic/123')

app.publish('topic/#', msg)
app.publish('topic/123', msg)

I was thinking potentially add or change the API to allow publishing a message to multiple topics in 1 call, which avoids copying the message internally unnecessarily:

app.publish(msg, topic1, topic2, topic3 ..)

[e3dio]

It does seem simple to add a wildcard syntax back in if needed, 'topic/#' if the topic matches the wildcard publish to those subscribers

[toluca70]

Adding it back in would be appreciated if it's simple.

We are formatting the topic like this:
ws1.subscribe('property1/abc/property2/+/property3/#')

Then we publish like so:
app.publish('property1/abc/property2/123/property3/xxx', msg)

This is allowing a socket to control using a filter what properties and values it wants to listen to.

[e3dio]

You can replicate that with this publish function:

const publish=(properties,msg)=>getTopics(properties).forEach(t=>app.publish(t,msg))

const getTopics=properties=>{
   const topics=[]
   for(let p in properties){
      if(topics.length){
         topics.forEach(t=>topics.push(t))
         topics.forEach((t,i,a)=>a[i]+=`/${p}/${i<a.length/2?properties[p]:'+'}`)
      }else{
         topics.push(`${p}/${properties[p]}`,`${p}/+`)
      }
   }
   return topics
}

// usage
ws1.subscribe('property1/abc/property2/123/property3/xxx')
ws2.subscribe('property1/+/property2/123/property3/xxx')
ws3.subscribe('property1/+/property2/+/property3/+')

properties={property1:123,property2:'abc',proporty3:'xxx'} // any number of properties works

publish(properties,msg)

If the publish API is updated to allow publishing a message to multiple topics in 1 call it will improve performance:

app.publish(msg, topic1, topic2, topic3 ..)