message: Consider unsubscribed channels in reconcileMessages #1912

chrisbobbe · 2025-10-10T22:38:55Z

This fixes the "fourth buggy behavior" in #1798:
#1798 (comment)

Fixes-partly: #1798

chrisbobbe · 2025-10-10T23:50:15Z

(Oops, I'll need to rethink this a bit following some discussion with Greg in the office.)

gnprice

Thanks! At #1798 (comment) you asked for an initial look at this logic, so comments on that below.

gnprice · 2025-10-16T00:44:43Z

lib/model/message.dart

    // If the message is one we already know about (from a fetch),
    // clobber it with the one from the event system.
    // See [fetchedMessages] for reasoning.


(Is this existing comment meant to refer to reconcileMessages? I think fetchedMessages was a previous name for that.)

gnprice · 2025-10-16T00:46:29Z

lib/model/message.dart

+  /// (We have seen a few such events, actually --
+  /// maybe because the channel was recently subscribed? --
+  /// but not consistently, and we're not supposed to rely on them.)


Suggested change

/// (We have seen a few such events, actually --

/// maybe because the channel was recently subscribed? --

/// but not consistently, and we're not supposed to rely on them.)

/// (We have seen a few such events, actually --

/// maybe because the channel was recently unsubscribed? --

/// but not consistently, and we're not supposed to rely on them.)

?

gnprice · 2025-10-16T00:48:33Z

lib/model/message.dart

+    // [1] With one exception: if the version we have was in an unsubscribed
+    //     channel when we got it or sometime since, we take the fetched version


I like this formulation of which messages go in _maybeStaleChannelMessages. It'd be good to put it in that field's doc.

gnprice · 2025-10-16T00:55:16Z

lib/model/message.dart

+        // Side effect: update our "potentially stale" knowledge based on
+        // whether the channel is subscribed right now.
+        && (subscriptions[message.streamId] != null
+            ? _maybeStaleChannelMessages.remove(message.id)
+            : !_maybeStaleChannelMessages.add(message.id));


It's a bit confusing that we need this side effect on the maybe-stale set here. I think this logic could benefit from some unpacking.

In particular, it seems like there are really four cases here:

currently subscribed vs. not;

for each of those, currently in the maybe-stale set or not.

Thanks to Greg for pointing this out: zulip#1912 (comment)

chrisbobbe · 2025-10-16T21:04:40Z

Thanks for taking a look! Revision pushed.

gnprice

Thanks! I think that's helpful. A couple of comments below on the new version.

gnprice · 2025-10-17T00:14:00Z

lib/model/message.dart

+  Message _stripMatchFields(Message message) {
+    message.matchContent = null;
+    message.matchTopic = null;
+    return message;
  }


The factoring out of this step could happen in a prep commit. That would simplify the complexity of the main commit a bit — it'd be moving around where an expression like _stripMatchFields(incoming) appears, and adding a case where something else gets used instead, but the expression itself would remain the same.

(Or maybe the main commit would change from "message" to "incoming", but anyway that's a rather smaller change.)

gnprice · 2025-10-17T00:17:20Z

lib/model/message.dart

+  Message _reconcileWhenPresent(Message current, Message incoming) {
+    bool currentIsMaybeStale = false;
+    if (incoming is StreamMessage) {
+      if (subscriptions[incoming.streamId] != null) {
+        // The incoming message won't grow stale; it's in a subscribed channel.
+        // Remove it from _maybeStaleChannelMessages if it was there.
+        currentIsMaybeStale = _maybeStaleChannelMessages.remove(incoming.id);
+      } else {
+        assert(_maybeStaleChannelMessages.contains(incoming.id));
+        currentIsMaybeStale = true;
+      }
+    }
+
+    return currentIsMaybeStale
+      ? _stripMatchFields(incoming)
+      : current;


I think this logic would benefit from having the explanation in the long comment above worked into it.

Probably that mostly means just moving that comment from reconcileMessages into this method. After all, it starts with:

What to do when some of the just-fetched messages are already known?

so it's entirely about this "present" case.

Beyond that, I think once the comment is here in this method, the "normally" vs. "with one exception" cases can be worked a bit into the corresponding cases in this code's control flow.

That makes a lot of sense, thanks!!

gnprice · 2025-10-17T00:18:50Z

lib/model/message.dart

+    }
+  }
+
+  Message _reconcileWhenPresent(Message current, Message incoming) {


nit: maybe give these names that are a bit more semantic: like "reconcile known message" vs. "reconcile unknown message" (or "new" or "previously unknown"?)

gnprice · 2025-10-17T00:20:35Z

lib/model/message.dart

+      : current;
+  }
+
+  Message _reconcileWhenAbsent(Message incoming) {


nit: maybe put the absent/unknown case first — it happens first chronologically, after all (except in the case of a message we learn about from an event), and it's also simpler, so I think conceptually prior

(can do the same at the call site too, thanks to Dart accepting named arguments before positional)

Thanks to Greg for pointing this out: zulip#1912 (comment)

chrisbobbe · 2025-10-17T21:13:25Z

Thanks for the review! Revision pushed.

gnprice · 2025-10-18T03:02:45Z

Thanks! I think that revision is helpful.

Over to @rajveermalviya for normal maintainer-review.

rajveermalviya

Thanks @chrisbobbe! LGTM, and tests great.
Moving over to Greg's review.

chrisbobbe · 2025-10-23T15:33:50Z

Thanks for the review!

gnprice

Thanks! Comments below after a first full review.

gnprice · 2025-10-30T19:42:26Z

lib/model/message.dart

+
+      messages[i] = this.messages.update(message.id,


nit:

Suggested change

messages[i] = this.messages.update(message.id,

messages[i] = this.messages.update(message.id,

gnprice · 2025-10-30T19:46:02Z

lib/model/message.dart

+    // This message is one we already know about. This is common and normal:
+    // in particular it happens when one message list overlaps another,
+    // e.g. a stream and a topic within it.


nit: keep the semantic line-breaks the existing comment has:

Suggested change

// This message is one we already know about. This is common and normal:

// in particular it happens when one message list overlaps another,

// e.g. a stream and a topic within it.

// This message is one we already know about.

// This is common and normal: in particular it happens when one message list

// overlaps another, e.g. a stream and a topic within it.

Separately, I think saying "just-fetched" here is helpful for connecting to the later paragraphs, and setting context of what "this" message is:

Suggested change

// This message is one we already know about. This is common and normal:

// in particular it happens when one message list overlaps another,

// e.g. a stream and a topic within it.

// This just-fetched message is one we already know about.

// This is common and normal: in particular it happens when one message list

// overlaps another, e.g. a stream and a topic within it.

gnprice · 2025-10-30T19:58:52Z

lib/model/message.dart

+    if (channelIds.length > 1) {
+      assert(channelIds is! Set);
+      channelIds = Set.from(channelIds); // optimization
+    }


At length 2, for a set of ints, I think List.contains is going to be considerably faster than Set.contains.

The former just has to take the "needle" int in question and compare it against two other ints in turn. The latter has to take a hash of the needle, look to see if there's an entry at the resulting spot in the hash table, then compare the needle against what it finds there.

Similarly for a handful of ints. At a guess, the breakpoint I'd use for this optimization would be 8 rather than 1.

(If this were a likely hot path so the optimization was important then naturally we'd want to measure. But it's definitely not worth that level of effort.)

Oh I see, at one call site channelIds is a List.map iterable rather than a List itself.

I think that doesn't change much. The map callback is just looking up a field on an object; so basically it just adds one load from memory through a pointer. Maybe that tips my guess toward 4 as the breakpoint rather than 8 (I was already waffling a bit between those).

Oh I see, at one call site channelIds is a List.map iterable rather than a List itself.

This is about to change to be a List too; we're about to add a List<int> channelIds to ChannelDeleteEvent; see 8b88703 🙂

So I'll use 8, and link to your comment here? 🙂

gnprice · 2025-10-30T20:03:16Z

lib/model/message.dart

-    // See [fetchedMessages] for reasoning.
-    messages[event.message.id] = event.message;
+    // See [reconcileMessages] for reasoning.
+    messages[event.message.id] = message;


nit:

Suggested change

messages[event.message.id] = message;

messages[message.id] = message;

gnprice · 2025-10-30T20:08:40Z

test/model/message_test.dart

+    if (isChannelSubscribed) {
+      subscription = eg.subscription(stream);


Should set subscription to null in the alternative case — otherwise the previous test's value for it leaks into this test

gnprice · 2025-10-30T20:11:00Z

test/model/message_test.dart

-      check(messages).single.identicalTo(message);
-      check(store.messages).deepEquals({1: message});
+    group('fetched message with ID already in store.messages', () {
+      late Message messageCopy;


It looks like we can avoid a shared variable here — instead make it local within each test body, by just inserting Message before the first assignment.

gnprice · 2025-10-30T20:11:43Z

test/model/message_test.dart

+      Message copyStoredMessage({String? displayRecipient}) {
+        final message = store.messages.values.single;
+
+        Map<String, dynamic> json = message.toJson();


nit:

Suggested change

Map<String, dynamic> json = message.toJson();

final json = message.toJson();

gnprice · 2025-10-30T20:22:36Z

test/model/message_test.dart

+          messageCopy = copyStoredMessage();
+          store.reconcileMessages([messageCopy]);
+          // The channel became subscribed,
+          // but the message's data hasn't been refreshed, so clobber…
+          checkStoredMessageIdenticalTo(messageCopy);
+
+          store.reconcileMessages([copyStoredMessage()]);
+          // …Now it's been refreshed, by reconcileMessages, so don't clobber.
+          checkStoredMessageIdenticalTo(messageCopy);


This test case is a bit long and fiddly to read. I think it could be simplified with a pair of helpers along these lines:

Suggested change

messageCopy = copyStoredMessage();

store.reconcileMessages([messageCopy]);

// The channel became subscribed,

// but the message's data hasn't been refreshed, so clobber…

checkStoredMessageIdenticalTo(messageCopy);

store.reconcileMessages([copyStoredMessage()]);

// …Now it's been refreshed, by reconcileMessages, so don't clobber.

checkStoredMessageIdenticalTo(messageCopy);

// The channel became subscribed,

// but the message's data hasn't been refreshed, so clobber…

checkClobber();

// …Now it's been refreshed, by reconcileMessages, so don't clobber.

checkNoClobber();

which should make it easier to scan and see all the different scenarios it's going through.

Thanks to Greg for pointing this out: zulip#1912 (comment)

chrisbobbe · 2025-10-31T22:03:12Z

Thanks for the review! Revision pushed.

gnprice · 2025-10-31T22:19:35Z

test/model/message_test.dart

+          store.reconcileMessages([copyStoredMessage()]);
+          checkStoredMessageIdenticalTo(messageBefore);


Suggested change

store.reconcileMessages([copyStoredMessage()]);

checkStoredMessageIdenticalTo(messageBefore);

final messageBefore = store.messages.values.single;

store.reconcileMessages([copyStoredMessage()]);

checkStoredMessageIdenticalTo(messageBefore);

This would be equivalent, right? And it's getting the existing message from the same place copyStoredMessage is, so the logic remains coherent.

Then I think the caller doesn't need checkClobber to return anything, either.

Oh nice, yep! Thanks!

gnprice

Thanks for the revision! This makes the tests much easier to read. Comments below (and one above).

gnprice · 2025-10-31T22:33:55Z

test/model/message_test.dart

+          await store.addStream(otherChannel);
+
+          store.reconcileMessages([message]);
+          checkStoredMessageIdenticalTo(message);


Then I think this and the similar call below no longer help tell the story — "check no clobber" says it already, and this line doesn't particularly connect to that anyway.

Suggested change

checkStoredMessageIdenticalTo(message);

gnprice · 2025-10-31T22:35:24Z

test/model/message_test.dart

+          final otherChannel = eg.stream();
+          await store.addStream(otherChannel);


I think these are clearest if moved to just before where they start getting used — i.e. to the start of the section where the message gets moved.

gnprice · 2025-10-31T22:35:44Z

test/model/message_test.dart

+          await store.handleEvent(
+            eg.updateMessageEventMoveFrom(origMessages: [message],
+            newStreamId: otherChannel.streamId));


nit:

Suggested change

await store.handleEvent(

eg.updateMessageEventMoveFrom(origMessages: [message],

newStreamId: otherChannel.streamId));

await store.handleEvent(

eg.updateMessageEventMoveFrom(origMessages: [message],

newStreamId: otherChannel.streamId));

gnprice · 2025-10-31T22:37:43Z

test/model/message_test.dart

+        test('various conditions', () async {
+          final channel = eg.stream();
+          await prepare(stream: channel, isChannelSubscribed: true);
+          final message = eg.streamMessage(id: 1, stream: channel);


Suggested change

final message = eg.streamMessage(id: 1, stream: channel);

final message = eg.streamMessage(stream: channel);

The message ID isn't relevant, right? (Similarly below.)

bump on this one 🙂 (though you handled the others below)

gnprice · 2025-10-31T22:39:42Z

test/model/message_test.dart

+          checkClobber();
+        });
+
+        test('new-message event when in unsubscribed channel', () async {


Another case to test: message event in subscribed channel

This test is now quite easy to write! 😄

test('new-message event when in a subscribed channel', () async { await prepare(isChannelSubscribed: true); final message = eg.streamMessage(); await store.handleEvent(eg.messageEvent(message)); checkNoClobber(); checkNoClobber(); });

Thanks to Greg for pointing this out: zulip#1912 (comment)

chrisbobbe · 2025-10-31T22:59:09Z

Thanks for the review! Revision pushed.

gnprice

Thanks! Two nits.

gnprice · 2025-11-01T00:10:41Z

test/model/message_test.dart

+          .deepEquals({message.id: conditionIdentical(message)});
+      }
+
+      Message checkClobber({Message? withMessageCopy}) {


nit: no need to return a value now, right?

(And the caller could always get the same value from store.messages.values.single.)

gnprice · 2025-11-01T00:11:46Z

test/model/message_test.dart

+        test('various conditions', () async {
+          final channel = eg.stream();
+          await prepare(stream: channel, isChannelSubscribed: true);
+          final message = eg.streamMessage(id: 1, stream: channel);


bump on this one 🙂 (though you handled the others below)

Thanks to Greg for pointing this out: zulip#1912 (comment)

chrisbobbe · 2025-11-01T00:17:08Z

Ah indeed, thanks!! Revision pushed.

Thanks to Greg for pointing this out: zulip#1912 (comment)

It's way more common to be using the app with subscribed channels than unsubscribed channels, so we might as well test with subscribed channels except where we specifically want to check behavior for unsubscribed channels.

Like we did in the compose-box tests in the previous commit.

…ulip#1798)

This fixes the "fourth buggy behavior" in zulip#1798: zulip#1798 (comment) Fixes-partly: zulip#1798

gnprice · 2025-11-01T00:18:32Z

Thanks! Looks good; merging.

chrisbobbe requested a review from rajveermalviya October 10, 2025 22:38

chrisbobbe assigned rajveermalviya Oct 10, 2025

chrisbobbe added the maintainer review PR ready for review by Zulip maintainers label Oct 10, 2025

chrisbobbe marked this pull request as draft October 10, 2025 23:49

chrisbobbe force-pushed the pr-unsubscribed-channels-fourth-buggy-behavior branch from 3612e86 to 4fa0a1a Compare October 15, 2025 23:49

chrisbobbe marked this pull request as ready for review October 15, 2025 23:49

chrisbobbe mentioned this pull request Oct 15, 2025

In unsubscribed channel, handle that message-send/edit events are not expected #1798

Open

gnprice reviewed Oct 16, 2025

View reviewed changes

chrisbobbe added a commit to chrisbobbe/zulip-flutter that referenced this pull request Oct 16, 2025

message [nfc]: Fix a reference in a comment

73b74ca

Thanks to Greg for pointing this out: zulip#1912 (comment)

chrisbobbe force-pushed the pr-unsubscribed-channels-fourth-buggy-behavior branch from 4fa0a1a to 6ed2244 Compare October 16, 2025 21:04

chrisbobbe force-pushed the pr-unsubscribed-channels-fourth-buggy-behavior branch 3 times, most recently from e8f8054 to a9b4906 Compare October 16, 2025 21:20

gnprice reviewed Oct 17, 2025

View reviewed changes

chrisbobbe added a commit to chrisbobbe/zulip-flutter that referenced this pull request Oct 17, 2025

message [nfc]: Fix a reference in a comment

bb598a8

Thanks to Greg for pointing this out: zulip#1912 (comment)

chrisbobbe force-pushed the pr-unsubscribed-channels-fourth-buggy-behavior branch from a9b4906 to 6f75b2b Compare October 17, 2025 21:13

rajveermalviya approved these changes Oct 23, 2025

View reviewed changes

rajveermalviya added integration review Added by maintainers when PR may be ready for integration and removed maintainer review PR ready for review by Zulip maintainers labels Oct 23, 2025

rajveermalviya assigned gnprice and unassigned rajveermalviya Oct 23, 2025

rajveermalviya requested a review from gnprice October 23, 2025 12:56

gnprice reviewed Oct 30, 2025

View reviewed changes

chrisbobbe added a commit to chrisbobbe/zulip-flutter that referenced this pull request Oct 31, 2025

message [nfc]: Fix a reference in a comment

7408e04

Thanks to Greg for pointing this out: zulip#1912 (comment)

chrisbobbe force-pushed the pr-unsubscribed-channels-fourth-buggy-behavior branch from 6f75b2b to bb7b66a Compare October 31, 2025 22:03

gnprice reviewed Oct 31, 2025

View reviewed changes

chrisbobbe added a commit to chrisbobbe/zulip-flutter that referenced this pull request Oct 31, 2025

message [nfc]: Fix a reference in a comment

613f459

Thanks to Greg for pointing this out: zulip#1912 (comment)

chrisbobbe force-pushed the pr-unsubscribed-channels-fourth-buggy-behavior branch from bb7b66a to 12ba5b8 Compare October 31, 2025 22:59

gnprice reviewed Nov 1, 2025

View reviewed changes

chrisbobbe added a commit to chrisbobbe/zulip-flutter that referenced this pull request Nov 1, 2025

message [nfc]: Fix a reference in a comment

03c9fdd

Thanks to Greg for pointing this out: zulip#1912 (comment)

chrisbobbe force-pushed the pr-unsubscribed-channels-fourth-buggy-behavior branch from 12ba5b8 to 956fe13 Compare November 1, 2025 00:15

chrisbobbe added 7 commits October 31, 2025 17:18

message [nfc]: Fix a reference in a comment

fee6de6

Thanks to Greg for pointing this out: zulip#1912 (comment)

msglist test: Switch to using subscribed channels, where boring

f89ce2e

Like we did in the compose-box tests in the previous commit.

message test [nfc]: Add conditionIdentical helper, for reconcileMessages

0bacdbc

message [nfc]: Pull out _stripMatchFields helper for reconcileMessages

ab5458a

message [nfc]: Pull out some helpers for reconcileMessages; add TODO(z…

5469cbe

…ulip#1798)

message: Consider unsubscribed/unknown channels in reconcileMessages

2ab661c

This fixes the "fourth buggy behavior" in zulip#1798: zulip#1798 (comment) Fixes-partly: zulip#1798

gnprice force-pushed the pr-unsubscribed-channels-fourth-buggy-behavior branch from 956fe13 to 2ab661c Compare November 1, 2025 00:18

gnprice merged commit 2ab661c into zulip:main Nov 1, 2025
1 check passed

chrisbobbe deleted the pr-unsubscribed-channels-fourth-buggy-behavior branch November 1, 2025 00:19

		// [1] With one exception: if the version we have was in an unsubscribed
		// channel when we got it or sometime since, we take the fetched version


	messages[i] = this.messages.update(message.id,
	messages[i] = this.messages.update(message.id,

	messages[event.message.id] = message;
	messages[message.id] = message;

		if (isChannelSubscribed) {
		subscription = eg.subscription(stream);

	Map<String, dynamic> json = message.toJson();
	final json = message.toJson();

		store.reconcileMessages([copyStoredMessage()]);
		checkStoredMessageIdenticalTo(messageBefore);

		final otherChannel = eg.stream();
		await store.addStream(otherChannel);

	final message = eg.streamMessage(id: 1, stream: channel);
	final message = eg.streamMessage(stream: channel);

message: Consider unsubscribed channels in reconcileMessages #1912

message: Consider unsubscribed channels in reconcileMessages #1912

Uh oh!

Conversation

chrisbobbe commented Oct 10, 2025

Uh oh!

chrisbobbe commented Oct 10, 2025

Uh oh!

gnprice left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrisbobbe commented Oct 16, 2025

Uh oh!

gnprice left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrisbobbe Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrisbobbe commented Oct 17, 2025

Uh oh!

gnprice commented Oct 18, 2025

Uh oh!

rajveermalviya left a comment

Choose a reason for hiding this comment

Uh oh!

chrisbobbe commented Oct 23, 2025

Uh oh!

gnprice left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrisbobbe commented Oct 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gnprice left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

chrisbobbe Oct 17, 2025 •

edited

Loading