Hash stored in map causes panick, type inconsistency between JS and Ruby

[wkirby]

We are attempting to modify the same ydoc in the browser using yjs, and on our server using yrb. I have a pretty simple setup client side (this is simplified further for the sake of this report, but captures our use case fairly):

const ydoc = new Y.Doc();
const localBinder = bindImmerYjs(ydoc.getMap("data"));

localBinder.update((s) => {
  return {
    testString: "foo",
    testInt: 1,
    testFloat: 3.1415,
    testBool: false,
    testArray: [1, "two", true],
    testHash: { a: 1, b: "2", c: false }
  }
})

This ydoc is encoded as a uint8array, base64'd, then sent to the server where it's saved to a file store. In ruby, we load this content and try to access these values. This is mostly working. Roughly, this looks like:

ydata = Base64.decode64(stored_data)
ydoc = Y::Doc.new
ydoc.transact { |tx| tx.apply(ydata.bytes.to_a) }

ymap = ydoc.get_map("data")

All that is working, but this is where we run into problems:

# Works as expected
ymap[:testString] # => "foo"
ymap[:testBool] # => false
ymap[:testFloat] # => 3.1415

# Works close enough, but still wrong
ymap[:testInt] # => 1.0 (notice this is a Float now)

# Panick!
ymap[:testArray]
ymap[:testHash]

These Panicks are identical, producing:

thread '<unnamed>' panicked at /bundle/ruby/3.1.0/gems/y-rb-0.5.2/ext/yrb/.rb-sys/stable/cargo/registry/src/index.crates.io-6f17d22bba15001f/yrs-0.16.9/src/doc.rs:775:41:
called `Option::unwrap()` on a `None` value

There's an obvious workaround here where we can encode our hashes and arrays as a JSON string, but if we're going to do that, we might as well encode the whole map as a JSON string instead of as a map in the first place.

Note that this is not a problem when we decode these values on the client side.

[wkirby]

@eliias should I open this as an issue upstream with yrs?

[eliias]

@eliias should I open this as an issue upstream with yrs?

@wkirby I am traveling atm. I can take a look next week. If it's really an upstream issue I'll move the issue.

[eliias]

@wkirby I recall having a discussion w/ the other yrs folks about support for nested “complex” data structures (like Array, and Object). I need to check if it's technically possible to do that as long as you know what you do (there will be no conflict-free replication for nested objects). Text editors usually utilize the XML-like structure to achieve proper replication with nested data-structures.

Works close enough, but still wrong

ymap[:testInt] # => 1.0 (notice this is a Float now)

There is technically no Integer in JavaScript, so not sure if there is an easy “fix” for this. Probably best to do value.to_i when you know it should be an Integer in Ruby.

[eliias]

@wkirby I actually can't reproduce the panic, even using your example values. Could you provide me a value for ydata = Base64.decode64(stored_data) -> stored_data to allow me debug this further?

https://github.com/y-crdt/yrb/blob/main/spec/y/map_spec.rb#L179-L192

[wkirby]

@eliias Just got back from some time off myself. I'll send over the encoded document we have and work on setting up a more concise repro. There's always the chance that the error is on the encoding end with y-immer.

[wkirby]

@eliias attached is a file containing a Base64 string that presents the error. Here's how I reproduce with this data:

ydata = File.read('ydoc.base64.txt')
ydata = Base64.decode64(ydata)
ydoc = Y::Doc.new
ydoc.transact { |tx| tx.apply(ydata.bytes.to_a) }

ymap = ydoc.get_map("data")

ymap[:testInt] # => 1.0
ymap[:testFloat] # => 3.1415
ymap[:testString] # => "foo"
ymap[:testBool] # => false

ymap[:testArray] # => Panick!
ymap[:testHash] # => Panick!

ydoc.base64.txt

Because this blob is pulled directly from our application, the Map at data also contains the following keys:

title
fields
fieldValues
agents
signatories

In addition, the ydoc includes an XML fragment at content which is serialized tiptap state. I don't believe either of these are causing the issue, as the same problem was present without these values in the ydoc.

Here are the relevant lines from my Gemfile.lock:

    y-rb (0.5.2)
      rake (~> 13.0)
      rb_sys (~> 0.9.71)
    y-rb (0.5.2-x86_64-linux)
      rake (~> 13.0)
      rb_sys (~> 0.9.71)

And from package.json:

"immer-yjs": "^1.1.0",
...
"yjs": "^13.6.7",

On the client side, the code (roughly) to serialize this data:

const useSyncedData = (doc) => {
  const ydoc = useMemo(() => new Y.Doc(), [doc.id]);
  const [binder, setBinder] = useState(undefined);

  useEffect(() => {
    const localBinder = bindImmerYjs(ydoc.getMap("data"));

    if (localBinder) {
      setBinder(localBinder);

      // optionally set initial data
      localBinder.update((s) => {
        return {
          testString: "foo",
          testInt: 1,
          testFloat: 3.1415,
          testBool: false,
          testArray: [1, "two", true],
          testHash: { a: 1, b: "2", c: false }
        };
      });
    }

    return () => {
      localBinder.unbind();
    };
  }, [ydoc, doc]);

  return ydoc;
}

const MyReactComponent = () => {
  const { doc } = useDoc();
  const ydoc = useSyncedData(doc);

  console.log(ydoc);

  return <p>Oh hey, a component</p>
}

[wkirby]

@eliias hope I'm not bugging you, but were the above resources enough to reproduce? If not I'm happy to schedule a call to work through this together. Just shoot me an email at wyatt@apsis.io if you'd like.

[eliias]

title

Yeah, I am able to reproduce this in a test case with your data. I also found the actual issue, but I do not have an immediate solution that is shippable. The logic that converts the internal CRDT representation for a Map/Array requests a new read-only transaction on the CRDT store, which is usually fine. The Ruby library always maintains a read-write transaction, so it causes a panic in this case. It is also unclear to me why it only fails on a sync. It should fail in both cases.

FWIW… I was able to access your test data after a somewhat hacky fix, so it is definitely doable.
#134

[wkirby]

@eliias I'm not super versed in Rust, so I can't comment on the hackiness present here. I really appreciate your attention on this, and do let me know if there's any other way I can assist!

[eliias]

@wkirby I am going to release 0.5.3 later today that includes a fix. The spec passes for your test data. Would be great if you can confirm it works for you.

[wkirby]

@eliias sorry, we moved on in our project to other things, but are returning now. I will confirm within the next week or so whether your implementation fixes our issue. Thank you so much!