Represent cells or cell outputs as subdocuments?

[krassowski]

Maybe this is a way to make sure that outputs don't inflate the size of yjs document?
https://docs.yjs.dev/api/subdocuments

Also jupyter-ai-contrib/jupyter-server-documents#153

[davidbrochart]

What do you mean by inflating the size of Yjs documents? Subdocuments can be lazily loaded, but once they are, they take up the same amount of memory.

[krassowski]

Well, I mean the same thing which led to creation of https://github.com/jupyter-ai-contrib/jupyter-server-documents: if one output is 1GB, the notebook document is 1GB. If you are updating the output multiple times, the document is even larger (set aside the stream outputs which gets optimized a bit by having a special silent origin). If these can be lazily loaded then the notebook document as a whole is still responsive, can be stored in say YIndexedDB etc.

[krassowski]

Further my thinking is that maybe we can squash the history of output updates without squashing the history of the entire notebook document or at least control this with much more granularity.

[davidbrochart]

Well, I mean the same thing which led to creation of https://github.com/jupyter-ai-contrib/jupyter-server-documents

It is not clear to me why this was created.

If you are updating the output multiple times, the document is even larger

I don't agree with this, as the garbage collector in pycrdt would prevent that.

Further my thinking is that maybe we can squash the history of output updates without squashing the history of the entire notebook document or at least control this with much more granularity.

That is a good point.

[dmonad]

If outputs need to be loaded anyway, you probably won't benefit from subdocuments.

You can also optimize the history by putting the output in a Y.Map. I.e.

Y.Map<Y.Array>

Output can be streamed to the Y.Array. Output can be replaced by setting a new Y.Array: ymap.set('output', Y.Array).

The metadata will be squashed very efficiently by deleting a child-node.

However, if you don't want the output to be part of snapshots, then this would be a good point for subdocuments.