feat: support text_mode collapse recursive

[CahierX]

To solve #557 issues
Add text_mode collapse recursive

[josdejong]

That is impressive, this looks promising! 😎

The code looks really neat. Whilst testing I came across two issues though:

1. When I collapse the whole document with refJsonEditor.collapse([], true), the recursiveness doesn't work: only the root object or array is collapsed.
2. When I open a large document and I perform refJsonEditor.collapse([], true), only the first part of the items in an array is collapsed. For example take the "10k users" example from here. This may be related to CodeMirror only rendering the visible part of the contents, I'm not sure.

I think issue (1) is likely a small bug but (2) may be tricky.

[CahierX]

That is impressive, this looks promising! 😎

The code looks really neat. Whilst testing I came across two issues though:

1. When I collapse the whole document with refJsonEditor.collapse([], true), the recursiveness doesn't work: only the root object or array is collapsed.
2. When I open a large document and I perform refJsonEditor.collapse([], true), only the first part of the items in an array is collapsed. For example take the "10k users" example from here. This may be related to CodeMirror only rendering the visible part of the contents, I'm not sure.

I think issue (1) is likely a small bug but (2) may be tricky.

I will try to fix it

[CahierX]

Hi,

I have completely rewritten the collapse logic.

Previously, the foldAll function had issues. As stated in the official documentation:

Fold all top-level foldable ranges. Note that, in most cases, folding information will depend on the syntax tree, and folding everything may not work reliably when the document hasn't been fully parsed (either because the editor state was only just initialized, or because the document is so big that the parser decided not to parse it entirely).

This explains why folding doesn’t work reliably for large JSON files.

To address this, I used ensureSyntaxTree to obtain a fully parsed syntax tree, then iterated through the nodes to collect foldable.from and foldable.to. Finally, I dispatched the folding ranges using:

codeMirrorView.dispatch({
  effects: foldRanges.map((range) => foldEffect.of({ from: range.from, to: range.to }))
});

After testing, I found that folding works correctly for a 10k JSON! However, due to the large data size, dispatch the fold effects is quite time-consuming—it takes almost 10 seconds on my Mac M3 Pro.

The time-consuming method in our code is getFoldRanges. 1K -> 100ms, 10k -> 720ms


So this performance problem is a codeMirrorView problem, and it may not be solved

Additionally, I created a custom foldService (createCustomFoldService).This is because the built-in foldable function relies on syntaxTree, which, as the docs explain, might return an incomplete tree.

Get the syntax tree for a state, which is the current (possibly incomplete) parse tree of the active, or the empty tree if there is no language available.

My custom foldService replicates the logic of syntaxFolding, but uses ensureSyntaxTree instead of syntaxTree to guarantee a complete tree. This way, we can reliably retrieve from and to using foldable, and apply folding.

The changes are quite substantial — let me know if anything looks off.

[CahierX]

I also submitted a PR to CodeMirror. If it gets merged, we’ll be able to remove our createCustomFoldService.

[josdejong]

Impressive! The folding/unfolding works reliable now 🎉.

The performance is indeed an issue. I think we can do two things (until maybe the performance improves some day).

1. Limit recursive collapse/expand to a certain document size like 100KB or so, and throw an error "Too large to expand/collapse recursively" when a using the method, or show an error message to the user when expanding/collapsing recursively via a button click (we do not yet have these buttons but I hope we can implement that soon).
2. Implement recursive collapse/expand in such a way that it can be cancelled when it takes too long. We can used an abord controller for the method API, and show a message with a cancel button in case of a user interaction (and maybe even a progress percentage).

I think option (2) would be nicest if possible, but it is also the most complex to implement. What do you think?

[CahierX]

Impressive! The folding/unfolding works reliable now 🎉.

The performance is indeed an issue. I think we can do two things (until maybe the performance improves some day).

1. Limit recursive collapse/expand to a certain document size like 100KB or so, and throw an error "Too large to expand/collapse recursively" when a using the method, or show an error message to the user when expanding/collapsing recursively via a button click (we do not yet have these buttons but I hope we can implement that soon).
2. Implement recursive collapse/expand in such a way that it can be cancelled when it takes too long. We can used an abord controller for the method API, and show a message with a cancel button in case of a user interaction (and maybe even a progress percentage).

I think option (2) would be nicest if possible, but it is also the most complex to implement. What do you think?

I also agree with option 2. I will try to optimize the performance first, but if it doesn't work, I will use option 2 for optimization, though it may take some time.

[josdejong]

Thanks!

[CahierX]

Sorry for the delay in submitting the code - I've been quite busy with work lately.

Regarding the performance issue with codeMirrorView.dispatch({effects}) when handling large JSON files, I investigated the CodeMirror source code but couldn't find any viable directions for performance optimization. Therefore, I've implemented solution 2 to address this issue.

I've added a folding progress bar with cancellation support.

collapseBatchSize = 100 - This value ensures folding speed while keeping the editor responsive during operations. It's a relatively optimal value determined through testing.

showCollapseProgressSize = 5000 - Progress bar is not displayed when folding fewer than 5000 items, as folding typically takes less than 0.5s in such cases.

[josdejong]

That is awesome news! I'll review your latest update asap but that will probably be within one or two weeks from now.

[josdejong]

First: sorry for the delay in review, first I was on holiday and then I was sick 😅.

I've just tested this PR and it works like a charm, I'm really happy you managed to pull this off. Thanks for this awesome piece of work! The code is well taken care and easy to follow. I made only one small inline remark. Besides that I think this PR is ready to be merged.

I have a couple of ideas for further improvement, but it may be better to address that in separate PR's:

1. Add a (recursive) "Collapse all" and "Expand all" button to the main menu, like in tree mode
2. Implement ctrl+click to recursively expand/collapse a node in code mode, just like it works in tree mode
3. Change the expand/collapse methods to return a Promise so you know when the operation is actually finished. Also, either allow passing your own AbortController, or return the abort controller (alongside the promise?) so it is possible to programmatically cancel a recursive collapse too.

What do you think?

[CahierX]

First: sorry for the delay in review, first I was on holiday and then I was sick 😅.

I've just tested this PR and it works like a charm, I'm really happy you managed to pull this off. Thanks for this awesome piece of work! The code is well taken care and easy to follow. I made only one small inline remark. Besides that I think this PR is ready to be merged.

I have a couple of ideas for further improvement, but it may be better to address that in separate PR's:

1. Add a (recursive) "Collapse all" and "Expand all" button to the main menu, like in tree mode
2. Implement ctrl+click to recursively expand/collapse a node in code mode, just like it works in tree mode
3. Change the expand/collapse methods to return a Promise so you know when the operation is actually finished. Also, either allow passing your own AbortController, or return the abort controller (alongside the promise?) so it is possible to programmatically cancel a recursive collapse too.

What do you think?

I’m sorry to hear that you’ve been sick. Are you feeling better now?
Regarding the “Collapsing” text, I’ve already added it.
As for the points you mentioned (1, 2, and 3), I think they’re all very reasonable. I’ll try to add them in future PRs when I have the time.
Wishing you all the best!

[josdejong]

Thanks, awesome job!

Are you feeling better now?

Yes 😅. It was just the flu, nothing special though quite annoying.

As for the points you mentioned (1, 2, and 3), I think they’re all very reasonable. I’ll try to add them in future PRs when I have the time.

👍

[github-actions]

🎉 This PR is included in version 3.9.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[josdejong]

Just for reference: I've added "Expand All" and "Collapse All" buttons to the menu in text mode now, see #563. I love it, thanks again for making this possible @CahierX!

[CahierX]

Just for reference: I've added "Expand All" and "Collapse All" buttons to the menu in text mode now, see #563. I love it, thanks again for making this possible @CahierX!

🎉🎉