Introduce pagination for the array fields

[emirotin]

It would be nice and, in fact, required for the certain use cases to introduce paging to the array fields.

That, IMO, should extend to all the 3 API flavors and the UI.

Specifically, I have an array field that manages the relation field (games collection to game), and one doc has 5K entries under it.

The API request for that doc crashes the CMS about 60% of times, I guess simply due to the memory limits.

The UI nicely handles arrays of 500 records, but of course it hangs on this one, I imagine due to both the high number of DOM nodes and to 5K lookup requests flying in parallel.

With the REST API, I would Imagine something like page[fieldName]=1&page_size[fieldName]=100, or something along that lines.

In the UI some form of the traditional paging controls. One challenge I can see here is the drag-n-drop functionality (which we use and need) when a certain array element needs to be moved between the pages.

[DanRibbens]

Hi @emirotin,

This is interesting! I'm not sure I understand how you are modeling this data. You have a games collection with an array field containing a relationship to games? Can you share some more detail?

I understand why you would want to support pagination if your array is this long, but it might just be better to come up with a different shape for your collections and fields that is easier to work with.

We don't support subdocument projection (or even top level on a document) and while we could possibly do something with this, other parts of the app will be subject to crashing given the memory use unless we are able to safeguard it in our operations. Imagine if you have to update a document like the one you described, but you have a hook on a nested field. The operation would need to iterate and update the document in batches, which isn't something we could support. Or how could validation functions work when the entire document isn't loaded?

There is more work needed for UI performance on larger documents in general, but I am uncertain about accommodating your example without coming up with an entirely new approach.

[emirotin]

This is interesting! I'm not sure I understand how you are modeling this data. You have a games collection with an array field containing a relationship to games? Can you share some more detail?

Yes, exactly, I have the Game entity, which has some props, like the name (localized), description (same), image upload, etc.
Then I have the Collection entity, with the similar pros.
Finally, the Collection needs to contain games, so I have an array prop with a single field inside — a reference to Game.

We also have similar relations in other places, sometimes more complex (i.e.: "Virtual Games" are the Game-like entities that have the array prop of "Real Games", linking to the Game and also to Country). But none of them in practice can be as massive.

[DanRibbens]

Is there any logical way you could break this up better?
Some ideas would be to add remove your array and instead add a relationTo from the Collection back to Game. Then you can use the collection list view with a filter to say "show me all the Collections where Game = X".

I assume the Collection documents are being made via the API and not manually, so the editor experience wouldn't really matter. If you wanted to have Collection count or some calculated field on the Game, that could give you that at a glance, or you make a virtual field that queries to reduce updates via hooks.

Would that be a possible improvement on how you are handling this?

[emirotin]

The collections and games are created via the API once, but then are supposed to be manually edited (games added to collections, removed, reordered, etc.) So, the UI matters, that's why I've mentioned the drag-and-drop functionality for reordering the items.

I don't see how replacing an array of relations with a single relation field would let me encode the same domain: the game to collection is many-to-many: collection contains multiple games, and the same game can be in several collections