High volumes of data - best practices

[ziverlichson]

Hi everyone,
What are the best practices to follow when dealing with high volumes of data? For example, 1 milion entries (rows).

Pagination is probably one answer, but what else?
How would you handle actions performed on the entire set of data, such as sorting/filtering?

Thanks.

[sbimochan]

Enable paginations with dropdown giving option for number of rows.
Sort/Filter only the result shown by pagination. For example 10/1M. It will sort only among 10 or 100 selected by user.
Add a search too with asyncDebounce of around 200ms to prevent high CPU usage.

[ziverlichson]

I have enabled pagination, but in my case, sorting & filtering are tabular operations - meaning, they are performed on all rows in the table, not only those on the current page. For example, if I sort by 'name', the last row on the last page of the table, might become the first row on the first page.

[gargroh]

One suggestion could be - Try to reduce cost of render...

[hkurra]

In one of my use-case, we are using react-virtual to virtualize the rows, codesandbox here.
I have also enabled Lazy-loading to load more data when user scrolled to the end only.
Now coming to sorting, I am using server side sorting which means, If user change the sorting, i will make query for DB and get few initial records(You can say page1) then user can keep scrolling with this filter on

[ziverlichson]

Thanks @hkurra .
Were you able to work use react-virtual with dynamic sized rows or rows that can expand?

[hkurra]

Hmmm, in our case rows are of fixed size but i am passing ref of the row to the react-virtual item through measureRef property, as per my understanding, it is meant for same purpose which is to calculate the dimension of row dynamically based from ref.
More info here

You can also provide estimateSize option to react-virtual, but i dont this will be suitable in your case

[danvln]

Hey folks,

I am facing the same challenge. I am looking for a solution to use expand with react-window/VariableSizeList or Fixed. Did you manage to get anything working? I would appreciate any pointers. Please share if you have any example.

Thanks!

[hkurra]

@danvln I am using react-virtual for virtualization and also tried react-window for virtualization it also works fine but I was facing some issues with the variable Height that's what I switch back to the react-virtual, I am also using row expansion with the react-virtual without any major issues as of now, although the changes are still in development still, things are good enough