Updating data based on timestamp

[tomyates]

Hello!
I'm hoping that someone can give me a brief overview of the best way to fetch data based on a timestamp:

IE. on initial load of the app, I download a full set of data and keep it cached, but on subsequent calls to the API include a timestamp so it only fetches the new/updated data from the API and merges it with the existing set.

I think I might need to use hydration - but not completely sure.

My initial call to the API is expensive and slow, so only want to do it once while use is using the app.

Thanks in advance,
Tom

[TkDodo]

My initial call to the API is expensive and slow, so only want to do it once while use is using the app.

This can be achieved by setting staleTime: Infinity. I'm not really sure what this has to do with including a timestamp when sending the request to the server...?

[tomyates]

Thanks.
I'll try and be clearer:

Say there was an end point '/news'

The first time the app loads the query fetches all the news data. (1000 rows).

However, the the query runs again, I don't want want the API to return all the news data again, as it's slow and inefficient, I just want to return updated or new data. (A few rows) and the query merge this with the data that had originally been fetched.

I can do this with the /news end-point by passing a timestamp and it only returns data that's timestamp is younger than the passed timestamp.

For example /news&timestamp=1620937819

I'm trying to work out the best way to do this with a query.

[TkDodo]

right, I see two ways of doing this:

1. You have one query, and do the merge in the queryFn, something like:

const useNews = () => {
  const queryClient = useQueryClient()
  return useQuery(
    'news',
    async () => {
      const { data } = await axios.get(...)
      const previousData = queryClient.getQueryData('news')
      return mergeDataSomehow(previousData, data)
    }
}

this is probably easiest, but I don't quite understand how this will work with refetches. Will the backend send the same data again when you re-fetch with the same timestamp, or is it stateful and knows that you've already retrieved this data? Or is your merge more than just an appending...

2. use two queries:

const newsQuery = useQuery('news', fetchNews)
const nextNews = useQuery('news', 'next', fetchNextNews, { enabled: Boolean(newsQuery.data) })

return mergeDataSomehow(newsQuery.data, nextNews.data)

not sure if that will work better though.

---

generally, when I'm reading that you have one fetch that returns 1000 rows - I don't think that's generally a good idea. What is the frontend going to do with so many rows? It will be slow to render, and then you need to use virtualization or windowing etc to not be slow.
When I think about a news endpoint, I would think about useInfiniteQuery, where I'll get e.g. 50 or 100 news on the first page, and then I can scroll down to get more data. react-query works very well with such a setup. But of course I don't know your business requirements :)

[tomyates]

Really appreciate your reply. I've going to give these techniques a go and I'll report back. I'm not actually fetching news stories, it's rows of business stats that I'm displaying in a searchable table. It maybe I will have to use an infinite loading solution but I wanted to explore this method first of all.
Thanks again.

[TkDodo]

regarding searchable table: I would really consider a backend search here. react-query works very well with that. I totally regret the one page we have where sorting / filtering is done on the frontend, and we deliver 2500 entries on page load. It takes forever to load, and then is junky on the frontend because of rendering issues. Not good :/

[tomyates]

I hear you. I'm using react-table and at the moment it seems to be handling the large data set pretty well. I've not tested on a low power machine though - I would rather do the heavy lifting on the server side, however, that involves getting a different team involved etc etc etc.

[TkDodo]

got you :)

[tomyates]

@TkDodo Just to update you, your first option worked well and achieved what I wanted so thank you.
However, in the end, the dataset was so much larger than expected I went for serverside pagination solution using react-query. Thanks again.