Expensive initial list API call, refetch interval, inexpensive list API call with updated_since parameter

[jcready]

I have a somewhat expensive (read: large) API call which will pull in a list of entities. Each entity has an updated_at property and the API call allows the client to request only entities which have been updated_since. So ideally the workflow would be:

1. Fetch initial list
2. On some interval, find the most recent (largest) updated_at of all our entities in the cache
3. Then fetch from the same API but this time using the value from 2 as the updated_since parameter
4. Use that result to update the cache

Maybe I missed something from the docs/examples, but I cannot for the life of me figure out how to make this work. The best I can figure I would need two useQuery hook, one for the original fetch and another with a refetchInterval and then somewhere (inside the queryFn, inside the onSuccess, 🤷) I then need to iterate through a shallow clone of the cached array (using the query client and the key from the first useQuery), find the index in that array where the entity matches (by a unique ID) one of the entities from my polling hook response and splice that into the existing array or push the new entity into the array. Then use the query client to again set query data with the new array.

Is this the best way to do this or am I missing something?

[tnoetzel]

@jcready - FWIW, I've been working on a similar problem, and this is what I came up with

Not sure it's the best approach, though... feels like this should be built into React Query, but best I can tell it's not?

[TkDodo]

On mobile now but it's just a matter of how you setup your queryFn:

queryFn: async({ queryKey }) => {
  const data = queryClient.getQueryData(queryKey)
  const updatedSince = findLargestTimestamp(data)
  
  const newData = await fetch(...)

  return append(data, newData)
}

[jcready]

This somewhat works, but because the queryFn is accessing the cached data itself it means that when you invalidateQueries() from the query client it no longer has the desired effect because when we re-run the queryFn it still looks at the cache. We can always run removeQueries(), but that means the UI also loses access to the cached data while the queryFn is running. Not to mention the structural sharing aspect.

Edit: It looks like I can just getQueryData(queryKey, { stale: false }).

So yeah, this seems to work.