Limiting Parallelism for useQueries

[wraybowling]

The Scenario

We've got a bar chart where each day fetches data from a unique api/database endpoint. When the user selects a range for 365+ days, it takes exponentially longer to load.

The Problem

Under the hood, we're using useQueries to fetch and cache all the requests, and all of the API requests are fired off together. At high numbers, this has the effect of a DDOS, absolutely slaughtering the server.

My Research

I've gone spelunking for answers on "debouncing" and "throttling" TanStack Query, and all of the existing threads are concerned with useQuery and debouncing parameters being sent to the hook. That's a very different problem.

React Query does offer queryClient.isFetching which "returns an integer representing how many queries, if any, in the cache are currently fetching" but we don't have any control over the size of the queue. Our team understands that a back-end API rate limit is the ideal solution, but I still think that we should be able to tune the client's performance.

Feature Request?

I would prefer a solution that allows for a max value: an integer that sets how many queries are allowed to be enabled at once. At this time, I don't see many additional options for useQueries that helps manage the collection of queries as a whole. useQueries is designed to make many queries, but it offers no features to limit just how many will run at the same time. It would be even fancier if we could set a hard limit all the way up at the queryClient level how many concurrent fetches can be run.

Workaround Ideas

I've so far only thought of a couple workarounds that would only allow for 1 query to run at a time. In both cases, I'd use 1 running query to increment an index upon onSuccess() or maybe onSettled()

1. adding a "cursor" above useQueries that sets enabled to true
2. using useInfiniteQuery and externally requesting the next page until we hit the entire date range. This option seems to have the most safety built in if/when things change

[louis-young]

Hi 👋

It sounds like the issue may stem from an API design flaw in my opinion. It seems like you're making N requests (N being the number of days in the range) for a given date range when I'd imagine this could be done in a single request that specifies the interesting date range. Is this something you could consider?

I agree with your sentiment that rate limiting is the responsibility of the API. Also, remember that people can make requests to your API without using your client/application.

Does this help? 🙂

[wraybowling]

In this particular case, we aren't able to collect the complete range in a single request. The database queries are very expensive to calculate the results for. So we decided to distribute the load by using pagination over time. Pulling the data for 1 day isn't so bad, but requesting all the days at once undermines that effort. I looked into what the limits are in the browser. Chrome for example limits to 6 calls at a time. I'm being asked to come up with a way to limit the requests to fewer than that.

Again, yes, this should be something handled on the back-end. Other threads on similar issues tended to ONLY focus on useQuery but I saw an opportunity for improvement with useQueries which is designed to make many requests at once.

I'm now on my second day of research and I'll be seeing about doing something at the queryFn/fetch level and then I believe Query should be able to handle it fine.

[louis-young]

That's true for HTTP/1. HTTP/2 implements request multiplexing and can theoretically handle unlimited concurrent requests, although that number is still limited by browsers.

I think I agree that this is something you likely want to handle yourself through some composition of dependant queries 👍

[wraybowling]

I found a solution that works for the complex application I work on, and I want to make sure I share what I found for the next person, especially since I didn't find anyone else talking about how to do this sort of thing. My project is using React, but I think these should be usable across frameworks.

Here's a list of what you'll need installed:

1. TanStack Query
2. async-mutex

A time-paginated useQueries function

This makes use of a function getDaysArray to split up a start_time and end_time into a list of dates. useQueries uses the dates mapped out so that each queryKey uses the start and end of each day. The timezone is included in the query to ensure timezone differences obtain a unique queryKey. Additional options have been added so that today's data is allowed to refresh itself, but all other times are set to staleTime: Infinity.

import { useQueries } from 'react-query';

function dateIsToday(dateParameter) {
  var today = new Date();
  return dateParameter.getDate() === today.getDate() &&
    dateParameter.getMonth() === today.getMonth() &&
    dateParameter.getFullYear() === today.getFullYear();
}

function getDaysArray(start, end) {
  for (var arr = [], dt = new Date(start); dt <= new Date(end); dt.setDate(dt.getDate() + 1)) {
    arr.push(new Date(dt));
  }
  return arr;
}

export default function useChartData({ start_time, end_time }) {
  let dates = getDaysArray(start_time, end_time);

  return useQueries(
    dates.map(date => ({
      queryKey: [
        'chart',
        {
          query: {
            start_time: new Date(date.setHours(0, 0, 0, 0)).toISOString(),
            end_time: new Date(date.setHours(23, 59, 59, 999)).toISOString(),
            timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
          },
        },
      ],
      queryFn: fetchChartData,
      enabled: start_time !== undefined && end_time !== undefined,
      refetchOnWindowFocus: dateIsToday(date),
      staleTime: dateIsToday(date) ? 3 * 60 * 1000 : Infinity,
    })),
  );

A fetch function that uses a mutex and query cancellation

The best solution we came up with lives in the queryFn, not in some state above React Query attempting to manipulate enabled. It has two parts:

1. a mutex or semaphore that forces the fetch to wait for the thread to unlock. The JavaScript library I used, async-mutex supports both single-thread (mutex) or n-count threads (semaphore) by leveraging Promises. Seen below, we're using the mutex to limit all fetches to run 1 at a time. Each queryFn call will kick the above Query into a loading state and push itself into a queue. In that time, especially if the calls are very slow to come back, that queue can get full of things the user no longer needs but are still pending which is why we also added...
2. Query cancellation using the signal object. The AbortSignal passed is a simple {aborted: true} object that is actually supported by fetch() and will cancel the download completely. TanStack Query "doesn't cancel queries by default" but does pass down the signal object, ready to use. By making use of it in your fetch(), you are "enabling" it. It's pretty awesome! As we change the date range, Query marks those queryKeys as inactive and passes the {aborted: true} to the fetch() for any still in the mutex queue. As their turn comes up, they all immediately stop and free up the locks to fresh queries.

import { Mutex } from 'async-mutex';
const chartMutex = new Mutex();

export async function fetchChartData({ queryKey, signal }) {
  const [, { query }] = queryKey;

  const queryString = new URLSearchParams(query).toString();

  const data = await chartMutex.runExclusive(async () => {
    return await fetch(`/api/d2c/tenant/${tenant_id}/task-histogram?${queryString}`, { signal });
  });

  if (!data.ok) {
    throw new Error('Failed to get histogram data', data.statusText, data.status);
  }

  const response = await data.json();
  return response;
}

[mouhannad-sh]

Thank you for posting this, I'm sure you saved a lot of time for a lot of people 🥇

[pyramid-scheme-ceo]

Just adding this here in case someone else needs it in the future, I had a very similar problem but still wanted a few requests to be in flight at the same time (rather than one at a time), so came up with this hook:

type QueuedTask<T> = {
    task: () => Promise<T>;
    resolve: (value: T | PromiseLike<T>) => void;
    processing: boolean;
};

export const useConcurrentTaskQueue = <T>(maxConcurrency: number) => {
    const [taskQueue, setTaskQueue] = useState<QueuedTask<T>[]>([]);

    useEffect(() => {
        if (taskQueue.filter(t => !t.processing).length > 0 && taskQueue.filter(t => t.processing).length < maxConcurrency) {
            const nextTask = taskQueue.find(t => !t.processing);
            nextTask.processing = true;
            setTaskQueue(q => [...q.filter(t => t !== nextTask), nextTask]);

            nextTask.task().then(result => {
                nextTask.resolve(result);
            }).finally(() => {
                setTaskQueue(q => q.filter(i => i !== nextTask));
            });
        }
    }, [taskQueue, maxConcurrency]);

    const executeConcurrentTask = useCallback(async (task: () => Promise<T>) => new Promise<T>(resolve => {
        setTaskQueue(q => [...q, { task, resolve, processing: false }]);
    }), []);

    return { executeConcurrentTask };
};

Usage:

// Run up to 5 API requests at a time
const { executeConcurrentTask } executeConcurrentTask = useConcurrentTaskQueue(5);

const fetchData = useCallback((date: Date) => executeConcurrentTask(
  () => apiService.getResults(date, some, other, parameters)
), [executeConcurrentTask, some, other, parameters]);

const queries = datesToFetch.map(date => ({
  queryKey: ['results', { date.getTime() }],
  queryFn: () => fetchData(date),
}));

const results = useQueries({ queries });
const loading = results.some(r => r.isLoading);

...

[wraybowling]

Neat seeing your solution. I achieved a similar result using the Semaphore from async-mutex. https://github.com/DirtyHairy/async-mutex#semaphore-api but you're right that similar things can be achieved with a simple array. One thing I would urge you to add is the signal to cancel requests. We found after a few months of use that the thing that helped the most in adding a mutex/semaphore was that while the user was moving around swiftly, making sure that these heavy-weight requests were cancelled to free up slots did perhaps even more for the perceived loading times than the limits alone.