Source grouping interactions with fiber_limit for async data loader

[nickstanish]

I recently added async data loader to my rails application as mentioned here https://graphql-ruby.org/dataloader/async_dataloader and set a fiber_limit in hope to restrict how many database connections might be checked out simultaneously (I also increased my pool size after I had some errors where database connections could not be checked out within the limit)

However I could use some guidance on setting the fiber_limit value other than the minimum value of 4. I noticed that it is not 1:1 with the number of dataloader sources that can run, and when set too low it seems to unpredictably change a single batch load into n+1 queries that run serially.

class CustomAsyncDataloader < GraphQL::Dataloader::AsyncDataloader
  def get_fiber_variables
    vars = super
    vars[:connected_to] = {
      role: ActiveRecord::Base.current_role,
      shard: ActiveRecord::Base.current_shard,
      prevent_writes: ActiveRecord::Base.current_preventing_writes,
    }
    vars
  end

  def set_fiber_variables(vars)
    connection_config = vars.delete(:connected_to)
    ActiveRecord::Base.connecting_to(**connection_config)
    super
  end
end

use CustomAsyncDataloader, fiber_limit: ENV.fetch('RAILS_GQL_FIBER_LIMIT', 10).to_i

class Types::ViewerType < Types::BaseObject
    field :example_field, String do
      argument :key, String, required: false
    end

    def example_field(key: 'default')
      dataloader.with(Sources::SlowSource, key).load(SecureRandom.uuid)
    end
end

query {
  viewer {
    default: exampleField

    # simulate loading the same nested source for a connection type
    b1result1: exampleField(key: "batch1")
    b1result2: exampleField(key: "batch1")
    b1result3: exampleField(key: "batch1")
    b1result4: exampleField(key: "batch1")
    b1result5: exampleField(key: "batch1")
    b1result6: exampleField(key: "batch1")
    b1result7: exampleField(key: "batch1")
    b1result8: exampleField(key: "batch1")
    b1result9: exampleField(key: "batch1")
    b1result10: exampleField(key: "batch1")
    b1result11: exampleField(key: "batch1")
    b1result12: exampleField(key: "batch1")
    b1result13: exampleField(key: "batch1")
    b1result14: exampleField(key: "batch1")
    b1result15: exampleField(key: "batch1")
  }
}

I would expect that this performs 2 queries: one for the "default" batch and 1 for "batch1"
But instead we see this:

00:00:35.373351 Rails -- [default] Fetching ids: ["56015c91-35d3-4843-a194-f9c52c24a0ef"]
00:00:35.375195 Rails -- [batch1] Fetching ids: ["64acf353-cf27-4e93-a46d-364196bc5f13", "2521c561-9183-42cc-9ee5-bdefa5862388", "4db4d60e-8639-4978-bf7b-1b64b2894bd7", "98ade717-5e1d-4e77-8272-46edf771d050", "e64be3b6-6c7b-4789-8ec6-156d04cae73e", "bfd59cc0-0f23-4db2-83de-0191f212e220"]
00:00:37.378597 ActiveRecord::Base --    (2004.4ms)  select pg_sleep(2) 
00:00:37.409277 ActiveRecord::Base --    (2033.3ms)  select pg_sleep(2) 
00:00:37.412677 Rails -- [batch1] Fetching ids: ["270cd2e4-eb26-4e21-b9db-1b923058f7c5", "f9c17bdc-47ad-4f97-8b69-dcd142748787", "011c1986-42b9-4b36-9864-a3e16cbe3fe0", "61cb7178-5c30-4fac-a95b-bd534a89a9d2", "74c93d5f-1d3b-4e47-865e-1024ba97a51d", "c11259e8-48b2-4912-a4d9-ff05d89259fa", "080040cd-09b2-456a-8214-a5faac750765"]
00:00:39.442014 ActiveRecord::Base --    (2028.7ms)  select pg_sleep(2) 
00:00:39.444284 Rails -- [batch1] Fetching ids: ["b9caade4-cba1-4e08-a47e-a5ef8325d95b", "6ce90e58-af0c-46a4-b486-5f3bd319d549"]
00:00:41.463919 ActiveRecord::Base --    (2019.0ms)  select pg_sleep(2) 

The first batch for "default" works as expected but the second "batch" is split into 3 different groups which don't complete until the previous completed.

Increasing the fiber_limit makes this work as expected. I'm guessing this needs to be in M*N minimum to work well with M sources that might be batch loading N items?

Using:
graphql-pro (1.29.14)
graphql (2.5.16)
async (2.24.0)
rails (7.2.x)

[rmosolgo]

Hey, thanks for the detailed write-up and sorry about the confusion. You're right that fiber_limit doesn't correspond to the number of sources. It actually corresponds to the number of graphql fields that call dataloader...load and thereby put the current fiber on hold (via Fiber.yield). fiber_limit is the upper limit on how many paused execution fibers Dataloader will allow to accumulate before resolving those fibers by calling out to Dataloader sources.

If you want the query in the example to make two database queries, you'd need fiber_limit: 19, because of accounting for a top-level fiber and two (todo: why 2?) source resolution fibers:

graphql-ruby/lib/graphql/dataloader.rb

Lines 335 to 337 in 5065756

	total_fiber_limit -= 1 # deduct one fiber for `manager`
	# Deduct at least one fiber for sources
	jobs_fiber_limit = total_fiber_limit - 2

As far as recommended limits, I bet you could do 1000 pretty safely. The option was added because some (really) large queries where queuing up over 10,000 fibers and causing Ruby to run out of memory completely!

[nickstanish]

Okay that makes sense, memory wasn't my concern for setting it, and so I think my problem is misusing the fiber_limits thinking that it would help with predicting the number of database connections that will be used simultaneously, which didn't work as expected. Unfortunately even with fiber_limit: 200 I was still frequently running into ConnectionTimeoutErrors with 5 request threads (50 db pool and checkout_timeout: 3s). I'm sure this would be less of an issue if all the db queries were more performant or if these sources were primarily network requests (that don't require rate limiting).

Would it make sense to introduce a way that the Sources could opt-in/out of limits with Async::Semaphore? Maybe something like the source registers a group and then can configure limits for each group:

class ExampleSourceDb < GraphQL::Dataloader::Source
  def self.concurrency_key
     'database'
   end
   # ...
end

class ExampleSourceGithub < GraphQL::Dataloader::Source
  def self.concurrency_key
     'api.github.com'
   end
   # ...
end

use GraphQL::Dataloader::AsyncDataloader, concurrency_limits: { 'database' => 5, 'api.github.com' => 100 }

[rmosolgo]

I'm definitely open to supporting concurrency keys like that, but I don't have a vision for implementing it (or time to commit to it...!)

I was going to suggest using pooling at the ORM layer, but it sounds like you've already got ActiveRecord set up to allow 50 open connections and wait up to 3s for a connection to become available before raising an error, is that right? Or, are those 50 connections shared across up-to-5 active requests?

Another (easier...) possibility would be limit the number of active Source tasks. Currently, it uses fiber_limit: minus any active execution fibers. But we could accept a config and keep the number of concurrent Source tasks below that.