Webhooks fail when multiple inserts are done

[devdras]

Bug report

Describe the bug

I have a webhook configured to watch a table for inserts. On an insert a POST request should be fired to an API hosted on Vercel.
This works when one insert operation is carried out. If multiple inserts are carried out: .insert([{message: 1}, {message: 2}, {message: 3}]) then the webhooks aren't fired at all. If instead we carry out multiple inserts by calling the Supabase API multiple times then the webhook is fired for only some of the inserts.

To Reproduce

Steps to reproduce the behavior, please provide code snippets or a repository:

1. Configure a webhook on a table to watch for insert operations and call an external API

2. Use the Supabase JS library to perform multiple inserts like: let { data, error } = await supabase .from("example_table") .insert([{message: 1}, {message: 2}, {message: 3}, {message: 4}, {message: 5}, {message: 6}, etc ]) .select("message");

3. Check logs for external API to confirm it didn't receive any requests

4. See error

Something to note is that I was inserting 32 things at once.

Expected behavior

For each insert, I would expect the external API to be called. It would also help if there was a log in Supabase that showed what happens when the webhook is run.

Screenshots

If applicable, add screenshots to help explain your problem.

System information

• OS: N/A
• Browser (if applies): N/A
• Version of supabase-js: 2.0.0-rc.8
• Version of Node.js: v16.17.0 on desktop, 18.x on Vercel (per their docs)

Additional context

Add any other context about the problem here.

[danrasmuson]

Thank you for posting this. I'm receiving a similar error.

Surprisingly when I create 10 records at the same time it fires the function for each 10 records...

INSERT INTO public.profiles (id, first_name, last_name, stripe_customer_id, email)
SELECT u.id, NULL, NULL, NULL, u.email
FROM auth.users u
LEFT JOIN public.profiles p ON u.id = p.id
WHERE p.id IS NULL
LIMIT 10;

However, if I fire the function for 100 records...

INSERT INTO public.profiles (id, first_name, last_name, stripe_customer_id, email)
SELECT u.id, NULL, NULL, NULL, u.email
FROM auth.users u
LEFT JOIN public.profiles p ON u.id = p.id
WHERE p.id IS NULL
LIMIT 100;

It fails

[devdras]

Yup, basically the webhook behaviour is flakey. I haven't tried it recently. I wanted to use this to build an event driven architecture but can't without it being reliable. Ended up using QStash as a replacement for the webhook. Would love Supabase to have some kind of full featured event queue like SQS/Eventbridge/Service Bus.

[steve-chavez]

We've tested pg_net sending 1K inserts per second, this to a local nginx server though.

@devdras @danrasmuson Which webhook service are you using? Likely it's rate limiting the requests.

[danrasmuson]

We've tested pg_net sending 1K inserts per second, this to a local nginx server though.

@devdras @danrasmuson Which webhook service are you using? Likely it's rate limiting the requests.

Im using vercel with serverless functions

[matt-aitken]

We're experiencing the same problem. When inserting records in sequence all the webhooks are received. But if we insert 30 rows in parallel none of the webhooks send and the net/_http_response table is full of Timeout was reached errors.

[steve-chavez]

@matt-aitken Are you using vercel serverless functions too?

[lohrmann]

@steve-chavez We're able to reliably reproduce it based on the number of new entries that get bulk added to the table—have been in touch with your support team about this as well, but unable to resolve it. 

• The webhook trigger is calling an AWS Lambda function
• The lambda function does not record any invocations if the number of records inserted is larger than >150
  • This is an all or nothing behaviour. If we insert fewer records, all invocations happen, if we insert more records none of the invocations happen
  • It's not the case that only some of the invocations don't come through
  • It's not the case that the invocation reaches the Lambda function and the webhook times out before the function responds
• In supabase, the _http_response table only logs "Timeout was reached", nothing else
• We have tested this with different webhooks — including webhook.site. No invocations of the webhook are received

Happy to prepare a screen-recording or do a walk through if helpful.

[steve-chavez]

The lambda function does not record any invocations if the number of records inserted is larger than >150

@lohrmann That might be related to lambdas rate limiting https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html#api-requests

We're able to reliably reproduce it based on the number of new entries that get bulk added to the table

By default there are 200 requests per iteration https://github.com/supabase/pg_net/blob/master/src/worker.c#L479-L483

I'll make this parameter user configurable.

[steve-chavez]

Curl has:

curl --rate 2/s -O https://example.com/[1-100].jpg

https://everything.curl.dev/usingcurl/transfers/request-rate#examples

I think that should be the final solution here.

---

Note: the above is not available on libcurl.

[matt-aitken]

Sorry for the slow response on this. No, we're receiving webhooks to AWS EC2 (a long-running server).

There's nothing in our server logs where we receive requests from Supabase (when inserting many rows) and the net/_http_response table is full of Timeout was reached errors.

We (Trigger.dev) make it easy for our customers to subscribe to these Supabase database webhooks and quite a few people are hitting this problem now.

[ahanusek]

We're experiencing the same problem. When inserting records in sequence all the webhooks are received. But if we insert 30 rows in parallel none of the webhooks send and the net/_http_response table is full of Timeout was reached errors.

I have the same issue. I'm using Google Cloud Functions v2 (Cloud Run). Table is full of Timeout was reached, but in Google Cloud Monitoring don't have any logs.