Skip to content

Conversation

steve-chavez
Copy link
Member

@steve-chavez steve-chavez commented Aug 29, 2025

Currently memory contexts are used to work around the short memory context of SPI blocks (SPI_connect and SPI_finish), which are used inside some functions. For example:

pg_net/src/core.c

Lines 139 to 170 in 314d00e

uint64 delete_expired_responses(char *ttl, int batch_size){
SPI_connect();
int ret_code = SPI_execute_with_args("\
WITH\
rows AS (\
SELECT ctid\
FROM net._http_response\
WHERE created < now() - $1\
ORDER BY created\
LIMIT $2\
)\
DELETE FROM net._http_response r\
USING rows WHERE r.ctid = rows.ctid",
2,
(Oid[]){INTERVALOID, INT4OID},
(Datum[]){
DirectFunctionCall3(interval_in, CStringGetDatum(ttl), ObjectIdGetDatum(InvalidOid), Int32GetDatum(-1))
, Int32GetDatum(batch_size)
}, NULL, false, 0);
uint64 affected_rows = SPI_processed;
if (ret_code != SPI_OK_DELETE)
{
ereport(ERROR, errmsg("Error expiring response table rows: %s", SPI_result_code_string(ret_code)));
}
SPI_finish();
return affected_rows;
}

pg_net/src/core.c

Lines 172 to 227 in 314d00e

uint64 consume_request_queue(CURLM *curl_mhandle, int batch_size, MemoryContext curl_memctx){
SPI_connect();
int ret_code = SPI_execute_with_args("\
WITH\
rows AS (\
SELECT id\
FROM net.http_request_queue\
ORDER BY id\
LIMIT $1\
)\
DELETE FROM net.http_request_queue q\
USING rows WHERE q.id = rows.id\
RETURNING q.id, q.method, q.url, timeout_milliseconds, array(select key || ': ' || value from jsonb_each_text(q.headers)), q.body",
1,
(Oid[]){INT4OID},
(Datum[]){Int32GetDatum(batch_size)},
NULL, false, 0);
if (ret_code != SPI_OK_DELETE_RETURNING)
ereport(ERROR, errmsg("Error getting http request queue: %s", SPI_result_code_string(ret_code)));
uint64 affected_rows = SPI_processed;
for (size_t j = 0; j < affected_rows; j++) {
bool tupIsNull = false;
int64 id = DatumGetInt64(SPI_getbinval(SPI_tuptable->vals[j], SPI_tuptable->tupdesc, 1, &tupIsNull));
EREPORT_NULL_ATTR(tupIsNull, id);
int32 timeout_milliseconds = DatumGetInt32(SPI_getbinval(SPI_tuptable->vals[j], SPI_tuptable->tupdesc, 4, &tupIsNull));
EREPORT_NULL_ATTR(tupIsNull, timeout_milliseconds);
Datum method = SPI_getbinval(SPI_tuptable->vals[j], SPI_tuptable->tupdesc, 2, &tupIsNull);
EREPORT_NULL_ATTR(tupIsNull, method);
Datum url = SPI_getbinval(SPI_tuptable->vals[j], SPI_tuptable->tupdesc, 3, &tupIsNull);
EREPORT_NULL_ATTR(tupIsNull, url);
NullableDatum headersBin = {
.value = SPI_getbinval(SPI_tuptable->vals[j], SPI_tuptable->tupdesc, 5, &tupIsNull),
.isnull = tupIsNull
};
NullableDatum bodyBin = {
.value = SPI_getbinval(SPI_tuptable->vals[j], SPI_tuptable->tupdesc, 6, &tupIsNull),
.isnull = tupIsNull
};
init_curl_handle(curl_mhandle, curl_memctx, id, url, bodyBin, headersBin, method, timeout_milliseconds);
}
SPI_finish();
return affected_rows;
}

But it's unnecessary to do that, instead we can use a single SPI block in the worker loop and share the same memory context for the curl operations.

This makes the code simpler and makes memory allocations more visible.

Note

The implementation is basically reordering function calls, there's no change of logic.

@steve-chavez steve-chavez requested a review from za-arthur August 29, 2025 00:59
@steve-chavez steve-chavez marked this pull request as ready for review August 29, 2025 00:59
@steve-chavez steve-chavez requested review from laurenceisla and removed request for za-arthur August 29, 2025 17:33
Currently memory contexts are used to work around the short memory
context of SPI blocks (SPI_connect and SPI_finish), which are used
inside some functions. For example:

https://github.com/supabase/pg_net/blob/314d00e34e22378ca2b733838f96b4c179d727a9/src/core.c#L139-L170

https://github.com/supabase/pg_net/blob/314d00e34e22378ca2b733838f96b4c179d727a9/src/core.c#L172-L227

But it's unnecessary to do that, instead we can use a single SPI block
in the worker loop and share the same memory context for the curl
operations.

This makes the code simpler and makes memory allocations more visible.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants