update queue_jobs errors in logs #5545
Description
I have recently upgraded to matrix authentication service v. 1.12.0
# mas-cli -V
mas-cli 1.12.0After a reboot I noticed these new postgresql errors in my syslog, complaining about update statements violating table constraints.
2026-03-07T13:55:00.659298+00:00 anon matrix-authentication-service: 2026-03-07T13:55:00.659193Z INFO mas_tasks::new_queue:728 worker-run-loop-42 - One scheduled job marked as available
2026-03-07T13:55:00.660887+00:00 anon matrix-authentication-service: 2026-03-07T13:55:00.660826Z INFO mas_tasks::new_queue:910 job-cleanup-queue-jobs-44 - Running job job.id=01KK479FB7AHEJZH524KKC804M job.queue.name=cleanup-queue-jobs job.attempt=0 job.timeout=600s
2026-03-07T13:55:00.673877+00:00 anon postgres[8633]: [7-1] ERROR: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"
2026-03-07T13:55:00.673889+00:00 anon postgres[8633]: [7-2] DETAIL: Key (queue_job_id)=(019c1949-ea26-1724-5463-a546413e24cb) is still referenced from table "queue_schedules".
2026-03-07T13:55:00.673895+00:00 anon postgres[8633]: [7-3] STATEMENT:
2026-03-07T13:55:00.673902+00:00 anon postgres[8633]: [7-4] #011 WITH to_delete AS (
2026-03-07T13:55:00.673907+00:00 anon postgres[8633]: [7-5] #011 SELECT queue_job_id
2026-03-07T13:55:00.673912+00:00 anon postgres[8633]: [7-6] #011 FROM queue_jobs
2026-03-07T13:55:00.673918+00:00 anon postgres[8633]: [7-7] #011 WHERE (status = 'completed' OR status = 'failed')
2026-03-07T13:55:00.673923+00:00 anon postgres[8633]: [7-8] #011 AND ($1::uuid IS NULL OR queue_job_id > $1)
2026-03-07T13:55:00.673928+00:00 anon postgres[8633]: [7-9] #011 AND queue_job_id <= $2
2026-03-07T13:55:00.673933+00:00 anon postgres[8633]: [7-10] #011 ORDER BY queue_job_id
2026-03-07T13:55:00.673938+00:00 anon postgres[8633]: [7-11] #011 LIMIT $3
2026-03-07T13:55:00.673943+00:00 anon postgres[8633]: [7-12] #011 )
2026-03-07T13:55:00.673948+00:00 anon postgres[8633]: [7-13] #011 DELETE FROM queue_jobs
2026-03-07T13:55:00.673953+00:00 anon postgres[8633]: [7-14] #011 USING to_delete
2026-03-07T13:55:00.673992+00:00 anon postgres[8633]: [7-15] #011 WHERE queue_jobs.queue_job_id = to_delete.queue_job_id
2026-03-07T13:55:00.673998+00:00 anon postgres[8633]: [7-16] #011 RETURNING queue_jobs.queue_job_id
2026-03-07T13:55:00.674002+00:00 anon postgres[8633]: [7-17] #011
2026-03-07T13:55:00.674323+00:00 anon matrix-authentication-service: 2026-03-07T13:55:00.674049Z ERROR mas_storage_pg::queue::job:449 job-cleanup-queue-jobs-44 - error=error returned from database: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"
2026-03-07T13:55:00.674333+00:00 anon matrix-authentication-service: 2026-03-07T13:55:00.674095Z WARN mas_tasks::new_queue:960 job-cleanup-queue-jobs-44 - Job failed, will retry in 5s [polls: 4, cpu: 0.6ms, elapsed: 13.3ms] error=error returned from database: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules" error.sources=[update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"] job.id=01KK479FB7AHEJZH524KKC804M job.queue.name=cleanup-queue-jobs job.attempt=0
2026-03-07T13:55:03.680885+00:00 anon matrix-authentication-service: 2026-03-07T13:55:03.680825Z INFO http.server.response GET-52 - "GET /.well-known/openid-configuration HTTP/1.1" 200 OK "Synapse/1.148.0" [polls: 0, cpu: 0.0ms, elapsed: 0.7ms]
2026-03-07T13:55:03.701376+00:00 anon matrix-authentication-service: 2026-03-07T13:55:03.701290Z INFO http.server.response POST-55 - "POST /oauth2/introspect HTTP/1.1" 200 OK "Synapse/1.148.0" [polls: 12, cpu: 1.2ms, elapsed: 13.2ms]
2026-03-07T13:55:06.486611+00:00 anon matrix-authentication-service: 2026-03-07T13:55:06.486533Z INFO mas_tasks::new_queue:728 worker-run-loop-57 - One scheduled job marked as available
2026-03-07T13:55:06.488373+00:00 anon matrix-authentication-service: 2026-03-07T13:55:06.488191Z INFO mas_tasks::new_queue:910 job-cleanup-queue-jobs-59 - Running job job.id=01KK498ZE3YWP8RCT80XXAV3TK job.queue.name=cleanup-queue-jobs job.attempt=1 job.timeout=600s
2026-03-07T13:55:06.494567+00:00 anon postgres[8633]: [8-1] ERROR: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"
2026-03-07T13:55:06.494577+00:00 anon postgres[8633]: [8-2] DETAIL: Key (queue_job_id)=(019c1949-ea26-1724-5463-a546413e24cb) is still referenced from table "queue_schedules".
2026-03-07T13:55:06.494582+00:00 anon postgres[8633]: [8-3] STATEMENT:
2026-03-07T13:55:06.494588+00:00 anon postgres[8633]: [8-4] #011 WITH to_delete AS (
2026-03-07T13:55:06.494593+00:00 anon postgres[8633]: [8-5] #011 SELECT queue_job_id
2026-03-07T13:55:06.494598+00:00 anon postgres[8633]: [8-6] #011 FROM queue_jobs
2026-03-07T13:55:06.494604+00:00 anon postgres[8633]: [8-7] #011 WHERE (status = 'completed' OR status = 'failed')
2026-03-07T13:55:06.494609+00:00 anon postgres[8633]: [8-8] #011 AND ($1::uuid IS NULL OR queue_job_id > $1)
2026-03-07T13:55:06.494614+00:00 anon postgres[8633]: [8-9] #011 AND queue_job_id <= $2
2026-03-07T13:55:06.494619+00:00 anon postgres[8633]: [8-10] #011 ORDER BY queue_job_id
2026-03-07T13:55:06.494624+00:00 anon postgres[8633]: [8-11] #011 LIMIT $3
2026-03-07T13:55:06.494900+00:00 anon postgres[8633]: [8-12] #011 )
2026-03-07T13:55:06.494916+00:00 anon postgres[8633]: [8-13] #011 DELETE FROM queue_jobs
2026-03-07T13:55:06.494921+00:00 anon postgres[8633]: [8-14] #011 USING to_delete
2026-03-07T13:55:06.494926+00:00 anon postgres[8633]: [8-15] #011 WHERE queue_jobs.queue_job_id = to_delete.queue_job_id
2026-03-07T13:55:06.494931+00:00 anon postgres[8633]: [8-16] #011 RETURNING queue_jobs.queue_job_id
2026-03-07T13:55:06.494935+00:00 anon postgres[8633]: [8-17] #011
2026-03-07T13:55:06.495042+00:00 anon matrix-authentication-service: 2026-03-07T13:55:06.495000Z ERROR mas_storage_pg::queue::job:449 job-cleanup-queue-jobs-59 - error=error returned from database: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"
2026-03-07T13:55:06.495104+00:00 anon matrix-authentication-service: 2026-03-07T13:55:06.495074Z WARN mas_tasks::new_queue:960 job-cleanup-queue-jobs-59 - Job failed, will retry in 10s [polls: 3, cpu: 0.1ms, elapsed: 6.9ms] error=error returned from database: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules" error.sources=[update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"] job.id=01KK498ZE3YWP8RCT80XXAV3TK job.queue.name=cleanup-queue-jobs job.attempt=1
2026-03-07T13:55:16.614926+00:00 anon matrix-authentication-service: 2026-03-07T13:55:16.614844Z INFO mas_tasks::new_queue:728 worker-run-loop-70 - One scheduled job marked as available
2026-03-07T13:55:16.616623+00:00 anon matrix-authentication-service: 2026-03-07T13:55:16.616545Z INFO mas_tasks::new_queue:910 job-cleanup-queue-jobs-72 - Running job job.id=01KK49954005AY89W7T2978K3F job.queue.name=cleanup-queue-jobs job.attempt=2 job.timeout=600s
2026-03-07T13:55:16.622793+00:00 anon postgres[8633]: [9-1] ERROR: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"
2026-03-07T13:55:16.622803+00:00 anon postgres[8633]: [9-2] DETAIL: Key (queue_job_id)=(019c1949-ea26-1724-5463-a546413e24cb) is still referenced from table "queue_schedules".
2026-03-07T13:55:16.622808+00:00 anon postgres[8633]: [9-3] STATEMENT:
2026-03-07T13:55:16.622814+00:00 anon postgres[8633]: [9-4] #011 WITH to_delete AS (
2026-03-07T13:55:16.622819+00:00 anon postgres[8633]: [9-5] #011 SELECT queue_job_id
2026-03-07T13:55:16.622824+00:00 anon postgres[8633]: [9-6] #011 FROM queue_jobs
2026-03-07T13:55:16.622829+00:00 anon postgres[8633]: [9-7] #011 WHERE (status = 'completed' OR status = 'failed')
2026-03-07T13:55:16.622834+00:00 anon postgres[8633]: [9-8] #011 AND ($1::uuid IS NULL OR queue_job_id > $1)
2026-03-07T13:55:16.622839+00:00 anon postgres[8633]: [9-9] #011 AND queue_job_id <= $2
2026-03-07T13:55:16.622844+00:00 anon postgres[8633]: [9-10] #011 ORDER BY queue_job_id
2026-03-07T13:55:16.622849+00:00 anon postgres[8633]: [9-11] #011 LIMIT $3
2026-03-07T13:55:16.622853+00:00 anon postgres[8633]: [9-12] #011 )
2026-03-07T13:55:16.622858+00:00 anon postgres[8633]: [9-13] #011 DELETE FROM queue_jobs
2026-03-07T13:55:16.622863+00:00 anon postgres[8633]: [9-14] #011 USING to_delete
2026-03-07T13:55:16.622882+00:00 anon postgres[8633]: [9-15] #011 WHERE queue_jobs.queue_job_id = to_delete.queue_job_id
2026-03-07T13:55:16.622887+00:00 anon postgres[8633]: [9-16] #011 RETURNING queue_jobs.queue_job_id
2026-03-07T13:55:16.622905+00:00 anon postgres[8633]: [9-17] #011
2026-03-07T13:55:16.623162+00:00 anon matrix-authentication-service: 2026-03-07T13:55:16.622938Z ERROR mas_storage_pg::queue::job:449 job-cleanup-queue-jobs-72 - error=error returned from database: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"
2026-03-07T13:55:16.623173+00:00 anon matrix-authentication-service: 2026-03-07T13:55:16.622979Z WARN mas_tasks::new_queue:960 job-cleanup-queue-jobs-72 - Job failed, will retry in 20s [polls: 3, cpu: 0.1ms, elapsed: 6.5ms] error=error returned from database: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules" error.sources=[update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"] job.id=01KK49954005AY89W7T2978K3F job.queue.name=cleanup-queue-jobs job.attempt=2
2026-03-07T13:55:20.066655+00:00 anon matrix-authentication-service: 2026-03-07T13:55:20.066423Z INFO mas_handlers::activity_tracker::worker:246 Flushing 1 activity records to the database
2026-03-07T13:55:20.770762+00:00 anon matrix-authentication-service: 2026-03-07T13:55:20.770678Z INFO mas_tasks::new_queue:535 worker-run-loop-77 - Sending heartbeat
2026-03-07T13:55:21.253285+00:00 anon matrix-authentication-service: 2026-03-07T13:55:21.253203Z INFO http.server.response GET-81 - "GET /health HTTP/1.1" 200 OK "-" [polls: 3, cpu: 0.1ms, elapsed: 0.5ms]
2026-03-07T13:55:29.531183+00:00 anon matrix-authentication-service: 2026-03-07T13:55:29.531099Z INFO http.server.response GET-92 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.3ms]
2026-03-07T13:55:36.963330+00:00 anon matrix-authentication-service: 2026-03-07T13:55:36.963271Z INFO mas_tasks::new_queue:728 worker-run-loop-99 - One scheduled job marked as available
2026-03-07T13:55:36.965347+00:00 anon matrix-authentication-service: 2026-03-07T13:55:36.965290Z INFO mas_tasks::new_queue:910 job-cleanup-queue-jobs-101 - Running job job.id=01KK499F0FY1HVW0STCB6C5N50 job.queue.name=cleanup-queue-jobs job.attempt=3 job.timeout=600s
2026-03-07T13:55:36.971625+00:00 anon postgres[8633]: [10-1] ERROR: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"
2026-03-07T13:55:36.971636+00:00 anon postgres[8633]: [10-2] DETAIL: Key (queue_job_id)=(019c1949-ea26-1724-5463-a546413e24cb) is still referenced from table "queue_schedules".
2026-03-07T13:55:36.971642+00:00 anon postgres[8633]: [10-3] STATEMENT:
2026-03-07T13:55:36.971647+00:00 anon postgres[8633]: [10-4] #011 WITH to_delete AS (
2026-03-07T13:55:36.971654+00:00 anon postgres[8633]: [10-5] #011 SELECT queue_job_id
2026-03-07T13:55:36.971659+00:00 anon postgres[8633]: [10-6] #011 FROM queue_jobs
2026-03-07T13:55:36.971664+00:00 anon postgres[8633]: [10-7] #011 WHERE (status = 'completed' OR status = 'failed')
2026-03-07T13:55:36.971669+00:00 anon postgres[8633]: [10-8] #011 AND ($1::uuid IS NULL OR queue_job_id > $1)
2026-03-07T13:55:36.971674+00:00 anon postgres[8633]: [10-9] #011 AND queue_job_id <= $2
2026-03-07T13:55:36.971679+00:00 anon postgres[8633]: [10-10] #011 ORDER BY queue_job_id
2026-03-07T13:55:36.971684+00:00 anon postgres[8633]: [10-11] #011 LIMIT $3
2026-03-07T13:55:36.971688+00:00 anon postgres[8633]: [10-12] #011 )
2026-03-07T13:55:36.971703+00:00 anon postgres[8633]: [10-13] #011 DELETE FROM queue_jobs
2026-03-07T13:55:36.971708+00:00 anon postgres[8633]: [10-14] #011 USING to_delete
2026-03-07T13:55:36.971714+00:00 anon postgres[8633]: [10-15] #011 WHERE queue_jobs.queue_job_id = to_delete.queue_job_id
2026-03-07T13:55:36.971718+00:00 anon postgres[8633]: [10-16] #011 RETURNING queue_jobs.queue_job_id
2026-03-07T13:55:36.971723+00:00 anon postgres[8633]: [10-17] #011
2026-03-07T13:55:36.971815+00:00 anon matrix-authentication-service: 2026-03-07T13:55:36.971770Z ERROR mas_storage_pg::queue::job:449 job-cleanup-queue-jobs-101 - error=error returned from database: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"
2026-03-07T13:55:36.971863+00:00 anon matrix-authentication-service: 2026-03-07T13:55:36.971819Z WARN mas_tasks::new_queue:960 job-cleanup-queue-jobs-101 - Job failed, will retry in 40s [polls: 3, cpu: 0.1ms, elapsed: 6.7ms] error=error returned from database: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules" error.sources=[update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"] job.id=01KK499F0FY1HVW0STCB6C5N50 job.queue.name=cleanup-queue-jobs job.attempt=3
2026-03-07T13:55:37.803705+00:00 anon matrix-authentication-service: 2026-03-07T13:55:37.803596Z INFO http.server.response GET-106 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.3ms]
2026-03-07T13:55:46.106926+00:00 anon matrix-authentication-service: 2026-03-07T13:55:46.106843Z INFO http.server.response GET-118 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.3ms]
2026-03-07T13:55:54.343933+00:00 anon matrix-authentication-service: 2026-03-07T13:55:54.343834Z INFO http.server.response GET-129 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.3ms]
2026-03-07T13:56:02.771342+00:00 anon matrix-authentication-service: 2026-03-07T13:56:02.771291Z INFO http.server.response GET-140 - "GET /health HTTP/1.1" 200 OK "-" [polls: 1, cpu: 0.1ms, elapsed: 0.3ms]
2026-03-07T13:56:11.008943+00:00 anon matrix-authentication-service: 2026-03-07T13:56:11.008857Z INFO http.server.response GET-151 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.2ms]
2026-03-07T13:56:17.988446+00:00 anon matrix-authentication-service: 2026-03-07T13:56:17.988340Z INFO mas_tasks::new_queue:728 worker-run-loop-157 - One scheduled job marked as available
2026-03-07T13:56:17.990230+00:00 anon matrix-authentication-service: 2026-03-07T13:56:17.989962Z INFO mas_tasks::new_queue:910 job-cleanup-queue-jobs-159 - Running job job.id=01KK49A2WCBRBHNFTKQA0TVE79 job.queue.name=cleanup-queue-jobs job.attempt=4 job.timeout=600s
2026-03-07T13:56:17.996198+00:00 anon postgres[8633]: [11-1] ERROR: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"
2026-03-07T13:56:17.996209+00:00 anon postgres[8633]: [11-2] DETAIL: Key (queue_job_id)=(019c1949-ea26-1724-5463-a546413e24cb) is still referenced from table "queue_schedules".
2026-03-07T13:56:17.996215+00:00 anon postgres[8633]: [11-3] STATEMENT:
2026-03-07T13:56:17.996221+00:00 anon postgres[8633]: [11-4] #011 WITH to_delete AS (
2026-03-07T13:56:17.996226+00:00 anon postgres[8633]: [11-5] #011 SELECT queue_job_id
2026-03-07T13:56:17.996231+00:00 anon postgres[8633]: [11-6] #011 FROM queue_jobs
2026-03-07T13:56:17.996237+00:00 anon postgres[8633]: [11-7] #011 WHERE (status = 'completed' OR status = 'failed')
2026-03-07T13:56:17.996242+00:00 anon postgres[8633]: [11-8] #011 AND ($1::uuid IS NULL OR queue_job_id > $1)
2026-03-07T13:56:17.996246+00:00 anon postgres[8633]: [11-9] #011 AND queue_job_id <= $2
2026-03-07T13:56:17.996251+00:00 anon postgres[8633]: [11-10] #011 ORDER BY queue_job_id
2026-03-07T13:56:17.996256+00:00 anon postgres[8633]: [11-11] #011 LIMIT $3
2026-03-07T13:56:17.996260+00:00 anon postgres[8633]: [11-12] #011 )
2026-03-07T13:56:17.996265+00:00 anon postgres[8633]: [11-13] #011 DELETE FROM queue_jobs
2026-03-07T13:56:17.996270+00:00 anon postgres[8633]: [11-14] #011 USING to_delete
2026-03-07T13:56:17.996275+00:00 anon postgres[8633]: [11-15] #011 WHERE queue_jobs.queue_job_id = to_delete.queue_job_id
2026-03-07T13:56:17.996280+00:00 anon postgres[8633]: [11-16] #011 RETURNING queue_jobs.queue_job_id
2026-03-07T13:56:17.996284+00:00 anon postgres[8633]: [11-17] #011
2026-03-07T13:56:17.996591+00:00 anon matrix-authentication-service: 2026-03-07T13:56:17.996353Z ERROR mas_storage_pg::queue::job:449 job-cleanup-queue-jobs-159 - error=error returned from database: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"
2026-03-07T13:56:17.996601+00:00 anon matrix-authentication-service: 2026-03-07T13:56:17.996407Z WARN mas_tasks::new_queue:960 job-cleanup-queue-jobs-159 - Job failed, will retry in 80s [polls: 3, cpu: 0.1ms, elapsed: 6.5ms] error=error returned from database: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules" error.sources=[update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"] job.id=01KK49A2WCBRBHNFTKQA0TVE79 job.queue.name=cleanup-queue-jobs job.attempt=4
2026-03-07T13:56:19.415061+00:00 anon matrix-authentication-service: 2026-03-07T13:56:19.414957Z INFO http.server.response GET-165 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.3ms]
2026-03-07T13:56:21.020591+00:00 anon matrix-authentication-service: 2026-03-07T13:56:21.020490Z INFO mas_tasks::new_queue:535 worker-run-loop-166 - Sending heartbeat
2026-03-07T13:56:27.939474+00:00 anon matrix-authentication-service: 2026-03-07T13:56:27.939125Z INFO http.server.response GET-176 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.2ms]
2026-03-07T13:56:36.175726+00:00 anon matrix-authentication-service: 2026-03-07T13:56:36.175672Z INFO http.server.response GET-188 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.2ms]
2026-03-07T13:56:44.411185+00:00 anon matrix-authentication-service: 2026-03-07T13:56:44.411098Z INFO http.server.response GET-199 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.2ms]
2026-03-07T13:56:52.859896+00:00 anon matrix-authentication-service: 2026-03-07T13:56:52.859810Z INFO http.server.response GET-210 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.3ms]
2026-03-07T13:57:01.095989+00:00 anon matrix-authentication-service: 2026-03-07T13:57:01.095902Z INFO http.server.response GET-221 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.2ms]
2026-03-07T13:57:09.577228+00:00 anon matrix-authentication-service: 2026-03-07T13:57:09.577145Z INFO http.server.response GET-233 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.2ms]
2026-03-07T13:57:17.847141+00:00 anon matrix-authentication-service: 2026-03-07T13:57:17.847054Z INFO http.server.response GET-245 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.2ms]
2026-03-07T13:57:21.677355+00:00 anon matrix-authentication-service: 2026-03-07T13:57:21.677253Z INFO mas_tasks::new_queue:535 worker-run-loop-248 - Sending heartbeat
2026-03-07T13:57:26.309644+00:00 anon matrix-authentication-service: 2026-03-07T13:57:26.309574Z INFO http.server.response GET-256 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.3ms]
2026-03-07T13:57:34.547331+00:00 anon matrix-authentication-service: 2026-03-07T13:57:34.547274Z INFO http.server.response GET-267 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.2ms]
2026-03-07T13:57:38.023808+00:00 anon matrix-authentication-service: 2026-03-07T13:57:38.023732Z INFO mas_tasks::new_queue:728 worker-run-loop-270 - One scheduled job marked as available
2026-03-07T13:57:38.025489+00:00 anon matrix-authentication-service: 2026-03-07T13:57:38.025434Z INFO mas_tasks::new_queue:910 job-cleanup-queue-jobs-272 - Running job job.id=01KK49BAYDWAYD25XGX6XJBJCH job.queue.name=cleanup-queue-jobs job.attempt=5 job.timeout=600s
2026-03-07T13:57:38.031709+00:00 anon postgres[8633]: [12-1] ERROR: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"
2026-03-07T13:57:38.031718+00:00 anon postgres[8633]: [12-2] DETAIL: Key (queue_job_id)=(019c1949-ea26-1724-5463-a546413e24cb) is still referenced from table "queue_schedules".
2026-03-07T13:57:38.031724+00:00 anon postgres[8633]: [12-3] STATEMENT:
2026-03-07T13:57:38.031729+00:00 anon postgres[8633]: [12-4] #011 WITH to_delete AS (
2026-03-07T13:57:38.031735+00:00 anon postgres[8633]: [12-5] #011 SELECT queue_job_id
2026-03-07T13:57:38.031740+00:00 anon postgres[8633]: [12-6] #011 FROM queue_jobs
2026-03-07T13:57:38.031745+00:00 anon postgres[8633]: [12-7] #011 WHERE (status = 'completed' OR status = 'failed')
2026-03-07T13:57:38.031750+00:00 anon postgres[8633]: [12-8] #011 AND ($1::uuid IS NULL OR queue_job_id > $1)
2026-03-07T13:57:38.031754+00:00 anon postgres[8633]: [12-9] #011 AND queue_job_id <= $2
2026-03-07T13:57:38.031759+00:00 anon postgres[8633]: [12-10] #011 ORDER BY queue_job_id
2026-03-07T13:57:38.031764+00:00 anon postgres[8633]: [12-11] #011 LIMIT $3
2026-03-07T13:57:38.031769+00:00 anon postgres[8633]: [12-12] #011 )
2026-03-07T13:57:38.031774+00:00 anon postgres[8633]: [12-13] #011 DELETE FROM queue_jobs
2026-03-07T13:57:38.031779+00:00 anon postgres[8633]: [12-14] #011 USING to_delete
2026-03-07T13:57:38.031784+00:00 anon postgres[8633]: [12-15] #011 WHERE queue_jobs.queue_job_id = to_delete.queue_job_id
2026-03-07T13:57:38.031789+00:00 anon postgres[8633]: [12-16] #011 RETURNING queue_jobs.queue_job_id
2026-03-07T13:57:38.031793+00:00 anon postgres[8633]: [12-17] #011
2026-03-07T13:57:38.031915+00:00 anon matrix-authentication-service: 2026-03-07T13:57:38.031855Z ERROR mas_storage_pg::queue::job:449 job-cleanup-queue-jobs-272 - error=error returned from database: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"
2026-03-07T13:57:38.031926+00:00 anon matrix-authentication-service: 2026-03-07T13:57:38.031899Z WARN mas_tasks::new_queue:960 job-cleanup-queue-jobs-272 - Job failed, will retry in 160s [polls: 3, cpu: 0.1ms, elapsed: 6.5ms] error=error returned from database: update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules" error.sources=[update or delete on table "queue_jobs" violates foreign key constraint "queue_schedules_last_scheduled_job_id_fkey" on table "queue_schedules"] job.id=01KK49BAYDWAYD25XGX6XJBJCH job.queue.name=cleanup-queue-jobs job.attempt=5
2026-03-07T13:57:43.039767+00:00 anon matrix-authentication-service: 2026-03-07T13:57:43.039399Z INFO http.server.response GET-281 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.3ms]
2026-03-07T13:57:51.342804+00:00 anon matrix-authentication-service: 2026-03-07T13:57:51.342749Z INFO http.server.response GET-293 - "GET /health HTTP/1.1" 200 OK "-" [polls: 2, cpu: 0.1ms, elapsed: 0.2ms]
Both auth service and homeserver seems to be working as expected. Not sure if this is fatal or part of db migration code not doing its job.