Celery workers do not consume tasks with HTTPS on MacOS

[poundifdef]

When running SmoothMQ with HTTPS, we can add tasks with Celery but workers do not pick them up. This is only on MacOS - everything works as expected on Linux.

Here's a code example:

The Problem

from celery import Celery

app = Celery(
    "tasks",
    broker_url='sqs://7399377469860198052:27018ec9cb39d6d4d09b0f0378d3991c@7399377469860198052.demo.smoothmq.com:3001',
    broker_transport_options={'is_secure': True}
)

@app.task
def hello():
    return 'hello world'

if __name__ == '__main__':
        hello.delay()

This will successfully place the task on a queue, which you can see on this dashbaord. However, the worker will not pick up tasks:

celery -A tasks worker -l DEBUG --concurrency 1

It will successfully create queues, list queues, and prepare to call ReceiveMessage, but then loops infinitely:

[2024-08-04 16:07:29,165: DEBUG/MainProcess] Event choose-signer.sqs.ReceiveMessage: calling handler <function set_operation_specific_signer at 0x105778360>
[2024-08-04 16:07:29,166: DEBUG/MainProcess] Calculating signature using v4 auth.
[2024-08-04 16:07:29,166: DEBUG/MainProcess] CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.0
host:7399377469860198052.demo.smoothmq.com:3001
x-amz-date:20240804T200729Z
x-amz-target:AmazonSQS.ReceiveMessage

content-type;host;x-amz-date;x-amz-target
1a762bf8b84958772e686dd62bcc0d4c11957890e258359aaf2d6c9f894d2c12
[2024-08-04 16:07:29,166: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20240804T200729Z
20240804/us-east-1/sqs/aws4_request
3d33e059354446a823a67b47de5ad55fdb7ac7326d99013d3c147ceb4c84a6cf
[2024-08-04 16:07:29,166: DEBUG/MainProcess] Signature:
10cb9d37b40482dc470e6b17b556f7373a43399d6b3e39d65b0494d0192b5904

Interestingly, the following scenarios do work:

Running SmoothMQ locally

This will successfully connect and consume messages - non-HTTPS:

$ go run . server

app = Celery(
    "tasks",
    broker_url='sqs://DEV_ACCESS_KEY_ID:DEV_SECRET_ACCESS_KEY@localhost:3001',
    broker_transport_options={'is_secure': False}
)

Kombu without Celery

This will also successfully consume messages, even over HTTPS:

from kombu import Connection

with Connection(f'sqs://7399377469860198052:27018ec9cb39d6d4d09b0f0378d3991c@7399377469860198052.demo.smoothmq.com:3001', transport_options={'is_secure':True}) as conn:
    simple_queue = conn.SimpleQueue('q1')

    simple_queue.put('simple queue message')

    message = simple_queue.get(block=True, timeout=1)
    print(f'Received: {message.payload}')
    message.ack()
    simple_queue.close()