Skip to content

Commit 9371d24

Browse files
ewdurbinewdurbin
authored
Cleanup Celery/Redis settings and configuration (#17640)
* remove drain-sqs process * remove startup wrappers the only part of bin/redis-tls we used to need was the certifi bit, we can now just apply these parameters to url directly and skip specifiying a CA certfile since elasticache's CA is in the bookworm default CAs. **NEEDS UPDATE TO REDIS_URL BEFORE DEPLOY** * remove amqp/sqs broker support for celery
1 parent 801189c commit 9371d24

File tree

11 files changed

+39
-304
lines changed

11 files changed

+39
-304
lines changed

‎Procfile

Lines changed: 6 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,7 @@
11
release: bin/release
2-
web: bin/start-web ddtrace-run python -m gunicorn.app.wsgiapp -c gunicorn-prod.conf.py warehouse.wsgi:application
3-
web-api: bin/start-web ddtrace-run python -m gunicorn.app.wsgiapp -c gunicorn-prod.conf.py warehouse.wsgi:application
4-
web-uploads: bin/start-web ddtrace-run python -m gunicorn.app.wsgiapp -c gunicorn-uploads.conf.py warehouse.wsgi:application
5-
worker: bin/start-worker celery -A warehouse worker --concurrency=${CELERY_CONCURRENCY:-1} -Q default -l info --max-tasks-per-child 1024
6-
worker-beat: bin/start-worker celery -A warehouse beat -S redbeat.RedBeatScheduler -l info
7-
worker-traced: env DD_SERVICE=warehouse-worker bin/start-worker ddtrace-run celery -A warehouse worker --concurrency=${CELERY_CONCURRENCY:-1} -Q default -l info --max-tasks-per-child 32
8-
worker-drain-sqs: env BROKER_URL=sqs:///?region=us-east-2&queue_name_prefix=pypi-worker bin/start-worker --concurrency=${CELERY_CONCURRENCY:-1} celery -A warehouse worker -Q default -l info --max-tasks-per-child 1024
2+
web: ddtrace-run python -m gunicorn.app.wsgiapp -c gunicorn-prod.conf.py warehouse.wsgi:application
3+
web-api: ddtrace-run python -m gunicorn.app.wsgiapp -c gunicorn-prod.conf.py warehouse.wsgi:application
4+
web-uploads: ddtrace-run python -m gunicorn.app.wsgiapp -c gunicorn-uploads.conf.py warehouse.wsgi:application
5+
worker: celery -A warehouse worker --concurrency=${CELERY_CONCURRENCY:-1} -Q default -l info --max-tasks-per-child 1024
6+
worker-beat: celery -A warehouse beat -S redbeat.RedBeatScheduler -l info
7+
worker-traced: env DD_SERVICE=warehouse-worker ddtrace-run celery -A warehouse worker --concurrency=${CELERY_CONCURRENCY:-1} -Q default -l info --max-tasks-per-child 32

‎bin/redis-tls

Lines changed: 0 additions & 25 deletions
This file was deleted.

‎bin/start-web

Lines changed: 0 additions & 10 deletions
This file was deleted.

‎bin/start-worker

Lines changed: 0 additions & 10 deletions
This file was deleted.

‎docker-compose.yml

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -34,16 +34,6 @@ services:
3434
redis:
3535
image: redis:7.0
3636

37-
localstack:
38-
image: localstack/localstack:3.5
39-
stop_signal: SIGKILL
40-
environment:
41-
SERVICES: "sqs"
42-
LOCALSTACK_HOST: "localstack"
43-
LS_LOG: "error"
44-
ports:
45-
- "4566:4566"
46-
4737
opensearch:
4838
build:
4939
context: ./dev/compose/opensearch

‎requirements/main.in

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ b2sdk
66
Babel
77
bcrypt
88
boto3
9-
celery[sqs]>=5.3.5,<5.5
9+
celery[redis]>=5.3.5,<5.5
1010
celery-redbeat
1111
certifi
1212
click
@@ -24,7 +24,7 @@ html5lib
2424
humanize
2525
itsdangerous
2626
Jinja2>=2.8
27-
kombu[sqs]>=5.4,<5.5 # https://github.com/jazzband/pip-tools/issues/1577
27+
kombu[redis]>=5.4,<5.5 # https://github.com/jazzband/pip-tools/issues/1577
2828
limits
2929
linehaul
3030
lxml

‎requirements/main.txt

Lines changed: 5 additions & 43 deletions
Original file line numberDiff line numberDiff line change
@@ -112,10 +112,7 @@ billiard==4.2.1 \
112112
boto3==1.36.25 \
113113
--hash=sha256:41fb90a516995946563ec91b9d891e2516c58617e9556d5e86dfa62da3fdebe6 \
114114
--hash=sha256:a057c19adffb48737c192bdb10f9d85e0d9dcecd21327f51520c15db9022a835
115-
# via
116-
# -r requirements/main.in
117-
# celery
118-
# kombu
115+
# via -r requirements/main.in
119116
botocore==1.36.25 \
120117
--hash=sha256:04c8ff03531e8d92baa8c98d1850bdf01668a805467f4222b65e5325f94aa8af \
121118
--hash=sha256:3b0a857d2621c336fb82a36cb6da4b6e062d346451ac46d110b074e5e5fd7cfc
@@ -178,7 +175,7 @@ cbor2==5.6.5 \
178175
--hash=sha256:fde21ac1cf29336a31615a2c469a9cb03cf0add3ae480672d4d38cda467d07fc \
179176
--hash=sha256:fe11c2eb518c882cfbeed456e7a552e544893c17db66fe5d3230dbeaca6b615c
180177
# via webauthn
181-
celery[sqs]==5.4.0 \
178+
celery[redis]==5.4.0 \
182179
--hash=sha256:369631eb580cf8c51a82721ec538684994f8277637edde2dfc0dacd73ed97f64 \
183180
--hash=sha256:504a19140e8d3029d5acad88330c541d4c3f64c789d85f94756762d8bca7e706
184181
# via
@@ -941,7 +938,7 @@ jsonschema-specifications==2024.10.1 \
941938
# via
942939
# jsonschema
943940
# openapi-schema-validator
944-
kombu[sqs]==5.4.2 \
941+
kombu[redis]==5.4.2 \
945942
--hash=sha256:14212f5ccf022fc0a70453bb025a1dcc32782a588c49ea866884047d66e14763 \
946943
--hash=sha256:eef572dd2fd9fc614b37580e3caeafdd5af46c1eff31e7fba89138cdb406f2cf
947944
# via
@@ -1632,41 +1629,6 @@ pycparser==2.22 \
16321629
--hash=sha256:491c8be9c040f5390f5bf44a5b07752bd07f56edf992381b05c701439eec10f6 \
16331630
--hash=sha256:c3702b6d3dd8c7abc1afa565d7e63d53a1d0bd86cdc24edd75470f4de499cfcc
16341631
# via cffi
1635-
pycurl==7.45.4 \
1636-
--hash=sha256:0470bff6cc24d8c2f63c80931aa239463800871609dafc6bcc9ca10f5a12a04e \
1637-
--hash=sha256:1324a859b50bdb0abdbd5620e42f74240d0b7daf2d5925fa303695d9fc3ece18 \
1638-
--hash=sha256:13c4b18f44637859f34639493efd297a08670f45e4eec34ab2dcba724e3cb5fc \
1639-
--hash=sha256:13eb1643ab0bf4fdc539a2cdf1021029b07095d3196c5cee5a4271af268d3d31 \
1640-
--hash=sha256:236600bfe2cd72efe47333add621286667e8fa027dadf1247349afbf30333e95 \
1641-
--hash=sha256:247b4af8eab7d04137a7f1a98391930e04ea93dc669b64db5625070fe15f80a3 \
1642-
--hash=sha256:2548c3291a33c821f0f80bf9989fc43b5d90fb78b534a7015c8419b83c6f5803 \
1643-
--hash=sha256:26745c6c5ebdccfe8a828ac3fd4e6da6f5d2245696604f04529eb7894a02f4db \
1644-
--hash=sha256:32c8e237069273f4260b6ae13d1e0f99daae938977016021565dc6e11050e803 \
1645-
--hash=sha256:3452459668bd01d646385482362b021834a31c036aa1c02acd88924ddeff7d0d \
1646-
--hash=sha256:4f25d52c97dbca6ebea786f0961b49c1998fa05178abf1964a977c825b3d8ae6 \
1647-
--hash=sha256:561f88697f7540634b1c750146f37bdc0da367b15f6b4ab2bb780871ee6ab005 \
1648-
--hash=sha256:57971d7215fc6fdedcfc092f880a59f04f52fcaf2fd329151b931623d7b59a9c \
1649-
--hash=sha256:587a4891039803b5f48392066f97b7cd5e7e9a166187abb5cb4b4806fdb8fbef \
1650-
--hash=sha256:688d09ba2c6a0d4a749d192c43422839d73c40c85143c50cc65c944258fe0ba8 \
1651-
--hash=sha256:731c46e7c0acffaab19f7c2ecc3d9e7ee337500e87b260b4e0b9fae2d90fa133 \
1652-
--hash=sha256:73df3eb5940a7fbf4cf62f7271e9f23a8e9f80e352c838ee9a8448a70c01d3f5 \
1653-
--hash=sha256:9940e3234c1ca3d30f27a2202d325dbc25291605c98e9585100a351cacd935e8 \
1654-
--hash=sha256:9bd493ce598f1dc76c8e50043c47debec27c583fa313a836b2d3667640f875d5 \
1655-
--hash=sha256:a39f28f031885485325034918386be352036c220ca45625c7e286d3938eb579d \
1656-
--hash=sha256:acf25cfdaf914db21a2a6e9e274b6d95e3fa2b6018c38f2c58c94b5d8ac3d1b7 \
1657-
--hash=sha256:b0e38e3eb83b0c891f391853f798fc6a97cb5a86a4a731df0b6320e539ae54ae \
1658-
--hash=sha256:b485fdaf78553f0b8e1c2803bb7dcbe47a7b47594f846fc7e9d3b94d794cfc89 \
1659-
--hash=sha256:caec8b634763351dd4e1b729a71542b1e2de885d39710ba8e7202817a381b453 \
1660-
--hash=sha256:d14f954ecd21a070038d65ef1c6d1d3ab220f952ff703d48313123222097615c \
1661-
--hash=sha256:d192a48b3cec2e13ad432196b65c22e99620db92feae39c0476635354eff68c6 \
1662-
--hash=sha256:df5f94c051c5a163fa85064559ca94979575e2da26740ff91c078c50c541c465 \
1663-
--hash=sha256:e7ae49b88a5d57485fbabef004534225dfe04dc15716a61fae1a0c7f46f2279e \
1664-
--hash=sha256:f6c0e22052946bbfa25be67f9d1d6639eff10781c89f0cf6f3ff2099273d1bad \
1665-
--hash=sha256:fd167f73d34beb0cb8064334aee76d9bdd13167b30be6d5d36fb07d0c8223b71 \
1666-
--hash=sha256:ffd3262f98b8997ad04940061d5ebd8bab2362169b9440939c397e24a4a135b0
1667-
# via
1668-
# celery
1669-
# kombu
16701632
pydantic[email]==2.10.6 \
16711633
--hash=sha256:427d664bf0b8a2b34ff5dd0f5a18df00591adcee7198fbd71981054cef37b584 \
16721634
--hash=sha256:ca5daa827cce33de7a42be142548b0096bf05a7e7b365aebfa5f8eeec7128236
@@ -1951,7 +1913,9 @@ redis==5.2.1 \
19511913
--hash=sha256:ee7e1056b9aea0f04c6c2ed59452947f34c4940ee025f5dd83e6a6418b6989e4
19521914
# via
19531915
# -r requirements/main.in
1916+
# celery
19541917
# celery-redbeat
1918+
# kombu
19551919
referencing==0.36.2 \
19561920
--hash=sha256:df2e89862cd09deabbdba16944cc3f10feb6b3e6f18e902f7cc25609a34775aa \
19571921
--hash=sha256:e8699adbbf8b5c7de96d8ffa0eb5c158b3beafce084968e2ea8bb08c6794dcd0
@@ -2309,8 +2273,6 @@ urllib3==2.3.0 \
23092273
# via
23102274
# -r requirements/main.in
23112275
# botocore
2312-
# celery
2313-
# kombu
23142276
# opensearch-py
23152277
# requests
23162278
# sentry-sdk

‎tests/conftest.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -306,7 +306,7 @@ def get_app_config(database, nondefaults=None):
306306
"warehouse.ip_salt": "insecure salt",
307307
"camo.url": "http://localhost:9000/",
308308
"camo.key": "insecure key",
309-
"celery.broker_url": "amqp://",
309+
"celery.broker_redis_url": "redis://localhost:0/",
310310
"celery.result_url": "redis://localhost:0/",
311311
"celery.scheduler_url": "redis://localhost:0/",
312312
"database.url": database,

‎tests/unit/test_tasks.py

Lines changed: 1 addition & 132 deletions
Original file line numberDiff line numberDiff line change
@@ -117,9 +117,6 @@ def test_without_request(self, monkeypatch):
117117
assert apply_async.calls == [
118118
pretend.call(
119119
task,
120-
message_attributes={
121-
"task_name": {"StringValue": None, "DataType": "String"}
122-
},
123120
)
124121
]
125122
assert get_current_request.calls == [pretend.call()]
@@ -142,9 +139,6 @@ def test_request_without_tm(self, monkeypatch):
142139
assert apply_async.calls == [
143140
pretend.call(
144141
task,
145-
message_attributes={
146-
"task_name": {"StringValue": None, "DataType": "String"}
147-
},
148142
)
149143
]
150144
assert get_current_request.calls == [pretend.call()]
@@ -165,9 +159,6 @@ def test_request_after_commit(self, monkeypatch):
165159
args = (pretend.stub(), pretend.stub())
166160
kwargs = {
167161
"foo": pretend.stub(),
168-
"message_attributes": {
169-
"task_name": {"StringValue": None, "DataType": "String"}
170-
},
171162
}
172163

173164
assert task.apply_async(*args, **kwargs) is None
@@ -463,140 +454,21 @@ def test_make_celery_app():
463454
(
464455
"env",
465456
"ssl",
466-
"broker_url",
467457
"broker_redis_url",
468458
"expected_url",
469459
"transport_options",
470460
),
471461
[
472-
(
473-
Environment.development,
474-
False,
475-
"amqp://guest@rabbitmq:5672//",
476-
None,
477-
"amqp://guest@rabbitmq:5672//",
478-
{},
479-
),
480-
(
481-
Environment.production,
482-
True,
483-
"amqp://guest@rabbitmq:5672//",
484-
None,
485-
"amqp://guest@rabbitmq:5672//",
486-
{},
487-
),
488-
(
489-
Environment.development,
490-
False,
491-
"sqs://",
492-
None,
493-
"sqs://",
494-
{
495-
"client-config": {"tcp_keepalive": True},
496-
},
497-
),
498-
(
499-
Environment.production,
500-
True,
501-
"sqs://",
502-
None,
503-
"sqs://",
504-
{
505-
"client-config": {"tcp_keepalive": True},
506-
},
507-
),
508-
(
509-
Environment.development,
510-
False,
511-
"sqs://?queue_name_prefix=warehouse",
512-
None,
513-
"sqs://",
514-
{
515-
"queue_name_prefix": "warehouse-",
516-
"client-config": {"tcp_keepalive": True},
517-
},
518-
),
519-
(
520-
Environment.production,
521-
True,
522-
"sqs://?queue_name_prefix=warehouse",
523-
None,
524-
"sqs://",
525-
{
526-
"queue_name_prefix": "warehouse-",
527-
"client-config": {"tcp_keepalive": True},
528-
},
529-
),
530-
(
531-
Environment.development,
532-
False,
533-
"sqs://?region=us-east-2",
534-
None,
535-
"sqs://",
536-
{
537-
"region": "us-east-2",
538-
"client-config": {"tcp_keepalive": True},
539-
},
540-
),
541-
(
542-
Environment.production,
543-
True,
544-
"sqs://?region=us-east-2",
545-
None,
546-
"sqs://",
547-
{
548-
"region": "us-east-2",
549-
"client-config": {"tcp_keepalive": True},
550-
},
551-
),
552-
(
553-
Environment.development,
554-
False,
555-
"sqs:///?region=us-east-2&queue_name_prefix=warehouse",
556-
None,
557-
"sqs://",
558-
{
559-
"region": "us-east-2",
560-
"queue_name_prefix": "warehouse-",
561-
"client-config": {"tcp_keepalive": True},
562-
},
563-
),
564-
(
565-
Environment.production,
566-
True,
567-
"sqs:///?region=us-east-2&queue_name_prefix=warehouse",
568-
None,
569-
"sqs://",
570-
{
571-
"region": "us-east-2",
572-
"queue_name_prefix": "warehouse-",
573-
"client-config": {"tcp_keepalive": True},
574-
},
575-
),
576-
(
577-
Environment.production,
578-
True,
579-
"sqs:///?region=us-east-2&queue_name_prefix=warehouse",
580-
"redis://127.0.0.1:6379/10",
581-
"sqs://",
582-
{
583-
"region": "us-east-2",
584-
"queue_name_prefix": "warehouse-",
585-
"client-config": {"tcp_keepalive": True},
586-
},
587-
),
588462
(
589463
Environment.production,
590464
True,
591-
None,
592465
"redis://127.0.0.1:6379/10",
593466
"redis://127.0.0.1:6379/10",
594467
{},
595468
),
596469
(
597470
Environment.production,
598471
True,
599-
None,
600472
(
601473
"rediss://user:[email protected]:6379/10"
602474
"?socket_timeout=5&irreleveant=0"
@@ -612,9 +484,7 @@ def test_make_celery_app():
612484
),
613485
],
614486
)
615-
def test_includeme(
616-
env, ssl, broker_url, broker_redis_url, expected_url, transport_options
617-
):
487+
def test_includeme(env, ssl, broker_redis_url, expected_url, transport_options):
618488
registry_dict = {}
619489
config = pretend.stub(
620490
action=pretend.call_recorder(lambda *a, **kw: None),
@@ -625,7 +495,6 @@ def test_includeme(
625495
__setitem__=registry_dict.__setitem__,
626496
settings={
627497
"warehouse.env": env,
628-
"celery.broker_url": broker_url,
629498
"celery.broker_redis_url": broker_redis_url,
630499
"celery.result_url": pretend.stub(),
631500
"celery.scheduler_url": pretend.stub(),

‎warehouse/config.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -374,7 +374,6 @@ def configure(settings=None):
374374
default="https://api.github.com/meta/public_keys/token_scanning",
375375
)
376376
maybe_set(settings, "warehouse.downloads_table", "WAREHOUSE_DOWNLOADS_TABLE")
377-
maybe_set(settings, "celery.broker_url", "BROKER_URL")
378377
maybe_set_redis(settings, "celery.broker_redis_url", "REDIS_URL", db=10)
379378
maybe_set_redis(settings, "celery.result_url", "REDIS_URL", db=12)
380379
maybe_set_redis(settings, "celery.scheduler_url", "REDIS_URL", db=0)

0 commit comments

Comments
 (0)