Skip to content

Commit ae908da

Browse files
pwnage101pwnage101
committed
fix: For the openedx-events broker, use kafka _everywhere_ instead of redis
Devstack is currently split between services pre-configured to use redis, and others pre-configured to use kafka. This results in the exceedingly dumb result of events produced via one broker, but never consumed because the consuming apps are hooked up to the other broker.
1 parent fa6c178 commit ae908da

File tree

9 files changed

+48
-23
lines changed

9 files changed

+48
-23
lines changed

‎docker-compose-host.yml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ services:
4141
- ${DEVSTACK_WORKSPACE}/src:/edx/src
4242
- ${PWD}/py_configuration_files/cms.py:/edx/app/edxapp/edx-platform/cms/envs/devstack.py
4343
- ${PWD}/py_configuration_files/lms.py:/edx/app/edxapp/edx-platform/lms/envs/devstack.py
44+
- ${PWD}/py_private_requirements/lms.txt:/edx/private_requirements.txt
4445
lms-worker:
4546
volumes:
4647
- ${DEVSTACK_WORKSPACE}/edx-platform:/edx/app/edxapp/edx-platform
@@ -71,6 +72,7 @@ services:
7172
- ${DEVSTACK_WORKSPACE}/src:/edx/src
7273
- ${PWD}/py_configuration_files/cms.py:/edx/app/edxapp/edx-platform/cms/envs/devstack.py
7374
- ${PWD}/py_configuration_files/lms.py:/edx/app/edxapp/edx-platform/lms/envs/devstack.py
75+
- ${PWD}/py_private_requirements/cms.txt:/edx/private_requirements.txt
7476
cms-worker:
7577
volumes:
7678
- ${DEVSTACK_WORKSPACE}/edx-platform:/edx/app/edxapp/edx-platform
@@ -95,7 +97,11 @@ services:
9597
- ${DEVSTACK_WORKSPACE}/enterprise-subsidy:/edx/app/enterprise-subsidy
9698
- ${DEVSTACK_WORKSPACE}/src:/edx/src
9799
- ${PWD}/py_configuration_files/enterprise_subsidy.py:/edx/app/enterprise-subsidy/enterprise_subsidy/settings/devstack.py
98-
100+
enterprise-subsidy-consume_learner_credit_course_enrollment_lifecycle:
101+
volumes:
102+
- ${DEVSTACK_WORKSPACE}/enterprise-subsidy:/edx/app/enterprise-subsidy
103+
- ${DEVSTACK_WORKSPACE}/src:/edx/src
104+
- ${PWD}/py_configuration_files/enterprise_subsidy.py:/edx/app/enterprise-subsidy/enterprise_subsidy/settings/devstack.py
99105
enterprise-catalog:
100106
volumes:
101107
- ${DEVSTACK_WORKSPACE}/enterprise-catalog:/edx/app/enterprise_catalog/enterprise_catalog

‎docker-compose.yml‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -449,7 +449,7 @@ services:
449449

450450
lms:
451451
# Switch to `--settings devstack_with_worker` if you want to use lms-worker
452-
command: bash -c 'source /edx/app/edxapp/edxapp_env && while true; do python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack; sleep 2; done'
452+
command: bash -c 'source /edx/app/edxapp/edxapp_env && (pip install -r /edx/private_requirements.txt; while true; do python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack; sleep 2; done)'
453453
container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.lms"
454454
hostname: lms.devstack.edx
455455
depends_on:
@@ -638,7 +638,7 @@ services:
638638

639639
cms:
640640
# Switch to `--settings devstack_with_worker` if you want to use cms-worker
641-
command: bash -c 'source /edx/app/edxapp/edxapp_env && while true; do python /edx/app/edxapp/edx-platform/manage.py cms runserver 0.0.0.0:18010 --settings devstack; sleep 2; done'
641+
command: bash -c 'source /edx/app/edxapp/edxapp_env && (pip install -r /edx/private_requirements.txt; while true; do python /edx/app/edxapp/edx-platform/manage.py lms runserver 0.0.0.0:18000 --settings devstack; sleep 2; done)'
642642
container_name: "edx.${COMPOSE_PROJECT_NAME:-devstack}.cms"
643643
hostname: cms.devstack.edx
644644
depends_on:
@@ -905,6 +905,22 @@ services:
905905
volumes:
906906
- /edx/var/enterprise-subsidy
907907

908+
enterprise-subsidy-consume_learner_credit_course_enrollment_lifecycle:
909+
image: edxops/enterprise-subsidy-dev
910+
container_name: edx.devstack.enterprise-subsidy-consume_learner_credit_course_enrollment_lifecycle
911+
depends_on:
912+
- mysql80
913+
- memcached
914+
command: >-
915+
bash -c 'while true; do python /edx/app/enterprise-subsidy/manage.py consume_events -t learner-credit-course-enrollment-lifecycle -g enterprise_subsidy_dev; sleep 2; done'
916+
tty: true
917+
environment:
918+
DJANGO_SETTINGS_MODULE: enterprise_subsidy.settings.devstack
919+
working_dir: /edx/app/enterprise-subsidy
920+
ports:
921+
- "18281:18281"
922+
stdin_open: true
923+
908924
# ==========================================================================
909925
# edX Microfrontends
910926
#

‎py_configuration_files/cms.py‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@
9090

9191
################################ DEBUG TOOLBAR ################################
9292

93-
INSTALLED_APPS += ['debug_toolbar']
93+
INSTALLED_APPS += ('debug_toolbar',)
9494

9595
MIDDLEWARE.append('debug_toolbar.middleware.DebugToolbarMiddleware')
9696
INTERNAL_IPS = ('127.0.0.1',)
@@ -296,11 +296,12 @@ def should_show_debug_toolbar(request): # lint-amnesty, pylint: disable=missing
296296
]
297297

298298
#################### Event bus backend ########################
299-
300-
EVENT_BUS_PRODUCER = 'edx_event_bus_redis.create_producer'
301-
EVENT_BUS_REDIS_CONNECTION_URL = 'redis://:[email protected]:6379/'
299+
INSTALLED_APPS += ('edx_event_bus_kafka',)
300+
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL = 'http://edx.devstack.schema-registry:8081'
301+
EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS = 'edx.devstack.kafka:29092'
302+
EVENT_BUS_PRODUCER = 'edx_event_bus_kafka.create_producer'
303+
EVENT_BUS_CONSUMER = 'edx_event_bus_kafka.KafkaEventConsumer'
302304
EVENT_BUS_TOPIC_PREFIX = 'dev'
303-
EVENT_BUS_CONSUMER = 'edx_event_bus_redis.RedisEventConsumer'
304305

305306
course_catalog_event_setting = EVENT_BUS_PRODUCER_CONFIG['org.openedx.content_authoring.course.catalog_info.changed.v1']
306307
course_catalog_event_setting['course-catalog-info-changed']['enabled'] = True

‎py_configuration_files/course_discovery.py‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -86,9 +86,12 @@
8686

8787
CELERY_TASK_ALWAYS_EAGER = False
8888

89-
EVENT_BUS_CONSUMER = 'edx_event_bus_redis.RedisEventConsumer'
90-
EVENT_BUS_PRODUCER = 'edx_event_bus_redis.create_producer'
91-
EVENT_BUS_REDIS_CONNECTION_URL = 'redis://:[email protected]:6379/'
89+
#################### Event bus backend ########################
90+
INSTALLED_APPS += ('edx_event_bus_kafka',)
91+
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL = 'http://edx.devstack.schema-registry:8081'
92+
EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS = 'edx.devstack.kafka:29092'
93+
EVENT_BUS_PRODUCER = 'edx_event_bus_kafka.create_producer'
94+
EVENT_BUS_CONSUMER = 'edx_event_bus_kafka.KafkaEventConsumer'
9295
EVENT_BUS_TOPIC_PREFIX = 'dev'
9396

9497
#####################################################################

‎py_configuration_files/enterprise_access.py‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -61,10 +61,7 @@
6161

6262
# Install django-extensions for improved dev experiences
6363
# https://github.com/django-extensions/django-extensions#using-it
64-
INSTALLED_APPS += (
65-
'django_extensions',
66-
'edx_event_bus_kafka',
67-
)
64+
INSTALLED_APPS += ('django_extensions',)
6865

6966
# BEGIN CELERY
7067
CELERY_WORKER_HIJACK_ROOT_LOGGER = True
@@ -114,6 +111,7 @@
114111
################### Kafka Related Settings ##############################
115112

116113
# "Standard" Kafka settings as defined in https://github.com/openedx/event-bus-kafka/tree/main
114+
INSTALLED_APPS += ('edx_event_bus_kafka',)
117115
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL = 'http://edx.devstack.schema-registry:8081'
118116
EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS = 'edx.devstack.kafka:29092'
119117
EVENT_BUS_PRODUCER = 'edx_event_bus_kafka.create_producer'

‎py_configuration_files/enterprise_subsidy.py‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -33,10 +33,6 @@
3333

3434
}
3535

36-
INSTALLED_APPS += (
37-
'edx_event_bus_kafka',
38-
)
39-
4036
# Generic OAuth2 variables irrespective of SSO/backend service key types.
4137
OAUTH2_PROVIDER_URL = 'http://edx.devstack.lms:18000/oauth2'
4238

@@ -83,6 +79,7 @@
8379

8480
# Kafka Settings
8581
# "Standard" Kafka settings as defined in https://github.com/openedx/event-bus-kafka/tree/main
82+
INSTALLED_APPS += ('edx_event_bus_kafka',)
8683
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL = 'http://edx.devstack.schema-registry:8081'
8784
EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS = 'edx.devstack.kafka:29092'
8885
EVENT_BUS_PRODUCER = 'edx_event_bus_kafka.create_producer'

‎py_configuration_files/lms.py‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@
8080

8181
################################ DEBUG TOOLBAR ################################
8282

83-
INSTALLED_APPS += ['debug_toolbar']
83+
INSTALLED_APPS += ('debug_toolbar',)
8484
MIDDLEWARE += [
8585
'lms.djangoapps.discussion.django_comment_client.utils.QueryCountDebugMiddleware',
8686
'debug_toolbar.middleware.DebugToolbarMiddleware',
@@ -504,10 +504,12 @@ def should_show_debug_toolbar(request): # lint-amnesty, pylint: disable=missing
504504
CLOSEST_CLIENT_IP_FROM_HEADERS = []
505505

506506
#################### Event bus backend ########################
507-
EVENT_BUS_PRODUCER = 'edx_event_bus_redis.create_producer'
508-
EVENT_BUS_REDIS_CONNECTION_URL = 'redis://:[email protected]:6379/'
507+
INSTALLED_APPS += ('edx_event_bus_kafka',)
508+
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL = 'http://edx.devstack.schema-registry:8081'
509+
EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS = 'edx.devstack.kafka:29092'
510+
EVENT_BUS_PRODUCER = 'edx_event_bus_kafka.create_producer'
511+
EVENT_BUS_CONSUMER = 'edx_event_bus_kafka.KafkaEventConsumer'
509512
EVENT_BUS_TOPIC_PREFIX = 'dev'
510-
EVENT_BUS_CONSUMER = 'edx_event_bus_redis.RedisEventConsumer'
511513

512514
certificate_revoked_event_config = EVENT_BUS_PRODUCER_CONFIG['org.openedx.learning.certificate.revoked.v1']
513515
certificate_revoked_event_config['learning-certificate-lifecycle']['enabled'] = True

‎py_private_requirements/cms.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
confluent-kafka[avro,schema-registry]==2.10.0

‎py_private_requirements/lms.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
confluent-kafka[avro,schema-registry]==2.10.0

0 commit comments

Comments
 (0)