Sending e-mail async with celery (#4483)

This does a full implementation of celery for sending emails. The
functionality was already partially implemented but wasn't properly
configured to be used app-wide nor was it ever enabled.

**Running locally**
1. Get a Redis instance setup locally (I used docker, ie. `docker run -d
-p 6379:6379 redis`)
2. Install requirements
3. Configuration
    - `SEND_MESSAGES=True` so emails will appear in terminal
- `CELERY_TASK_ALWAYS_EAGER=False` so celery will add tasks to the
broker rather than run it immediately
- `CELERY_BROKER_URL` & `CELERY_RESULT_BACKEND` [are properly set to
your
broker](https://docs.celeryq.dev/en/stable/getting-started/first-steps-with-celery.html)
4. In a separate terminal from the running `make serve` (but with same
environment vars), run `celery -A hypha.celery worker -E
--loglevel=info` to start celery


Co-authored-by: Fredrik Jonsson <frjo@xdeb.org>

Author: wes-otf

Procfile

@@ -1,2 +1,3 @@
 release: python manage.py migrate --noinput && python manage.py clear_cache --cache=default && python manage.py sync_roles
 web: gunicorn hypha.wsgi:application --log-file -
+worker: celery worker --app=hypha.celery --autoscale=6,2 --events

docs/getting-started/architecture.md

@@ -1,25 +1,19 @@
-
-## Diagram
-
-```
-                            Integrations                           
-                            ┌────────────┐ ┌────────────┐          
-                            │   Email    │ │   Slack    │          
-                            └────────────┘ └────────────┘          
-                            ┌────────────┐ ┌────────────┐          
-                            │ Amazon S3  │ │   Sentry   │          
-                            └────────────┘ └────────────┘          
-                                                                    
-                                                                    
-
-    ┌────────────┐                                  Databases
-    │ APPLY SITE │◀────┐                            ┌ ─ ─ ─ ─ ─ ─ ─ ─┐
-    └────────────┘     │      ┌────────────┐        ╎ ┌────────────┐ ╎
-                       ├──────│  Django /  │◀───────╎ │ PostgreSQL │ ╎
-    ┌────────────┐     │      │  Wagtail   │        ╎ └────────────┘ ╎
-    │   WAGTAIL  │◀────┘      └────────────┘        └─ ─ ─ ─ ─ ─ ─ ─ ┘
-    │   ADMIN    │                                  
-    └────────────┘
+```mermaid
+---
+title: Hypha Architecture Diagram
+---
+flowchart LR
+    subgraph integrations[Integrations]
+    email[Email]
+    slack[Slack]
+    amazon[Amazon S3]
+    sentry[Sentry]
+    redis[Redis]
+    end
+    postgres[(PostgreSQL)] --> django[Django/Wagtail]
+    django --> apply[Apply Site]
+    django --> wagtail[Wagtail Admin]
+    django --> |"(Optional)"|celery[Celery]
 ```
 
 -----------
@@ -54,6 +48,10 @@ Media is encouraged to be split into two distinct storage locations. A Public an
 
 Media should also be served from a view that inherits from the [PrivateMediaView](https://github.com/HyphaApp/hypha/blob/main/hypha/apply/utils/storage.py) which will confirm that the file isn't made public and can be configured to return the file object from an authenticated view.
 
+### Celery
+
+An optional addition to allow certain tasks (ie. email/slack sending, other slower operations) to run asynchronously. Requires a separate celery worker to be running
+
 
 ## External Integrations
 
@@ -68,3 +66,7 @@ If configured, Hypha is able to send out notifications to different slack channe
 ### Email
 
 Emails in Hypha are used for password recovery and sending out important notifications to the users.
+
+### Redis
+
+Hypha uses Redis as a message broker for Celery

docs/setup/administrators/configuration.md

@@ -135,6 +135,36 @@ Good for testing, might not be a good idea in production.
 
 ----
 
+Celery settings for operations such as email/slack sending
+
+When enabled (default), all celery tasks will run synchronously and do not require a broker.
+
+    CELERY_TASK_ALWAYS_EAGER = env.bool("CELERY_TASK_ALWAYS_EAGER", True)
+
+
+Env var used by Heroku to set & update the Redis URL
+
+    REDIS_URL =  env.str("REDIS_URL", None)
+
+Used to set the TLS cert verification mode - needs to be `CERT_REQUIRED`, `CERT_OPTIONAL` or `CERT_NONE`
+    
+    REDIS_SSL_CERT_REQS = env.str("REDIS_SSL_CERT_REQS", "CERT_REQUIRED")
+
+Max connections allowed to Redis - defaulted to 20
+
+    CELERY_REDIS_MAX_CONNECTIONS = env.int("CELERY_REDIS_MAX_CONNECTIONS", 20)
+
+Used to specify the broker Celery should use, for more info see the [Celery getting started docs](https://docs.celeryq.dev/en/stable/getting-started/first-steps-with-celery.html)
+
+    CELERY_BROKER_URL = env.str("CELERY_BROKER_URL", None)
+    CELERY_RESULT_BACKEND = env.str("CELERY_RESULT_BACKEND", None)
+
+!!! note
+
+    If a `REDIS_URL` value is specified it will automatically be set as both `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND`. This can be overridden by manually setting either `CELERY_BROKER_URL` or `CELERY_RESULT_BACKEND`. `REDIS_URL` & `REDIS_SSL_CERT_REQS` is primarily intended for use with Heroku's Key-Value Store, for more complex TLS needs it is recommend to manually set the result backend and broker url and configure TLS via URL params.
+
+----
+
 Organisation name and e-mail address etc., used in e-mail templates etc.
 
     ORG_EMAIL = env.str('ORG_EMAIL', 'info@example.org')

hypha/apply/activity/adapters/emails.py

@@ -8,6 +8,7 @@
 from django.template.loader import render_to_string
 from django.utils.translation import gettext as _
 
+from hypha.apply.activity import tasks
 from hypha.apply.activity.models import ALL, APPLICANT_PARTNERS, PARTNER
 from hypha.apply.projects.models.payment import CHANGES_REQUESTED_BY_STAFF, DECLINED
 from hypha.apply.projects.templatetags.project_tags import display_project_status
@@ -23,7 +24,6 @@
 )
 
 from ..options import MESSAGES
-from ..tasks import send_mail
 from .base import AdapterBase
 from .utils import (
     get_compliance_email,
@@ -479,6 +479,6 @@ def send_message(self, message, source, subject, recipient, logs, **kwargs):
             logger.exception(e)
 
         try:
-            send_mail(subject, message, from_email, [recipient], logs=logs)
+            tasks.send_mail(subject, message, from_email, [recipient], logs=logs)
         except Exception as e:
             return "Error: " + str(e)

hypha/apply/activity/tasks.py

@@ -1,11 +1,7 @@
-from celery import Celery
+from celery import shared_task
 from django.conf import settings
 from django.core.mail import EmailMessage
 
-app = Celery("tasks")
-
-app.config_from_object(settings, namespace="CELERY", force=True)
-
 
 def send_mail(subject, message, from_address, recipients, logs=None):
     if settings.EMAIL_SUBJECT_PREFIX:
@@ -22,7 +18,7 @@ def send_mail(subject, message, from_address, recipients, logs=None):
     )
 
 
-@app.task
+@shared_task
 def send_mail_task(**kwargs):
     response = {"status": "", "id": None}
     email = EmailMessage(**kwargs)
@@ -42,7 +38,7 @@ def send_mail_task(**kwargs):
     return response
 
 
-@app.task
+@shared_task
 def update_message_status(response, message_pks):
     from .models import Message