SendinBlue: additional template/tags improvements

Additional changes related to SendinBlue improvements in #158:

* Support multiple tags in webhooks (closes #162)
* Remove additional outdated template code in backend
* Update integration tests
* Update docs and changelog; note breaking changes as discussed in #161

Author: medmunds

CHANGELOG.rst

@@ -30,13 +30,33 @@ vNext
 
 *Not yet released*
 
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+* **SendinBlue templates:** Support Sendinblue's new (ESP stored) Django templates and
+  new API for template sending. This removes most of the odd limitations in the older
+  (now-deprecated) SendinBlue template send API, but involves two breaking changes:
+
+  * You *must* `convert <https://help.sendinblue.com/hc/en-us/articles/360000991960>`_
+    each old Sendinblue template to the new language as you upgrade to Anymail vNext, or
+    certain features may be silently ignored on template sends (notably `reply_to` and
+    recipient display names).
+
+  * Sendinblue's API no longer supports sending attachments when using templates.
+
+  Ordinary, non-template sending is not affected by these changes. See
+  `docs <https://anymail.readthedocs.io/en/latest/esps/sendinblue/#batch-sending-merge-and-esp-templates>`_
+  for more info and alternatives. (Thanks `@Thorbenl`_.)
+
 Features
 ~~~~~~~~
 
 * **Mailgun:** Support Mailgun's new (ESP stored) handlebars templates via `template_id`.
   See `docs <https://anymail.readthedocs.io/en/latest/esps/mailgun/#batch-sending-merge-and-esp-templates>`__.
   (Thanks `@anstosa`_.)
 
+* **SendinBlue:** Support multiple `tags`. (Thanks `@Thorbenl`_.)
+
 
 Other
 ~~~~~
@@ -999,5 +1019,6 @@ Features
 .. _@mbk-ok: https://github.com/mbk-ok
 .. _@RignonNoel: https://github.com/RignonNoel
 .. _@sebbacon: https://github.com/sebbacon
+.. _@Thorbenl: https://github.com/Thorbenl
 .. _@varche1: https://github.com/varche1
 .. _@yourcelf: https://github.com/yourcelf

anymail/backends/sendinblue.py

@@ -83,12 +83,8 @@ def init_payload(self):
 
     def serialize_data(self):
         """Performs any necessary serialization on self.data, and returns the result."""
-
         if not self.data['headers']:
             del self.data['headers']  # don't send empty headers
-        if self.data.get('templateId') and (self.data.pop('textContent', False) or self.data.pop('htmlContent', False)):
-            self.unsupported_feature("overriding template body content")
-
         return self.serialize_json(self.data)
 
     #

anymail/webhooks/sendinblue.py

@@ -49,10 +49,17 @@ def esp_to_anymail_event(self, esp_event):
         except (KeyError, ValueError):
             timestamp = None
 
+        tags = []
         try:
-            tags = [esp_event["tag"]]
+            # If `tags` param set on send, webhook payload includes 'tags' array field.
+            tags = esp_event['tags']
         except KeyError:
-            tags = []
+            try:
+                # If `X-Mailin-Tag` header set on send, webhook payload includes single 'tag' string.
+                # (If header not set, webhook 'tag' will be the template name for template sends.)
+                tags = [esp_event['tag']]
+            except KeyError:
+                pass
 
         try:
             metadata = json.loads(esp_event["X-Mailin-custom"])

docs/esps/sendinblue.rst

@@ -108,7 +108,7 @@ SendinBlue can handle.
 **HTML body required**
   SendinBlue's API returns an error if you attempt to send a message with
   only a plain-text body. Be sure to :ref:`include HTML <sending-html>`
-  content for your messages.
+  content for your messages if you are not using a template.
 
   (SendinBlue *does* allow HTML without a plain-text body. This is generally
   not recommended, though, as some email systems treat HTML-only content as a
@@ -130,10 +130,12 @@ SendinBlue can handle.
   Anymail has no way to communicate an attachment's desired content-type
   to the SendinBlue API if the name is not set correctly.
 
-**Additional template limitations**
-  If you are sending using a SendinBlue template, their API doesn't support overriding the template's
-  body. See the :ref:`templates <sendinblue-templates>`
-  section below.
+**No attachments with templates**
+  If you are sending using a SendinBlue template, their API doesn't support ordinary
+  file attachments. Attempting to send an attachment with a template will result in the
+  SendinBlue API error message, "Please don't pass attachment content & templateId in same
+  request, instead use attachment url only."
+  See the :ref:`templates <sendinblue-templates>` section below.
 
 **Single Reply-To**
   SendinBlue's v3 API only supports a single Reply-To address.
@@ -164,43 +166,85 @@ SendinBlue can handle.
 Batch sending/merge and ESP templates
 -------------------------------------
 
-SendinBlue supports :ref:`ESP stored templates <esp-stored-templates>`
-populated with global merge data for all recipients, but does not
-offer :ref:`batch sending <batch-send>` with per-recipient merge data.
-Anymail's :attr:`~anymail.message.AnymailMessage.merge_data`
-and :attr:`~anymail.message.AnymailMessage.merge_metadata`
-message attributes are not supported with the SendinBlue backend.
+SendinBlue supports :ref:`ESP stored templates <esp-stored-templates>` populated with
+global merge data for all recipients, but does not offer :ref:`batch sending <batch-send>`
+with per-recipient merge data. Anymail's :attr:`~anymail.message.AnymailMessage.merge_data`
+and :attr:`~anymail.message.AnymailMessage.merge_metadata` message attributes are not
+supported with the SendinBlue backend, but you can use Anymail's
+:attr:`~anymail.message.AnymailMessage.merge_global_data` with SendinBlue templates.
+
+SendinBlue supports two different template styles: a `new template language`_
+that uses Django template syntax (with ``{{ param.NAME }}`` style substitutions),
+and an "old" template language that used percent-delimited ``%NAME%`` style
+substitutions. Anymail v7.0 and later require new style templates.
+
+.. versionchanged:: 7.0
+
+    Anymail switched to a SendinBlue API that supports the new template language
+    and removes several limitations from the earlier template send API. But the new API
+    does not support attachments, and can behave oddly if used with old style templates.
+
+.. caution::
+
+    Anymail v7.0 and later work *only* with Sendinblue's *new* template language. You should
+    follow SendinBlue's instructions to `convert each old template`_ to the new language.
+
+    Although unconverted old templates may appear to work with Anymail v7.0, some
+    features may not work properly. In particular, ``reply_to`` overrides and recipient
+    display names are silently ignored when *old* style templates are sent with the
+    *new* API used in Anymail v7.0.
 
 To use a SendinBlue template, set the message's
 :attr:`~anymail.message.AnymailMessage.template_id` to the numeric
 SendinBlue template ID, and supply substitution attributes using
-the messages's :attr:`~anymail.message.AnymailMessage.merge_global_data`:
+the message's :attr:`~anymail.message.AnymailMessage.merge_global_data`:
 
   .. code-block:: python
 
       message = EmailMessage(
-          subject="My Subject",  # optional for SendinBlue templates
-          body=None,  # required for SendinBlue templates
           to=["[email protected]"]  # single recipient...
           # ...multiple to emails would all get the same message
           # (and would all see each other's emails in the "to" header)
       )
-      message.from_email = None  # required for SendinBlue templates
-      message.template_id = 3  # use this SendinBlue template
+      message.template_id = 3   # use this SendinBlue template
+      message.from_email = None  # to use the template's default sender
       message.merge_global_data = {
           'name': "Alice",
           'order_no': "12345",
           'ship_date': "May 15",
       }
 
 Within your SendinBlue template body and subject, you can refer to merge
-variables using %-delimited names, e.g., `%order_no%` or `%ship_date%`
-from the example above.
+variables using Django template syntax, like ``{{ params.order_no }}`` or
+``{{ params.ship_date }}`` for the example above.
+
+The message's :class:`from_email <django.core.mail.EmailMessage>` (which defaults to
+your :setting:`DEFAULT_FROM_EMAIL` setting) will override the template's default sender.
+If you want to use the template's sender, be sure to set ``from_email`` to ``None``
+*after* creating the message, as shown in the example above.
+
+You can also override the template's subject and reply-to address (but not body)
+using standard :class:`~django.core.mail.EmailMessage` attributes.
+
+SendinBlue's template feature does not currently support providing attachment content
+directly with the message---you'll get a SendinBlue API error if you try. If you must
+send file attachments with SendinBlue templates, you can either upload them into
+SendinBlue's template designer, or arrange to have the attachment content hosted on
+a public URL and use Anymail's :attr:`~anymail.message.AnymailMessage.esp_extra`
+to pass the URL to the SendinBlue API (see the Anymail SendinBlue integration tests
+for an example of this). A better---and portable---option may be to avoid SendinBlue
+templates and instead render the email in your Django code, allowing you to add any
+file attachments you want. See :ref:`django-templates` for details.
+
+(Note that SendinBlue doesn't support *inline* image attachments at all, whether you're
+using a template or not.)
+
+
+.. _new template language:
+    https://help.sendinblue.com/hc/en-us/articles/360000268730
 
-Note that SendinBlue's API does not permit overriding a template's
-body. You *must* set it to `None` as shown above,
-or Anymail will raise an :exc:`~anymail.exceptions.AnymailUnsupportedFeature`
-error (if you are not ignoring unsupported features).
+.. _convert each old template:
+    https://help.sendinblue.com/hc/en-us/articles/360000991960
 
 
 .. _sendinblue-webhooks:

tests/test_sendinblue_backend.py

@@ -344,19 +344,6 @@ def test_template_id(self):
         self.assertEqual(data['subject'], 'My Subject')
         self.assertEqual(data['to'], [{'email': "[email protected]", 'name': 'Recipient'}])
 
-    def test_unsupported_template_overrides(self):
-        # SendinBlue doesn't allow overriding any template headers/content
-        message = mail.EmailMessage(to=['[email protected]'])
-        message.template_id = "9"
-
-        message.body = "nope, can't change text body"
-        with self.assertRaisesMessage(AnymailUnsupportedFeature, "overriding template body content"):
-            message.send()
-        message.content_subtype = "html"
-        with self.assertRaisesMessage(AnymailUnsupportedFeature, "overriding template body content"):
-            message.send()
-        message.body = None
-
     def test_merge_data(self):
         self.message.merge_data = {
             '[email protected]': {':name': "Alice", ':group': "Developers"},

tests/test_sendinblue_integration.py

@@ -63,7 +63,7 @@ def test_all_options(self):
             headers={"X-Anymail-Test": "value", "X-Anymail-Count": 3},
 
             metadata={"meta1": "simple string", "meta2": 2},
-            tags=["tag 1"],  # SendinBlue only supports single tags
+            tags=["tag 1", "tag 2"],
         )
         message.attach_alternative('<p>HTML content</p>', "text/html")  # SendinBlue requires an HTML body
 
@@ -76,22 +76,32 @@ def test_all_options(self):
 
     def test_template(self):
         message = AnymailMessage(
-            template_id=1,  # There is a template with this id in the Anymail test account
-            to=["[email protected]"],  # SendinBlue doesn't allow recipient display names with templates
-            reply_to=["[email protected]"],
+            template_id=5,  # There is a *new-style* template with this id in the Anymail test account
+            from_email='Sender <[email protected]>',  # Override template sender
+            to=["Recipient <[email protected]>"],  # No batch send (so max one recipient suggested)
+            reply_to=["Do not reply <[email protected]>"],
             tags=["using-template"],
             headers={"X-Anymail-Test": "group: A, variation: C"},
             merge_global_data={
-                # The Anymail test template includes `%SHIP_DATE%` and `%ORDER_ID%` variables
+                # The Anymail test template includes `{{ params.SHIP_DATE }}`
+                # and `{{ params.ORDER_ID }}` substitutions
                 "SHIP_DATE": "yesterday",
                 "ORDER_ID": "12345",
             },
             metadata={"customer-id": "ZXK9123", "meta2": 2},
         )
-        message.from_email = None  # Required for SendinBlue templates
 
-        # Attachments don't work with templates in Sendinblue's newer API:
-        # message.attach("attachment1.txt", "Here is some\ntext for you", "text/plain")
+        # Normal attachments don't work with Sendinblue templates:
+        #   message.attach("attachment1.txt", "Here is some\ntext for you", "text/plain")
+        # If you can host the attachment content on some publicly-accessible URL,
+        # this *non-portable* alternative allows sending attachments with templates:
+        message.esp_extra = {
+            'attachment': [{
+                'name': 'attachment1.txt',
+                # URL where Sendinblue can download the attachment content while sending:
+                'url': 'https://raw.githubusercontent.com/anymail/django-anymail/master/AUTHORS.txt',
+            }]
+        }
 
         message.send()
         self.assertEqual(message.anymail_status.status, {'queued'})  # SendinBlue always queues

tests/test_sendinblue_webhooks.py

@@ -21,8 +21,9 @@ def call_webhook(self):
 
 @tag('sendinblue')
 class SendinBlueDeliveryTestCase(WebhookTestCase):
-    # SendinBlue's webhook payload data doesn't seem to be documented anywhere.
-    # There's a list of webhook events at https://apidocs.sendinblue.com/webhooks/#3.
+    # SendinBlue's webhook payload data is partially documented at
+    # https://help.sendinblue.com/hc/en-us/articles/360007666479,
+    # but it's not completely up to date.
     # The payloads below were obtained through live testing.
 
     def test_sent_event(self):
@@ -40,7 +41,13 @@ def test_sent_event(self):
             "ts_epoch": 1520363423000,  # 2018-03-06 19:10:23.000+00:00 -- UTC (milliseconds)
 
             "X-Mailin-custom": '{"meta": "data"}',
-            "tag": "test-tag",  # note: for template send, is template name if no other tag provided
+            # "tag" is JSON-serialized tags array if `tags` param set on send,
+            #   else single tag string if `X-Mailin-Tag` header set on send,
+            #   else template name if sent using a template,
+            #   else not present.
+            # "tags" is tags list if `tags` param set on send, else not present.
+            "tag": '["tag1","tag2"]',
+            "tags": ["tag1", "tag2"],
             "template_id": 12,
             "sending_ip": "333.33.33.33",
         }
@@ -58,7 +65,7 @@ def test_sent_event(self):
         self.assertIsNone(event.event_id)  # SendinBlue does not provide a unique event id
         self.assertEqual(event.recipient, "[email protected]")
         self.assertEqual(event.metadata, {"meta": "data"})
-        self.assertEqual(event.tags, ["test-tag"])
+        self.assertEqual(event.tags, ["tag1", "tag2"])
 
     def test_delivered_event(self):
         raw_event = {
@@ -91,6 +98,7 @@ def test_hard_bounce(self):
             "message-id": "<[email protected]>",
             # the leading space in the reason is as received in actual testing:
             "reason": " RecipientError: 550 5.5.0 Requested action not taken: mailbox unavailable.",
+            "tag": "header-tag",
         }
         response = self.client.post('/anymail/sendinblue/tracking/',
                                     content_type='application/json', data=json.dumps(raw_event))
@@ -102,6 +110,7 @@ def test_hard_bounce(self):
         self.assertEqual(event.reject_reason, "bounced")
         self.assertEqual(event.mta_response,
                          " RecipientError: 550 5.5.0 Requested action not taken: mailbox unavailable.")
+        self.assertEqual(event.tags, ["header-tag"])
 
     def test_soft_bounce_event(self):
         raw_event = {