release: v1.2.0 (#9)

* feat: added flag for dry-run

* feat: self-hosted actions

* feat: iga support (#7)

* wip: adding IGA support

n.b. IGA's search is inaccurate

* wip: display support

* feat: add email support for IGA

* fix: various little tweaks

* feat: improved IGA search results

* refactor: moved merchants around

* feat: import shopping list from file

* feat: bulk operations with json (#8)

* refactor: allow from-addr to be taken from env-vars

* feat: support for bulk json args

* feat: added session caching for bulk runs

* fix: updated GH actions with self-hosted runner fall back

* docs: updated docs for release

Author: MattTimms

.env.example

@@ -1 +1,2 @@
 MAILERSEND_API_KEY=
+FROM_ADDRESS=

.github/workflows/run.yml

@@ -6,9 +6,14 @@ on:
     - cron: '30 20 * * 2'
   workflow_dispatch:
     inputs:
+      dry_run:
+        default: true
+        type: boolean
 
+
+# n.b shocked to learn GH actions don't support yaml anchors (https://github.com/actions/runner/issues/1182)
 jobs:
-  coles_vs_woolies:
+  github:
     runs-on: ubuntu-latest
     steps:
       - name: Check out repo
@@ -22,16 +27,41 @@ jobs:
         env:
           timezone: 'Australia/Melbourne'
         run: |
-          sudo timedatectl set-timezone $timezone
           pip install -r requirements.txt
-      - name: Unit Tests
-        run: python -m unittest
+      - name: Place shopping list
+        run: |
+          echo '${{ secrets.SHOPPING_LIST }}' >> shopping-list.json
+      - name: Email comparisons
+        env:
+          MAILERSEND_API_KEY: ${{ secrets.MAILERSEND_API_KEY }}
+          FROM_ADDRESS: no-reply@${{ secrets.DOMAIN }}
+        run: |
+          python coles_vs_woolies send shopping-list.json \
+            ${{ (github.event_name == 'workflow_dispatch' && inputs.dry_run) && '--dry_run' || '' }}
+  local:
+    needs: github
+    if: always() && contains(needs.*.result, 'failure')
+    runs-on: self-hosted
+    steps:
+      - name: Check out repo
+        uses: actions/checkout@v3
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.10'
+          cache: 'pip'
+      - name: Install dependencies
+        env:
+          timezone: 'Australia/Melbourne'
+        run: |
+          pip install -r requirements.txt
+      - name: Place shopping list
+        run: |
+          echo '${{ secrets.SHOPPING_LIST }}' >> shopping-list.json
       - name: Email comparisons
         env:
           MAILERSEND_API_KEY: ${{ secrets.MAILERSEND_API_KEY }}
+          FROM_ADDRESS: no-reply@${{ secrets.DOMAIN }}
         run: |
-          python coles_vs_woolies send \
-            ${{ vars.GROCERY_LIST }} \
-            --to_addrs ${{ inputs.to_addr || secrets.TO_ADDRS }} \
-            --from_addr no-reply@${{ secrets.DOMAIN }} 
-      
+          python coles_vs_woolies send shopping-list.json \
+            ${{ (github.event_name == 'workflow_dispatch' && inputs.dry_run) && '--dry_run' || '' }}

.github/workflows/test.yml

@@ -1,13 +1,14 @@
 name: Tests
 
 on:
+  workflow_dispatch:
   push:
     branches:
       - 'main'
 
 jobs:
   test:
-    runs-on: ubuntu-latest
+    runs-on: self-hosted
     strategy:
       matrix:
         python-version: [ "3.9", "3.10", "3.11" ]

.gitignore

@@ -1,4 +1,7 @@
 .idea
+.github/actions-runner
+shopping-list.json
+shopping-list.bkup.json
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

CHANGELOG.md

@@ -1,5 +1,11 @@
 # CHANGELOG.md
 
+## v1.2.0
+
+* Added 'iga' merchant
+* Added CLI import shopping list from txt file
+* Added CLI import of multiple shopping lists from json file
+
 ## v1.1.0 - 26/01/2023
 
 * Added `bcc` when sending to multiple recipients

README.md

@@ -1,9 +1,11 @@
 # 🍎 coles_vs_woolies 🍏
 
-![Python](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11-blue)  
+![Python](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11-blue)
 [![pass](https://github.com/MattTimms/coles_vs_woolies/actions/workflows/test.yml/badge.svg)](https://github.com/MattTimms/coles_vs_woolies/actions/workflows/test.yml)
 [![working just fine for me](https://github.com/MattTimms/coles_vs_woolies/actions/workflows/run.yml/badge.svg)](https://github.com/MattTimms/coles_vs_woolies/actions/workflows/run.yml)
 
+🍅 `iga` now supported.
+
 Receive an email every week comparing the price of products you buy often.
 
 Scrap Aussie grocers' public APIs, schedule a GitHub Action to run weekly, and
@@ -73,15 +75,23 @@ $ python coles_vs_woolies --help
 #         "Cadbury Dairy Milk Chocolate Block 180g"
 #         "Connoisseur Ice Cream Vanilla Caramel Brownie 4 Pack"
 #         --to_addrs <me@gmail.com> <you@gmail.com> 
-#         --from_addr <no-reply@domain.com>
-#         --mailersend_api_key=<MAILERSEND_API_KEY>
+```
+
+```shell
+cp .env.example .env
+# populate .env to simplify calls
 ```
 
 ## Install w/ GitHub Actions
 
+n.b. I found sporadic success with GitHub-hosted runners; I would recommend setting
+up [self-hosted GitHub runners](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners)
+for consistent success if you wished to continue using GitHub actions rather than running a simple cron job.
+
 1. Fork this repo
 2. Read the GitHub Action workflow [run.yml](.github/workflows/run.yml)
 3. Add GitHub Action Variables & Secrets for those in the [run.yml](.github/workflows/run.yml)
+    - n.b. storing a `shopping-list.json` as a minified json string should do the job
 4. Manually invoke the GitHub Action & confirm an email was received
 
 ## Getting the best results

coles_vs_woolies/__main__.py

@@ -1,6 +1,11 @@
 import argparse
+import json
+import os
 import textwrap
 from argparse import ArgumentParser
+from typing import List
+
+from pydantic import BaseModel
 
 from coles_vs_woolies.main import display, send
 
@@ -15,8 +20,6 @@ def cli():
         "Cadbury Dairy Milk Chocolate Block 180g"
         "Connoisseur Ice Cream Vanilla Caramel Brownie 4 Pack"
         --to_addrs <me@gmail.com> <you@gmail.com> 
-        --from_addr <no-reply@domain.com>
-        --mailersend_api_key=<MAILERSEND_API_KEY>
     '''
 
     parser = ArgumentParser(
@@ -27,31 +30,65 @@ def cli():
     )
 
     subparsers = parser.add_subparsers(dest='action')
-    display_parser = subparsers.add_parser('display', help='Display product price comparisons')
-    send_parser = subparsers.add_parser('send', help='Email product price comparisons')
 
-    help_product = 'List of descriptive product search terms. Brand, package weight or size should be included. E.g. ' \
-                   '"Cadbury Dairy Milk Chocolate Block 180g" "Connoisseur Ice Cream Vanilla Caramel Brownie 4 Pack"'
+    help_product = 'List of descriptive product search terms. Brand, package weight or size should be included. ' \
+                   'Can be file path. E.g. "Cadbury Dairy Milk Chocolate Block 180g"' \
+                   '"Connoisseur Ice Cream Vanilla Caramel Brownie 4 Pack"'
+
+    # Display parser
+    display_parser = subparsers.add_parser('display', help='Display product price comparisons')
     display_parser.add_argument('products', nargs='+', help=help_product)
-    send_parser.add_argument('products', nargs='+', help=help_product)
 
-    send_parser.add_argument('-t', '--to_addrs', nargs='+', help="Recipients' email address.", required=True)
-    send_parser.add_argument('-f', '--from_addr', type=str,
-                             help="Sender's email address. Domain must match that verified with MailerSend.",
-                             required=True)
-    send_parser.add_argument('-m', '--mailersend_api_key', type=str, help='MailerSend API key.', required=False)
+    # Send parser
+    send_parser = subparsers.add_parser('send', help='Email product price comparisons')
+    send_parser.add_argument('products', nargs='+', help=help_product)
+    send_parser.add_argument('-t', '--to_addrs', nargs='+', help="Recipients' email address.", required=False)
     send_parser.add_argument('-o', '--out_dir', type=str, help='Directory for saving copy of the email HTML template.',
                              required=False)
+    send_parser.add_argument('-d', '--dry_run', action='store_true', help='Disable email delivery',
+                             default=False, required=False)
 
-    args = vars(parser.parse_args())
+    # Parse inputs
+    kwargs = vars(parser.parse_args())
+    action = kwargs.pop('action')
 
-    action = args.pop('action')
-    products = sorted(list(set(args.pop('products'))))
+    _product_inputs = kwargs.pop('products')
+    if os.path.isfile(fp := _product_inputs[0]) and fp.endswith('.json'):
+        with open(fp, 'r') as f:
+            jobs = [_JsonInput.parse_obj(x) for x in json.load(f)]
+        _ = kwargs.pop('to_addrs', None)
+        for job in jobs:
+            _run(action, job.products, to_addrs=job.to_addrs, **kwargs)
+    else:
+        if action == 'send' and kwargs.get('to_addrs', None) is None:
+            parser.error('the following arguments are required: -t/--to_addrs')
+        products = _parse_product_inputs(_product_inputs)
+        _run(action, products, **kwargs)
+
+
+class _JsonInput(BaseModel):
+    to_addrs: List[str]
+    products: List[str]
+
+
+def _run(action: str, products: List[str], **kwargs):
     if action == 'send':
-        send(products=products, **args)
+        send(products=products, **kwargs)
     else:
         display(products=products)
 
 
+def _parse_product_inputs(args: List[str]) -> List[str]:
+    """ Return product list from input list of products/file-paths """
+    products = []
+    for input_ in args:
+        if os.path.isfile(input_):
+            with open(input_, 'r') as f:
+                products.extend(f.read().splitlines())
+        else:
+            products.append(input_)
+    return sorted(list(set(products)))
+
+
 if __name__ == '__main__':
     cli()

coles_vs_woolies/emailing/generate.py

@@ -21,30 +21,34 @@ def generate_weekly_email(product_offers: ProductOffers, out_path: str = None) -
     with open(_TEMPLATE_DIR / 'weekly.html', 'r', encoding="utf-8") as f:
         html_template = f.read()
     with open(_TEMPLATE_DIR / 'snippets/table.html', 'r', encoding="utf-8") as f:
-        html_table = f.read()
+        html_template_table = f.read()
     with open(_TEMPLATE_DIR / 'snippets/table_row.html', 'r', encoding="utf-8") as f:
-        html_table_row: str = f.read()
+        html_template_table_row: str = f.read()
 
     # Replace template variables
     rows = []
     green = '#008000'
     light_grey = '#afafaf'
     for product_name, offers in product_offers.items():
-        row_ = html_table_row
-        row_ = row_.replace('{{ product }}', product_name)
+        row_template = html_template_table_row
+        row_template = row_template.replace('{{ product }}', product_name)
+
+        # Replace merchant offers
         lowest_price = min(offers).price
         is_sales = any((offer.is_on_special for offer in offers))
         for offer in offers:
             merchant = offer.merchant
             price = offer.price if offer.price is not None else 'n/a'
             colour = green if is_sales and price == lowest_price else light_grey
-            row_ = row_.replace('{{ %(merchant)s_price }}' % {"merchant": merchant},
-                                f'<a href="{offer.link}" style="color:{colour};text-decoration:inherit;">${price}</a>')
+            zero_padding = '<span style="opacity:0;">0</span>' if len(str(price).split('.')[-1]) == 1 else ''
+
+            html_replacement =  f'<a href="{offer.link}" style="color:{colour};text-decoration:inherit;">${price}{zero_padding}</a>'
+            row_template = row_template.replace('{{ %(merchant)s_price }}' % {"merchant": merchant}, html_replacement)
 
-        rows.append(row_)
+        rows.append(row_template)
 
-    html_table = html_table.replace('{{ rows }}', ''.join(rows))
-    html_template = html_template.replace('{{ table }}', html_table)
+    html_template_table = html_template_table.replace('{{ rows }}', ''.join(rows))
+    html_template = html_template.replace('{{ table }}', html_template_table)
 
     # Add time
     year, week, weekday = datetime.datetime.now().isocalendar()

coles_vs_woolies/emailing/mailer_send.py

@@ -5,8 +5,15 @@
 from dotenv import load_dotenv, find_dotenv
 from mailersend import emails
 
+load_dotenv(dotenv_path=find_dotenv())
+_MAILERSEND_API_KEY = os.getenv('MAILERSEND_API_KEY')
+_FROM_ADDR = os.getenv('FROM_ADDRESS')
 
-def send(email_html: str, to_addrs: List[str], from_addr: str, mailersend_api_key: str = None):
+
+def send(email_html: str,
+         to_addrs: List[str],
+         from_addr: str = None,
+         mailersend_api_key: str = None):
     """
     Send an email with MailerSend.
     :param email_html: email html template or file path to template.
@@ -15,17 +22,19 @@ def send(email_html: str, to_addrs: List[str], from_addr: str, mailersend_api_ke
     :param mailersend_api_key: API key for MailerSend. If not given, pulled from env-vars.
     :return:
     """
+    # Validate MailerSend requirements
+    from_addr = from_addr or _FROM_ADDR
+    if from_addr is None:
+        raise ValueError("MailerSend `from_addr` not provided")
+    mailersend_api_key = mailersend_api_key or _MAILERSEND_API_KEY
+    if mailersend_api_key is None:
+        raise ValueError("MailerSend API Key not provided")
+
     # If file path, load email html template
     if os.path.isfile(email_html):
         with open(email_html, 'r', encoding='utf-8') as fp:
             email_html = fp.read()
 
-    # Setup MailerSend email
-    if mailersend_api_key is None:
-        load_dotenv(dotenv_path=find_dotenv())
-        mailersend_api_key = os.getenv('MAILERSEND_API_KEY')
-        if mailersend_api_key is None:
-            raise ValueError("MailerSend API Key not provided")
     mailer = emails.NewEmail(mailersend_api_key)
 
     # Define an empty dict to populate with mail values

coles_vs_woolies/emailing/templates/snippets/table.html

@@ -1,21 +1,26 @@
 <table width="100%" cellpadding="0" cellspacing="0" style="border-collapse:collapse;">
   <tr>
     <td
-      style="padding-top:10px;padding-bottom:20px;padding-right:0;padding-left:0;word-break:break-word;font-family:'Inter', Helvetica, Arial, sans-serif;font-size:16px;line-height:24px;">
+      style="
+      padding: 10px 0 20px;
+      word-break:break-word;
+      font-size:16px;
+      line-height:24px;"
+    >
       <table class="table" width="100%" cellpadding="0" cellspacing="0" style="border-collapse:collapse;">
         <tr>
-          <th align="left"
-              style="font-family:'Inter', Helvetica, Arial, sans-serif;padding-top:10px;padding-bottom:10px;padding-right:0;padding-left:0;color:#85878E;font-size:13px;font-weight:600;line-height:18px;">
+          <th align="left" class="product-table-header">
             Products
           </th>
-          <th align="right"
-              style="font-family:'Inter', Helvetica, Arial, sans-serif;padding-top:10px;padding-bottom:10px;padding-right:0;padding-left:0;color:#85878E;font-size:13px;font-weight:600;line-height:18px;width:60px;">
+          <th align="right" class="product-table-header" style="width:60px;">
             Coles
           </th>
-          <th align="right"
-              style="font-family:'Inter', Helvetica, Arial, sans-serif;padding-top:10px;padding-bottom:10px;padding-right:0;padding-left:0;color:#85878E;font-size:13px;font-weight:600;line-height:18px;width:60px;">
+          <th align="right" class="product-table-header" style="width:60px;">
             Woolies
           </th>
+          <th align="middle" class="product-table-header" style="width:60px;">
+            IGA
+          </th>
         </tr>
         {{ rows }}
       </table>