refactor: make listener a Django management command

Author: arcstur

docs/architecture.rst

@@ -33,9 +33,9 @@ This is a fairly reliable endpoint which also lets us resume the stream from a r
 ingestion stopped for some reason. By default, the listener tries to resume listening from the 
 date of the latest edit it has ingested.
 
-This process can be invoked directly as a script::
+This process can be invoked directly as a Django management command::
 
-   python listener.py
+   python3 manage.py listener
 
 It can be run as an `attached daemon to uwsgi <https://uwsgi-docs.readthedocs.io/en/latest/AttachingDaemons.html>`_.
 

docs/install.rst

@@ -33,7 +33,7 @@ start a local web server, you can then access your EditGroups instance at ``http
 By default, there will not be much to see as the database will be empty. To get some data in, you need
 to run the listener script, which reads the Wikidata event stream and populates the database::
 
-    python listener.py
+   python3 manage.py listener
 
 You will also need to run Celery, which will periodically annotate edits which need inspection,
 as well as providing the undo functionality (if you have set up OAuth, see below)::
@@ -130,7 +130,7 @@ Put the following content in ``~/www/python/uwsgi.ini``::
   static-map = /static=/data/project/editgroups/www/python/src/static
 
   master = true
-  attach-daemon = /data/project/editgroups/www/python/venv/bin/python3 /data/project/editgroups/www/python/src/listener.py
+  attach-daemon = /data/project/editgroups/www/python/venv/bin/python3 /data/project/editgroups/www/python/src/manage.py listener
 
 and run ``./manage.py collectstatic`` in the ``~/www/python/src`` directory. The listener will be an attached dameon, restarting with webservice restart.
 

listener.py

@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+import sys
+from datetime import datetime
+from datetime import timedelta
+
+from django.core.management.base import BaseCommand
+
+from store.stream import WikiEditStream
+from store.utils import grouper
+from store.models import Edit
+
+
+class Command(BaseCommand):
+    """
+    Amount of time to look back  when restarting
+    the listener. This helps make sure that we don't
+    lose any edit when the listener is restarted.
+    """
+
+    LOOKBEHIND_OFFSET = timedelta(minutes=5)
+    help = "Listens to edits with EventStream"
+
+    def handle(self, *args, **options):
+        print("Listening to edits...")
+        s = WikiEditStream()
+        try:
+            latest_edit_seen = Edit.objects.order_by("-timestamp")[0].timestamp
+            fetch_from = latest_edit_seen - self.LOOKBEHIND_OFFSET
+        except IndexError:
+            fetch_from = None
+        offset = fetch_from.isoformat() if fetch_from else "now"
+        print("Starting from offset %s" % offset)
+
+        for i, batch in enumerate(grouper(s.stream(fetch_from), 50)):
+            if i % 50 == 0:
+                print("batch %d" % i)
+                print(datetime.fromtimestamp(batch[0].get("timestamp")))
+            sys.stdout.flush()
+            Edit.ingest_edits(batch)
+
+        print("End of stream")