|
| 1 | +# Step 5a - Add Push Tasks to App Engine `webapp2` & `ndb` sample app |
| 2 | + |
| 3 | +## Introduction |
| 4 | + |
| 5 | +The goal of the Step 5 series of codelabs and repos like this is to help App Engine developers migrate from [Python 2 App Engine (Push) Task Queues](https://cloud.google.com/appengine/docs/standard/python/taskqueue/push) to [Google Cloud Tasks](https://cloud.google.com/tasks). They are meant to be *complementary* to the official [migrating push queues to Cloud Tasks documentation](https://cloud.google.com/appengine/docs/standard/python/taskqueue/push/migrating-push-queues) and [corresponding code samples](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard/migration/taskqueue) and offer some additional benefits: |
| 6 | +- Video content for those who prefer visual learning in addition to reading |
| 7 | +- Codelab tutorials give hands-on experience and build "migration muscle-memory" |
| 8 | +- More code samples gives developers a deeper understanding of migration steps |
| 9 | + |
| 10 | +In this codelab/repo, participants start with the code in the (completed) [Step 1 repo](https://github.com/googlecodelabs/migrate-python-appengine-datastore/tree/master/step1-flask-gaendb-py2) where developers migrated to the Flask web framework and add support for Push Task Queues using the App Engine `taskqueue` API library. As you will recall, the sample app registers each visit (`GET` request to `/`) by creating a new `Visit` Entity for it then fetches and displays the 10 most recent `Visit`s in the web UI. |
| 11 | + |
| 12 | +This codelab step adds a push task to delete all `Visit`s older than the oldest entry. If you haven't completed the [Step 1 codelab](https://codelabs.developers.google.com/codelabs/cloud-gae-python-migrate-1-flask), we recommend you do so to familiarize yourself with the Step 1 codebase. |
| 13 | + |
| 14 | +--- |
| 15 | + |
| 16 | +## Augment sample with new feature |
| 17 | + |
| 18 | +Rather than a migration, this step adds use of push Task Queues to the existing Step 1 app. The only modifications required are for the `main.py` application file and `templates/index.html` web template. The steps: |
| 19 | + |
| 20 | +1. Add new Python `import`s |
| 21 | + - Add use of Python standard library date and time utilities |
| 22 | + - (optional but helpful) Add logging and function ["docstrings"](http://python.org/dev/peps/pep-0257/#id15) |
| 23 | +1. Save timestamp of last (displayed) `Visit` |
| 24 | +1. Add "delete old(est) entries" task |
| 25 | +1. Display deletion message in web UI template |
| 26 | + |
| 27 | +### Add new Python `import`s |
| 28 | + |
| 29 | +It's useful to add logging to applications to give the developer (and the user) more information (as long as it's useful). For Python 2 App Engine, this is done by using the Python standard library `logging` module. For date & time functionality, add use of the `datetime.datetime` class as well as the `time` module. |
| 30 | + |
| 31 | +- BEFORE: |
| 32 | + |
| 33 | +```python |
| 34 | +from flask import Flask, render_template, request |
| 35 | +from google.appengine.ext import ndb |
| 36 | +``` |
| 37 | + |
| 38 | +The Python best practices of alphabetized group listing order: |
| 39 | +1. Standard library modules first |
| 40 | +1. Third-party globally-installed packages |
| 41 | +1. Locally-installed packages |
| 42 | +1. Application imports |
| 43 | + |
| 44 | +Following that recommendation, your imports should look like this when done: |
| 45 | + |
| 46 | +- AFTER: |
| 47 | + |
| 48 | +```python |
| 49 | +import logging |
| 50 | +from datetime import datetime |
| 51 | +import time |
| 52 | +from flask import Flask, render_template, request |
| 53 | +from google.appengine.api import taskqueue |
| 54 | +from google.appengine.ext import ndb |
| 55 | +``` |
| 56 | + |
| 57 | +### Save timestamp of last (displayed) `Visit` |
| 58 | + |
| 59 | +The `fetch_visits()` function queries for the most recent visits. Add code to save the timestamp of the last `Visit`. |
| 60 | + |
| 61 | +- BEFORE: |
| 62 | + |
| 63 | +```python |
| 64 | +def fetch_visits(limit): |
| 65 | + return (v.to_dict() for v in Visit.query().order( |
| 66 | + -Visit.timestamp).fetch(limit)) |
| 67 | +``` |
| 68 | + |
| 69 | +Instead of immediately returning all `Visit`s, we need to save the results, grab the last `Visit` and save its timestamp, both as a `str`ing (to display) and `float` (to send to the task). |
| 70 | + |
| 71 | +- AFTER: |
| 72 | + |
| 73 | +```python |
| 74 | +def fetch_visits(limit): |
| 75 | + 'get most recent visits & add task to delete older visits' |
| 76 | + data = Visit.query().order(-Visit.timestamp).fetch(limit) |
| 77 | + oldest = time.mktime(data[-1].timestamp.timetuple()) |
| 78 | + oldest_str = time.ctime(oldest) |
| 79 | + logging.info('Delete entities older than %s' % oldest_str) |
| 80 | + taskqueue.add(url='/trim', params={'oldest': oldest}) |
| 81 | + return (v.to_dict() for v in data), oldest_str |
| 82 | +``` |
| 83 | + |
| 84 | +The `data` variable holds the `Visit`s previously returned immediately, and `oldest` is the timestamp of the oldest displayed `Visit` in seconds (as a `float`) since the epoch, retrieved by (extracting `datetime` object, morphed to Python [time 9-tuple normalized form](https://docs.python.org/library/time), then converted to `float`). A string version is also created for display purposes. A new push task is added, calling the handler (`/trim`) with `oldest` as its only parameter. |
| 85 | + |
| 86 | +The same payload as the Step 1 `fetch_visits()` is returned to the caller in addition to `oldest` as a string. Following good practices, a function docstring was added (first unassigned string) along with an application log at the `INFO` level via `logging.info()`. |
| 87 | + |
| 88 | +### Add "delete old(est) entries" task |
| 89 | + |
| 90 | +While deletion of old `Visit`s could've easily been accomplished in `fetch_visits()`, this was a great excuse to make it a task which is handled asynchronously after `fetch_user()` returns, and the data is presented to the user. This improves the user experience because there is no delay in waiting for the deletion of the older Datastore entities to complete. |
| 91 | + |
| 92 | +```python |
| 93 | +@app.route('/trim', methods=['POST']) |
| 94 | +def trim(): |
| 95 | + '(push) task queue handler to delete oldest visits' |
| 96 | + oldest = request.form.get('oldest', type=float) |
| 97 | + keys = Visit.query( |
| 98 | + Visit.timestamp < datetime.fromtimestamp(oldest) |
| 99 | + ).fetch(keys_only=True) |
| 100 | + nkeys = len(keys) |
| 101 | + if nkeys: |
| 102 | + logging.info('Deleting %d entities: %s' % ( |
| 103 | + nkeys, ', '.join(str(k.id()) for k in keys))) |
| 104 | + ndb.delete_multi(keys) |
| 105 | + else: |
| 106 | + logging.info('No entities older than: %s' % time.ctime(oldest)) |
| 107 | + return '' # need to return SOME string w/200 |
| 108 | +``` |
| 109 | + |
| 110 | +Push tasks are `POST`ed to the handler, so that must be specified (default: `GET`). Once the timestamp of the `oldest` visit is decoded, a Datastore query to find all entities strictly older than its timestamp is created. None of the actual data is needed, so a faster "keys-only" query is used. The number of entities to delete is logged, and the deletion command (`ndb.delete_multi()`) given. Logging also occurs if there are no entities to delete. A return value is necessary to go along with the HTTP 200 return code, so use an empty string to be efficient. |
| 111 | + |
| 112 | + |
| 113 | +### Display deletion message in web UI template |
| 114 | + |
| 115 | +It's also a good practice to have "docstrings" to document functionality, so add those as well. Add the following snippet after the unnumbered list of `Visit`s but before the closing `</body>` tag: |
| 116 | + |
| 117 | +```html+jinja |
| 118 | +{% if oldest %} |
| 119 | + <b>Deleting visits older than:</b> {{ oldest }}</p> |
| 120 | +{% endif %} |
| 121 | +``` |
| 122 | + |
| 123 | +### Configuration |
| 124 | + |
| 125 | +There are no changes to any of the configuration (`app.yaml`, `appengine_config.py`, `requirements.txt`) files. |
| 126 | + |
| 127 | +--- |
| 128 | + |
| 129 | +## Next |
| 130 | + |
| 131 | +Try the app in the local development server (`dev_appserver.py app.yaml`), debug (if nec.), and deploy to App Engine and confirm everything still works. Once you're satisfied, move onto the next step: |
| 132 | + |
| 133 | +- [**Step 5b:**](/step5b-cloud-ndb-tasks-py2) Migrate your app away from App Engine built-in libraries like `ndb` & `taskqueue` to Cloud NDB & Cloud Tasks |
0 commit comments