Merge pull request getodk#748 from getodk/remove-google-backups

Remove Google Drive backups

Author: matthew-white

CONTRIBUTING.md

@@ -98,7 +98,7 @@ In general, we try to follow these principles in dealing with encrypted and sens
 
 Here is an overview of the directory structure inside of `lib/`:
 
-* `bin/` contains all the executable scripts. These include `run-server.js`, which actually starts the ODK Central Backend server, but also the `cli.js` command line administration utility, and scripts like the `backup.js` executable which runs the configured backup.
+* `bin/` contains all the executable scripts. These include `run-server.js`, which actually starts the ODK Central Backend server, but also the `cli.js` command line administration utility, and scripts like the `restore.js` executable which restores the database from a backup file.
 * `data/` files deal with Submission row data in various ways. They are very independent, callable from any context (they don't rely on any database code, for example), and so they are separated out here. The `schema.js` file contains a lot of utilities for understanding and using XForms XML schema definitions, while the other files deal with Submission row data.
 * `http/` defines a lot of core functionality for how we use Express, the Node.js server framework we rely on. The `endpoint.js` Endpoints are wrapper functions we use on most endpoints in our code (see `resources/`), `middleware.js` are middleware layers that catch the requests as they come in and decorate various common information onto them, and `service.js` is the glue code that actually ties together the middleware and resources to form an Express service.
 * `model/` is the most complex area of code; it defines all the Things (Users, Forms, etc) in the server and how they are stored and retrieved from the database. It also contains our database migrations. For a deeper understanding of the structure, please see this explanation of our [Database Design](./docs/database.md).
@@ -108,6 +108,6 @@ Here is an overview of the directory structure inside of `lib/`:
     * `model/container.js` is an important file; all the code in `model/frames` and `model/query` rely on dependency injection, and this package file is where they are all declared so that the server knows about them. If you are adding any `Frames`s or `query` modules, you'll need to visit this file and add a line or two at the bottom.
 * `outbound/` contains various utilities for templating and sending messages of various kinds: `mail.js`, for example, contains all the email messages we might send as well as the machinery to do so, and `openrosa.js` contains templates for different OpenRosa XML responses.
 * `resources/` is where all the HTTP API endpoints are declared; they are the REST Resources of the server (hence the name). This is, in the layers we describe above, the outermost layer. Most of our resources are wrapped in an `endpoint()` function of some sort, declared in `http/`. The `endpoint()` functions help process the final output of each endpoint, performing useful operations like resolving `Promise`s, managing output `Stream`s, and handling some errors.
-* `task/` contains many utility operations that we use to construct the scripts in `bin/`, like `cli.js` and the backup/restore scripts. They are native `Promise`s and can be easily composed with `.then()`, like any other `Promise`. In `task.js`, there are various helpers to help print meaningful output for these operations and log audit logs on operations. There is also a `task.withContainer()` method which helps if you need any `model`/`Instance` code in a task (see `task/account.js` for usage).
+* `task/` contains many utility operations that we use to construct the scripts in `bin/`, like `cli.js` and the restore script. They are native `Promise`s and can be easily composed with `.then()`, like any other `Promise`. In `task.js`, there are various helpers to help print meaningful output for these operations and log audit logs on operations. There is also a `task.withContainer()` method which helps if you need any `model`/`Instance` code in a task (see `task/account.js` for usage).
 * `util/` defines a large number of really tiny helper functions and data structures that are used all over the application, as well as some generic tasks like cryptography. Of note is `option.js`, which defines the `Option[T]` class we use when a return type might be empty.
 

config/default.json

@@ -26,10 +26,6 @@
       "sysadminAccount": "[email protected]"
     },
     "external": {
-      "google": {
-        "clientId": "660095633112-h7bhsjenhp1agd0c4v3cmqk6bccgkdu0.apps.googleusercontent.com",
-        "clientSecret": "lzu2vK1NFqqd6Y5HiN-7ByvE"
-      },
       "sentry": {},
       "analytics": {
         "url": "https://data.getodk.cloud/v1/key/eOZ7S4bzyUW!g1PF6dIXsnSqktRuewzLTpmc6ipBtRq$LDfIMTUKswCexvE0UwJ9/projects/1/submission",

docs/api.md

@@ -32,6 +32,12 @@ Finally, **system information and configuration** is available via a set of spec
 
 Here major and breaking changes to the API are listed by version.
 
+### ODK Central v2023.1
+
+**Removed**:
+
+- Scheduled backups to Google Drive are no longer supported. As a result, backups are no longer configurable. It is no longer possible to get or terminate a backups configuration or to use a backups configuration to GET a Direct Backup. For more information about these changes, please see [this topic](https://forum.getodk.org/t/backups-to-google-drive-from-central-will-stop-working-after-jan-31st/38895) in the ODK Forum.
+
 ### ODK Central v2022.3
 
 **Added**:
@@ -3770,91 +3776,7 @@ Identical to [the non-Draft version](/reference/odata-endpoints/odata-form-servi
 
 # Group System Endpoints
 
-There are some resources available for getting or setting system information and configuration. You can [set the Usage Reporting configuration](/reference/system-endpoints/usage-reporting-configuration) for the server, or you can [retrieve the Server Audit Logs](/reference/system-endpoints/server-audit-logs). If backups were configured using an earlier version of ODK Central, you can also [get the current configuration](/reference/system-endpoints/backups-configuration) or terminate it.
-
-## Backups Configuration [/v1/config/backups]
-
-Earlier versions of ODK Central allowed users to configure backups to Google Drive, but it is no longer possible to do so. If backups were configured using an earlier version of ODK Central, you can get the current configuration or terminate it. On January 31, 2023, backups to Google Drive will stop working entirely, and we will remove the endpoints to get the current configuration or terminate it. Although backups to Google Drive will stop working, it will still be possible to download a [Direct Backup](/reference/system-endpoints/direct-backup). For more information about these changes, please see [this topic](https://forum.getodk.org/t/backups-to-google-drive-from-central-will-stop-working-after-jan-31st/38895) in the ODK Forum.
-
-The backups configuration is essentially a singleton object: there can be only one backups configuration at a time.
-
-### Getting the current configuration [GET]
-
-If configured, this endpoint will return the present backups configuration type. For security reasons, none of the actual internal authentication and encryption information is returned.
-
-If no backups are configured, this endpoint will return a `404`.
-
-+ Response 200 (application/json)
-    + Body
-
-            {
-                "type": "google",
-                "setAt": "2018-01-06T00:32:52.787Z"
-            }
-
-+ Response 403 (application/json)
-    + Attributes (Error 403)
-
-+ Response 404 (application/json)
-    + Attributes (Error 404)
-
-### Terminating the current configuration [DELETE]
-
-Deleting the backups configuration will permanently remove it, stopping it from running as scheduled.
-
-+ Response 200 (application/json)
-    + Attributes (Success)
-
-+ Response 403 (application/json)
-    + Attributes (Error 403)
-
-## Direct Backup [/v1/backup]
-
-_(introduced: version 1.1)_
-
-ODK Central offers HTTP endpoints that will immediately perform a backup on the system database and send that encrypted backup as the response. You can `POST` with an encryption passphrase. If backups to Google Drive were [configured](/reference/system-endpoints/backups-configuration) using an earlier version of ODK Central, you can also `GET` to use the passphrase you configured then.
-
-Note that performing the backup takes a great deal of time, during which the request will be held open. Both of these endpoints trickle junk data every five seconds while that processing is occurring to prevent the request from timing out. Depending on how much data you have, it can take many minutes for the data stream to speed up to a full transfer rate.
-
-### Using an Ad-Hoc Passphrase [POST]
-
-A `POST` verb will start an direct download ad-hoc backup. You will want to supply a `passphrase` with your chosen encryption passphrase. It is possible to omit this, in which case the backup will still be encrypted, but it will decrypt given an empty passphrase.
-
-Please see the section notes above about the long-running nature of this endpoint.
-
-+ Request (application/json)
-    + Attributes
-        + passphrase: `my-password` (string, optional) - The passphrase with which to encrypt the backup.
-
-+ Response 200
-    + Headers
-
-            Content-Disposition: attachment; filename=central-backup-2020-01-01T00:00:00.000Z.zip
-
-    + Body
-
-            (binary data)
-
-+ Response 403 (application/json)
-    + Attributes (Error 403)
-
-### Using the Configured Scheduled Backups Passphrase [GET]
-
-A `GET` verb will start an direct download backup, but using the configured scheduled backups passphrase. If no scheduled backup has been configured, the endpoint will return not found.
-
-Please see the section notes above about the long-running nature of this endpoint.
-
-+ Response 200
-    + Headers
-
-            Content-Disposition: attachment; filename=central-backup-2020-01-01T00:00:00.000Z.zip
-
-    + Body
-
-            (binary data)
-
-+ Response 403 (application/json)
-    + Attributes (Error 403)
+There are some resources available for getting or setting system information and configuration. You can set the [Usage Reporting configuration](/reference/system-endpoints/usage-reporting-configuration) for the server, retrieve the [Server Audit Logs](/reference/system-endpoints/server-audit-logs), or perform a [Direct Backup](/reference/system-endpoints/direct-backup).
 
 ## Usage Reporting Configuration [/v1/config/analytics]
 
@@ -3991,8 +3913,8 @@ Server Audit Logs entries are created for the following `action`s:
 * `entity.create` when an Entity is created.
 * `entity.create.error` when there is an error during entity creation process.
 * `config.set` when a system configuration is set.
-* `backup` when a backup operation is attempted.
 * `analytics` when a Usage Report is attempted.
+* Deprecated: `backup` when a backup operation is attempted for Google Drive backups.
 
 ### Getting Audit Log Entries [GET /v1/audits{?action,start,end,limit,offset}]
 
@@ -4025,6 +3947,36 @@ This endpoint supports retrieving extended metadata; provide a header `X-Extende
 + Response 403 (application/json)
     + Attributes (Error 403)
 
+## Direct Backup [/v1/backup]
+
+_(introduced: version 1.1)_
+
+ODK Central offers an HTTP endpoint that will immediately perform a backup on the system database and send that encrypted backup as the response. To use it, `POST` with an encryption passphrase.
+
+Note that performing the backup takes a great deal of time, during which the request will be held open. As a result, the endpoint will trickle junk data every five seconds while that processing is occurring to prevent the request from timing out. Depending on how much data you have, it can take many minutes for the data stream to speed up to a full transfer rate.
+
+### Using an Encryption Passphrase [POST]
+
+Use the `POST` verb to start a direct download ad-hoc backup. You will want to supply a `passphrase` with your chosen encryption passphrase. It is possible to omit this, in which case the backup will still be encrypted, but it will decrypt given an empty passphrase.
+
+Please see the section notes above about the long-running nature of this endpoint.
+
++ Request (application/json)
+    + Attributes
+        + passphrase: `my-password` (string, optional) - The passphrase with which to encrypt the backup.
+
++ Response 200
+    + Headers
+
+            Content-Disposition: attachment; filename=central-backup-2020-01-01T00:00:00.000Z.zip
+
+    + Body
+
+            (binary data)
+
++ Response 403 (application/json)
+    + Attributes (Error 403)
+
 # Group Encryption
 
 ODK Central supports two types of encryption:

lib/bin/backup.js

@@ -7,7 +7,7 @@
 // including this file, may be copied, modified, propagated, or distributed
 // except according to the terms contained in the LICENSE file.
 //
-// This script, given a path to a backup archive created by backup.js, will
+// This script, given a path to a backup archive returned by /v1/backup, will
 // attempt to wipe the configured database and restore it from the archive.
 
 const { run } = require('../task/task');

lib/bin/restore.js

@@ -28,10 +28,6 @@ const env = config.get('default.env');
 const { mailer } = require('../external/mail');
 const mail = mailer(mergeRight(config.get('default.email'), { env }));
 
-// get a google client.
-const googler = require('../external/google');
-const google = googler(config.get('default.external.google'));
-
 // get a sentry and configure errors.
 const Sentry = require('../external/sentry').init(config.get('default.external.sentry'));
 Error.stackTrackLimit = 20;
@@ -49,7 +45,7 @@ const enketo = require('../external/enketo').init(config.get('default.enketo'));
 
 // initialize our container, then generate an http service out of it.
 const container = require('../model/container')
-  .withDefaults({ db, mail, env, google, Sentry, bcrypt, xlsform, enketo });
+  .withDefaults({ db, mail, env, Sentry, bcrypt, xlsform, enketo });
 const service = require('../http/service')(container);
 
 // insert the graceful exit middleware.

lib/bin/run-server.js

@@ -45,9 +45,7 @@ const messages = {
   accountResetDeleted: message('ODK Central account password reset', '<html>Hello!<p>A password reset has been requested for this email address, but the account has been deleted.</p><p>If this message is unexpected, simply ignore it. Otherwise, please double check the email address given for your account, and try contacting your ODK system administrator.</p></html>'),
 
   // Notifies a user that their password has been changed
-  accountPasswordChanged: message('ODK Central account password change', '<html>Hello!<p>We are emailing because you have an ODK Central data collection account, and somebody has just changed its password.</p><p>If this was you, please feel free to ignore this email.</p><p>Otherwise, please contact your local ODK system administrator immediately.</p></html>'),
-
-  backupFailed: message('ODK Central backup failed', '<html>Hello:<p>This is an automated system message to the listed ODK Central system administrator. ODK Central just attempted to perform a backup, but was unable to. Please visit <a href="{{{domain}}}/#/system/backups">the Backups settings page</a> on the administration website for more information.</p></html>')
+  accountPasswordChanged: message('ODK Central account password change', '<html>Hello!<p>We are emailing because you have an ODK Central data collection account, and somebody has just changed its password.</p><p>If this was you, please feel free to ignore this email.</p><p>Otherwise, please contact your local ODK system administrator immediately.</p></html>')
 };
 
 module.exports = { messages };

lib/external/google.js

@@ -7,7 +7,7 @@
 // including this file, may be copied, modified, propagated, or distributed
 // except according to the terms contained in the LICENSE file.
 
-const { always, identity, pick } = require('ramda');
+const { identity } = require('ramda');
 const { Frame, table, readable, species } = require('../frame');
 
 /* eslint-disable no-multi-spaces */
@@ -60,8 +60,6 @@ defineConfig('analytics', identity, (value) => {
   }
   return result;
 });
-defineConfig('backups.main', pick(['type']));
-defineConfig('backups.google', always('New or refreshed Google API credentials'));
 
 
 module.exports = { Config };