Merge pull request #95 from localgovdrupal/feature/long-term-storage

Feature: Long term storage of Webform submission data

Author: Adnan-cds

README.md

@@ -7,6 +7,9 @@ Provides additional configuration, styling and components for the Drupal Webform
 * LocalGov Forms Date - A date input field based on the [GDS Date Input pattern](https://design-system.service.gov.uk/components/date-input/)
 * LocalGov address lookup - Webform element with a configurable address lookup backend.  Geocoder plugins act as backends.
 
+## Plugins
+- Personally Identifiable Information (PII) redactor from Webform submissions: At the moment, a plugin manager `plugin.manager.pii_redactor` and a sample plugin are provided.
+
 ## Dependencies
 The geocoder-php/nominatim-provider package is necessary to run automated tests:
 ```
@@ -23,4 +26,3 @@ To avoid the configuration being removed by deployments, install the [Config ign
 webform.webform.*
 webform.webform_options.*
 ```
-

localgov_forms.services.yml

@@ -6,3 +6,8 @@ services:
   localgov_forms.address_lookup:
     class: Drupal\localgov_forms\AddressLookup
     arguments: ['@geocoder', '@localgov_forms.geocoder_selection']
+
+  # Plugin manager service for PII redaction from Webform submissions.
+  plugin.manager.pii_redactor:
+    class: Drupal\localgov_forms\Plugin\PIIRedactorPluginManager
+    parent: default_plugin_manager

modules/localgov_forms_lts/README.md

@@ -0,0 +1,108 @@
+## Long term storage for Webform submissions
+This module copies Webform submissions to a separate database for Long Term Storage (LTS).  The LTS database can then be used for data warehousing needs such as reporting and analysis.  Optionally, Personally Identifiable Information (PII) can be redacted from Webform submissions while they are copied to the LTS database.
+
+### Setup process
+- Create a database which will serve as the LTS database.
+- Declare it in Drupal's settings.php using the **localgov_forms_lts** key.  Example:
+  ```
+  $databases['localgov_forms_lts']['default'] = [
+    'database'  => 'our-long-term-storage-database',
+    'username'  => 'database-username-goes-here',
+    'password'  => 'database-password-goes-here',
+    'host'      => 'database-hostname-goes-here',
+    'port'      => '3306',
+    'driver'    => 'mysql',
+    'prefix'    => '',
+  ];
+  ```
+- Install the localgov_forms_lts submodule.
+- Check the module requirement report from Drupal's status page at `admin/reports/status`.  This should be under the **LocalGov Forms LTS** key.
+- [Optional] If all looks good in the previous step, run `drush localgov-forms-lts:copy --force` which will copy existing Webform submissions into the LTS database.
+- By default, periodic Webform submissions copying to the LTS database is disabled.  Activate it from `/admin/structure/webform/config/submissions-lts`.
+- Ensure cron is running periodically.  This will copy any new Webform submissions or changes to existing Webform submissions since the last cron run or the last `drush localgov-forms-lts:copy` run.
+- [Optional] Tell individual Webforms to purge submissions older than a chosen period.  This is configured for each Webform from its `Settings > Submissions > Submission purge settings` configuration section.
+
+### Inspection
+To inspect Webform submissions kept in Long term storage, look for the **LTS** tab in the Webform submissions listing page.  This is usually at `/admin/structure/webform/submissions/manage`.
+
+### Good to know
+- Each cron run copies 50 Webform submissions.  If your site is getting more than that many Webform submissions between subsequent cron runs, not all Webform submissions will get copied to LTS during a certain period.  If that happens, adjust cron run frequency.
+- Files attached to Webform submissions are *not* moved to LTS.
+- You can choose to redact elements with Personally Identifiable Information (PII) while they are copied to the LTS database.  For that, select *Best effort PII redactor* (or another redactor) from the `PII redactor plugin` dropdown in the LTS config page at `/admin/structure/webform/config/submissions-lts`.  At the moment, this plugin redacts all name, email, telephone, number, and various address type elements.  Additionally, any text or radio or checkbox element whose machine name (AKA Key) contains the following also gets redacted: name, mail, phone, contact_number, date_of_birth, dob_, personal_, title, nino, passport, postcode, address, serial_number, reg_number, pcn_, and driver_.
+- If you are using this module in multiple instances of the same site (e.g. dev/stage/live), ensure that the database settings array points to *different* databases.  Alternatively, disable copying for the non-live environments from `/admin/structure/webform/config/submissions-lts`.  The relevant settings `localgov_forms_lts.settings:is_copying_enabled` can be [overridden](https://www.drupal.org/docs/drupal-apis/configuration-api/configuration-override-system#s-global-overrides) from settings.php.  The [config_split](https://www.drupal.org/project/config_split) module can be handy as well.
+- This module is currently in experimental stage.
+
+### Todo
+- Removal of Webform submissions from LTS after a predefined period e.g. 5 years.
+
+### Testing in DDEV
+
+To set up testing in ddev, we'll need to set up a second database.
+
+There are a few ways to do this, but the following seems to work.
+
+#### 1. Add a post-start hook to your .ddev/config.yml
+
+Edit `.ddev/config.yml` and add the following to create a new database on start.
+
+```
+hooks:
+  post-start:
+  - exec: mysql -uroot -proot -e "CREATE DATABASE IF NOT EXISTS localgov_forms_lts; GRANT ALL ON localgov_forms_lts.* to 'db'@'%';"
+    service: db
+```
+
+#### 2. Add the database connection string to settings.php:
+
+Edit sites/default/settings.php and add a new database connection string at the
+end of the file.
+
+```
+// Database connection for localgov_forms_lts.
+$databases['localgov_forms_lts']['default'] = [
+  'database'  => 'localgov_forms_lts',
+  'username'  => 'db',
+  'password'  => 'db',
+  'host'      => 'db',
+  'port'      => '3306',
+  'driver'    => 'mysql',
+  'prefix'    => '',
+];
+```
+
+#### 3. Install Adminer
+
+Adminer is useful if you want to inspect databases and tables.
+
+```
+ddev get ddev/ddev-adminer
+```
+
+#### 4. Restart ddev
+
+```
+ddev restart
+```
+
+#### 5. Require and install the module.
+
+```
+ddev composer require localgovdrupal/localgov_forms
+ddev drush si localgov_forms_lts -y
+```
+
+#### 5. Make some submissions.
+
+For example, in LocalGov Drupal we tend to have a contact form at /form/contact.
+
+Make a couple of submissions there.
+
+#### 6. Run cron.
+
+ddev drush cron
+
+#### 7. Inspect the LTS tab
+
+Go to /admin/structure/webform/submissions/lts
+
+Here you should see your submissions with redacted name and email address.

modules/localgov_forms_lts/config/install/localgov_forms_lts.settings.yml

@@ -0,0 +1,2 @@
+is_copying_enabled: false
+pii_redactor_plugin_id: ''

modules/localgov_forms_lts/config/schema/localgov_forms_lts.schema.yml

@@ -0,0 +1,12 @@
+# Schema for the configuration files of the localgov_forms_lts submodule.
+
+localgov_forms_lts.settings:
+  type: config_object
+  label: 'Webform submissions LTS config'
+  mapping:
+    is_copying_enabled:
+      type: boolean
+      label: 'Is copying to LTS database enabled?'
+    pii_redactor_plugin_id:
+      type: machine_name
+      label: 'PII redactor plugin id'

modules/localgov_forms_lts/localgov_forms_lts.info.yml

@@ -0,0 +1,10 @@
+name: LocalGov Forms long term storage
+type: module
+description: Long term storage for Webform submissions.
+core_version_requirement: ^10 || ^11
+php: 8.0
+package: LocalGov Drupal
+lifecycle: experimental
+
+dependencies:
+- webform:webform

modules/localgov_forms_lts/localgov_forms_lts.install

@@ -0,0 +1,106 @@
+<?php
+
+/**
+ * @file
+ * Install and update hook implementations.
+ */
+
+use Drupal\Core\Database\Database;
+use Drupal\Core\Entity\ContentEntityTypeInterface;
+use Drupal\localgov_forms_lts\Constants;
+use Drupal\webform\WebformSubmissionStorageSchema;
+
+/**
+ * Implements hook_install().
+ *
+ * Creates all entity storage tables associated with the webform_submission
+ * entity type.
+ */
+function localgov_forms_lts_install($is_syncing): void {
+
+  if (!localgov_forms_lts_has_db()) {
+    return;
+  }
+
+  $webform_submission_schema = _localgov_forms_lts_get_webform_submission_storage_schema();
+
+  foreach ($webform_submission_schema as $table_name => $table_schema) {
+    _localgov_forms_lts_copy_table($table_name, $table_schema, db: Constants::LTS_DB_KEY);
+  }
+}
+
+/**
+ * Implements hook_requirements().
+ *
+ * Checks for the presence of the localgov_forms_lts database.
+ */
+function localgov_forms_lts_requirements($phase) {
+
+  $requirements = [
+    'localgov_forms_lts' => [
+      'title'       => t('LocalGov Forms LTS'),
+      'value'       => t('Available'),
+      'description' => t('LocalGov Forms LTS database available.'),
+      'severity'    => REQUIREMENT_OK,
+    ],
+  ];
+
+  if (!function_exists('localgov_forms_lts_has_db')) {
+    // Some necessary files have not been loaded yet during the "install" phase.
+    require_once __DIR__ . '/localgov_forms_lts.module';
+    require_once __DIR__ . '/src/Constants.php';
+  }
+
+  if (!localgov_forms_lts_has_db()) {
+    $requirements['localgov_forms_lts']['value'] = t('Unavailable');
+    $requirements['localgov_forms_lts']['description'] = t('The LocalGov Forms LTS database must exist for this module to function.');
+    $requirements['localgov_forms_lts']['severity'] = REQUIREMENT_ERROR;
+  }
+
+  return $requirements;
+}
+
+/**
+ * Extracts entity storage schema.
+ *
+ * Returns the entity storage schema for the webform_submission content entity.
+ */
+function _localgov_forms_lts_get_webform_submission_storage_schema(): array {
+
+  $entity_type_manager  = Drupal::service('entity_type.manager');
+  $entity_storage       = $entity_type_manager->getStorage('webform_submission');
+  $entity_type          = $entity_storage->getEntityType();
+  $entity_field_manager = Drupal::service('entity_field.manager');
+  $db_service           = Drupal::service('database');
+
+  // @phpstan-ignore-next-line Avoid incorrect type inference of $entity_type.
+  $entity_schema = (new class($entity_type_manager, $entity_type, $entity_storage, $db_service, $entity_field_manager) extends WebformSubmissionStorageSchema {
+
+    /**
+     * Public wrapper over protected method.
+     */
+    public function getEntitySchemaWrapper(ContentEntityTypeInterface $entity_type) {
+
+      return parent::getEntitySchema($entity_type);
+    }
+
+  })->getEntitySchemaWrapper($entity_type);
+
+  return $entity_schema;
+}
+
+/**
+ * Creates database tables.
+ */
+function _localgov_forms_lts_copy_table(string $table_name, array $table_schema, string $db): void {
+
+  $db_connection = Database::getConnection(key: $db);
+  $tx = $db_connection->startTransaction();
+
+  try {
+    $db_connection->schema()->createTable($table_name, $table_schema);
+  }
+  catch (Exception $e) {
+    $tx->rollBack();
+  }
+}

modules/localgov_forms_lts/localgov_forms_lts.links.task.yml

@@ -0,0 +1,27 @@
+# Submissions > LTS
+entity.webform_submission.lts_collection:
+  title: 'LTS'
+  route_name: entity.webform_submission.lts_collection
+  parent_id: entity.webform_submission.collection
+  weight: 21
+
+# View
+entity.webform_submission.lts_view:
+  title: 'View'
+  route_name: entity.webform_submission.lts_view
+  base_route: entity.webform_submission.lts_view
+  weight: 1
+
+# Notes
+entity.webform_submission.lts_notes:
+  title: 'Notes'
+  route_name: entity.webform_submission.lts_notes
+  base_route: entity.webform_submission.lts_view
+  weight: 3
+
+# Config tab
+localgov_forms_lts.lts_config:
+  title: 'LTS'
+  route_name: localgov_forms_lts.lts_config
+  parent_id: webform.config
+  weight: 43

modules/localgov_forms_lts/localgov_forms_lts.module

@@ -0,0 +1,90 @@
+<?php
+
+/**
+ * @file
+ * Hook implementations.
+ */
+
+declare(strict_types=1);
+
+use Drupal\Component\Render\MarkupInterface;
+use Drupal\Core\Database\Database;
+use Drupal\localgov_forms_lts\Constants;
+use Drupal\localgov_forms_lts\LtsCopy;
+
+/**
+ * Tests presence of LTS database.
+ */
+function localgov_forms_lts_has_db(): bool {
+
+  try {
+    Database::getConnection(key: Constants::LTS_DB_KEY);
+  }
+  catch (Exception $e) {
+    return FALSE;
+  }
+
+  return TRUE;
+}
+
+/**
+ * Implements hook_cron().
+ */
+function localgov_forms_lts_cron() {
+
+  localgov_forms_lts_copy_recently_added_n_updated_subs();
+}
+
+/**
+ * Copies Webform submissions.
+ *
+ * Copies Webform submissions added or updated since last run to the lts
+ * storage.  At most 50 Webform submissions are copied.
+ */
+function localgov_forms_lts_copy_recently_added_n_updated_subs() :void {
+
+  $service_container = Drupal::service('service_container');
+  $lts_config        = $service_container->get('config.factory')->get(Constants::LTS_CONFIG_ID);
+
+  $is_copying_enabled = $lts_config->get(Constants::LTS_CONFIG_COPY_STATE);
+  if (!$is_copying_enabled) {
+    return;
+  }
+
+  $pii_redactor_plugin_id = $lts_config->get(Constants::LTS_CONFIG_PII_REDACTOR_PLUGIN_ID);
+  $pii_redactor_plugin    = ($pii_redactor_plugin_id && $service_container->has(Constants::PII_REDACTOR_PLUGIN_MANAGER)) ? $service_container->get(Constants::PII_REDACTOR_PLUGIN_MANAGER)->createInstance($pii_redactor_plugin_id) : NULL;
+
+  $lts_copy_obj = LtsCopy::create(Drupal::getContainer(), $pii_redactor_plugin);
+  $copy_results = $lts_copy_obj->copy();
+
+  $feedback_msg = _localgov_forms_lts_prepare_feedback_msg($copy_results);
+  Drupal::service('logger.factory')
+    ->get(Constants::LTS_LOGGER_CHANNEL_ID)
+    ->info($feedback_msg);
+}
+
+/**
+ * Prepares feedback message for copying.
+ *
+ * The feedback message is prepared from the outcome of Webform submission copy
+ * operations.
+ */
+function _localgov_forms_lts_prepare_feedback_msg(array $copy_results): MarkupInterface {
+
+  $copy_successes = array_filter($copy_results);
+  $copy_failures  = array_diff_key($copy_results, $copy_successes);
+
+  $successfully_copied_sid_list   = array_keys($copy_successes);
+  $unsuccessfully_copied_sid_list = array_keys($copy_failures);
+
+  $successfully_copied_sid_list_msg   = $successfully_copied_sid_list ? implode(', ', $successfully_copied_sid_list) : 'None';
+  $unsuccessfully_copied_sid_list_msg = $unsuccessfully_copied_sid_list ? implode(', ', $unsuccessfully_copied_sid_list) : 'None';
+
+  $feedback_msg = t('Successfully copied Webform submission ids: %successes. :newline Failed copies: %failures.', [
+    '%successes' => $successfully_copied_sid_list_msg,
+    '%failures'  => $unsuccessfully_copied_sid_list_msg,
+    ':newline'   => PHP_EOL,
+  ]);
+
+  return $feedback_msg;
+}