Merge branch 'main' into feature/CCM-11164-rc-api-put

Author: alexnuttall

README.md

@@ -16,11 +16,11 @@
 - Then you need to install the following plugins:
 
 ```shell
-  asdf plugin-add nodejs
-  asdf plugin-add direnv
-  asdf plugin-add terraform
-  asdf plugin-add gitleaks
-  asdf plugin-add pre-commit
+  asdf plugin add nodejs
+  asdf plugin add direnv
+  asdf plugin add terraform
+  asdf plugin add gitleaks
+  asdf plugin add pre-commit
 ```
 
 - Now you can install the tools, and they will be runnable from within the `nhs-notify-web-template-management` directory:

data-migration/user-transfer/.gitignore

@@ -1,26 +1,80 @@
 # User Transfer Data Migration
 
-The purpose of this tool is to transfer templates stored in DynamoDB from one owner to another. It does not transfer ownership of any files in S3, this would need to be done separately.
+The purpose of this tool is to transfer templates stored in DynamoDB from a single user owner to the user's client.
 
-The owner field is the partition key for database entries and as such they are deleted and re-created with the new owner. They will retain their unique ID.
+This is a 2-stage process:
 
-A local backup of the data is persisted before any updates into a `./backup` directory.
+1. Plan (your migration)
+2. Apply (your migration)
 
-## Parameters
+## Plan (your migration)
 
-| Parameter          | Optional | Description                                     |
-| ------------------ | -------- | ----------------------------------------------- |
-| --sourceOwner      | Required | The current owner of the data, typically a UUID |
-| --destinationOwner | Required | The new owner of the data, typically a UUID     |
-| --environment      | Required | The environment name, e.g. main                 |
-|                    |          |                                                 |
+This creates a `transfer-plan-*.json` file in `./migrations` local directory and a copy in `main-acct-migration-backup/<environment>/transfer-plan-*/**` S3 bucket.
+
+```bash
+npm run plan -- \
+    --environment "main" \
+    --userPoolId "abc123" \
+    --iamAccessKeyId "abc1234" \
+    --iamSecretAccessKey "abc123" \
+    --iamSessionToken "abc123"
+```
+
+### Parameters
+
+| Parameter             | Optional | Description                                                                                    |
+| --------------------- | -------- | ---------------------------------------------------------------------------------------------- |
+| --environment         | Required | The environment name, e.g. main                                                                |
+| --userPoolId          | Required | This Cognito `UserPoolId` (if running in `sbx` then this can be your `sbx` Cognito userPoolId) |
+| --iamAccessKeyId      | Optional | Access key id of the IAM account (dev/prod)                                                    |
+| --iamSecretAccessKey  | Optional | Secret Access key of the IAM account (dev/prod)                                                |
+| --iamSessionToken     | Optional | Session token of the IAM account (dev/prod)                                                    |
+
+#### Why?
+
+The `transfer-plan-*.json` is used to keep a record of the data that will be migrated to Client ownership.
+
+## Apply (your migration)
+
+Run the migration process for data stored in `transfer-plan-*.json`.
+
+When doing a `dryRun=false` the data will be backed up:
+
+1. transfers all user related files in `internal` S3 bucket
+2. retrieves and stores all related DynamoDB data before executing
+
+Backed-up data is stored `main-acct-migration-backup/<environment>/transfer-plan-*/**`
+
+```bash
+npm run apply -- \
+    --environment "main" \
+    --file "./migrations/file.json" \
+    --dryRun "true"
+```
+
+The result of this script will output a file named the same as the input file but with `run/dryrun` appended to the name. This file is a record of what happened to each migration, whether it failed or passed and which stage the migration ended at.
+
+### Parameters
+
+| Parameter             | Optional | Description                                                                                    |
+| --------------------- | -------- | ---------------------------------------------------------------------------------------------- |
+| --environment         | Required | The environment name, e.g. main                                                                |
+| --file                | Required | The path of the `transfer-plan-*.json` file                                                    |
+| --dryRun              | Optional | Defaults to `true`. When true will _not_ execute migration                                     |
 
 ## Authentication
 
-You should establish a local AWS authentication session to the target AWS account with sufficient permissions to read, write and delete template data from DynamoDB.
+You should establish a local AWS authentication session to the Templates (dev/prod) AWS account with sufficient permissions to read, write and delete template data from DynamoDB.
 
-## Example
+## Suggestions
 
-```shell
-npm run transfer -- --sourceOwner 26d202d4-f001-7043-1781-77c935224d18 --destinationOwner e6f232b4-6051-7096-a238-5527b8615d11 --environment main
+When executing the migration it's a good idea to also write out the logs to a file. For example
+
+```bash
+npm run plan -- \
+    --environment "main" \
+    --file "./migrations/file.json" \
+    --dryRun "false" >> migration.logs.txt
 ```
+
+Then upload this log file to S3. This should then keep a decent record of _what_ happened during a migration.

data-migration/user-transfer/README.md

@@ -33,7 +33,7 @@ const config: Config = {
 
   collectCoverageFrom: ['src/**/*.ts*'],
 
-  coveragePathIgnorePatterns: ['handler.ts', 'constants.ts'],
+  coveragePathIgnorePatterns: ['migrate-cli.ts', 'plan-cli.ts', 'types.ts'],
 
   // Set the absolute path for imports
   moduleNameMapper: pathsToModuleNameMapper(compilerOptions.paths, {

data-migration/user-transfer/jest.config.ts

@@ -0,0 +1 @@
+*.json

data-migration/user-transfer/migrations/.gitignore

@@ -1,8 +1,10 @@
 {
   "dependencies": {
+    "@aws-sdk/client-cognito-identity-provider": "^3.864.0",
     "@aws-sdk/client-dynamodb": "^3.864.0",
     "@aws-sdk/client-s3": "^3.864.0",
     "@aws-sdk/client-sts": "^3.864.0",
+    "@aws-sdk/credential-providers": "^3.864.0",
     "yargs": "^17.7.2"
   },
   "description": "Transfers template ownership from one user to another",
@@ -20,8 +22,9 @@
   "scripts": {
     "lint": "eslint ./src",
     "lint:fix": "eslint ./src --fix",
+    "apply": "tsx src/migrate-cli.ts",
+    "plan": "tsx src/plan-cli.ts",
     "test:unit": "jest",
-    "transfer": "tsx src/handler.ts",
     "typecheck": "tsc --noEmit"
   }
 }