Merge pull request #173 from HarperDB/operations-utilities-refactor

Breaking operations-api/utilities out to separate files

Author: kriszyp

.gitbook.yaml

@@ -1,6 +1,7 @@
 root: ./docs/
 
 redirects:
+  developers/operations-api/utilities: developers/operations-api/system-operations.md
   install-harperdb: deployments/install-harper/README.md
   install-harperdb/linux: deployments/install-harper/linux.md
   install-harperdb/other: deployments/install-harper/README.md

docs/SUMMARY.md

@@ -31,7 +31,9 @@
   * [Registration](developers/operations-api/registration.md)
   * [Jobs](developers/operations-api/jobs.md)
   * [Logs](developers/operations-api/logs.md)
-  * [Utilities](developers/operations-api/utilities.md)
+  * [System Operations](developers/operations-api/system-operations.md)
+  * [Configuration](developers/operations-api/configuration.md)
+  * [Certificate Management](developers/operations-api/certificate-management.md)
   * [Token Authentication](developers/operations-api/token-authentication.md)
   * [SQL Operations](developers/operations-api/sql-operations.md)
   * [Advanced JSON SQL Examples](developers/operations-api/advanced-json-sql-examples.md)

docs/developers/operations-api/README.md

@@ -26,7 +26,9 @@ The operations API reference is available below and categorized by topic:
 * [Registration](registration.md)
 * [Jobs](jobs.md)
 * [Logs](logs.md)
-* [Utilities](utilities.md)
+* [System Operations](system-operations.md)
+* [Configuration](configuration.md)
+* [Certificate Management](certificate-management.md)
 * [Token Authentication](token-authentication.md)
 * [SQL Operations](sql-operations.md)
 * [Advanced JSON SQL Examples](advanced-json-sql-examples.md)

docs/developers/operations-api/bulk-operations.md

@@ -1,5 +1,36 @@
 # Bulk Operations
 
+## Export Local
+Exports data based on a given search operation to a local file in JSON or CSV format.
+
+* operation _(required)_ - must always be `export_local`
+* format _(required)_ - the format you wish to export the data, options are `json` & `csv`
+* path _(required)_ - path local to the server to export the data
+* search_operation _(required)_ - search_operation of `search_by_hash`, `search_by_value`, `search_by_conditions` or `sql`
+* filename _(optional)_ - the name of the file where your export will be written to (do not include extension in filename). If one is not provided it will be autogenerated based on the epoch. 
+
+### Body
+```json
+{
+  "operation": "export_local",
+  "format": "json",
+  "path": "/data/",
+  "search_operation": {
+      "operation": "sql",
+      "sql": "SELECT * FROM dev.breed"
+  }
+}
+```
+
+### Response: 200
+```json
+{
+  "message": "Starting job with id 6fc18eaa-3504-4374-815c-44840a12e7e5"
+}
+```
+
+---
+
 ## CSV Data Load
 Ingests CSV data, provided directly in the operation as an `insert`, `update` or `upsert` into the specified database table.
 
@@ -92,6 +123,43 @@ Ingests CSV data, provided via URL, as an `insert`, `update` or `upsert` into th
 
 ---
 
+## Export To S3
+Exports data based on a given search operation from table to AWS S3 in JSON or CSV format.
+
+* operation _(required)_ - must always be `export_to_s3`
+* format _(required)_ - the format you wish to export the data, options are `json` & `csv`
+* s3 _(required)_ - details your access keys, bucket, bucket region and key for saving the data to S3
+* search_operation _(required)_ - search_operation of `search_by_hash`, `search_by_value`, `search_by_conditions` or `sql`
+
+### Body
+```json
+{
+    "operation": "export_to_s3",
+    "format": "json",
+    "s3": {
+        "aws_access_key_id": "YOUR_KEY",
+        "aws_secret_access_key": "YOUR_SECRET_KEY",
+        "bucket": "BUCKET_NAME",
+        "key": "OBJECT_NAME",
+        "region": "BUCKET_REGION"
+    },
+    "search_operation": {
+        "operation": "sql",
+        "sql": "SELECT * FROM dev.dog"
+    }
+}
+```
+
+### Response: 200
+```json
+{
+  "message": "Starting job with id 9fa85968-4cb1-4008-976e-506c4b13fc4a",
+  "job_id": "9fa85968-4cb1-4008-976e-506c4b13fc4a"
+}
+```
+
+---
+
 ## Import from S3
 This operation allows users to import CSV or JSON files from an AWS S3 bucket as an `insert`, `update` or `upsert`.
 
@@ -129,4 +197,35 @@ This operation allows users to import CSV or JSON files from an AWS S3 bucket as
   "message": "Starting job with id 062a1892-6a0a-4282-9791-0f4c93b12e16",
   "job_id": "062a1892-6a0a-4282-9791-0f4c93b12e16"
 }
+```
+
+---
+
+## Delete Records Before
+
+Delete data before the specified timestamp on the specified database table exclusively on the node where it is executed. Any clustered nodes with replicated data will retain that data.
+
+_Operation is restricted to super_user roles only_
+
+* operation _(required)_ - must always be `delete_records_before`
+* date _(required)_ - records older than this date will be deleted. Supported format looks like: `YYYY-MM-DDThh:mm:ss.sZ`
+* schema _(required)_ - name of the schema where you are deleting your data
+* table _(required)_ - name of the table where you are deleting your data
+
+### Body
+```json
+{
+  "operation": "delete_records_before",
+  "date": "2021-01-25T23:05:27.464",
+  "schema": "dev",
+  "table": "breed"
+}
+```
+
+### Response: 200
+```json
+{
+  "message": "Starting job with id d3aed926-e9fe-4ec1-aea7-0fb4451bd373",
+  "job_id": "d3aed926-e9fe-4ec1-aea7-0fb4451bd373"
+}
 ```

docs/developers/operations-api/certificate-management.md

@@ -0,0 +1,120 @@
+# Certificate Management
+
+## Add Certificate
+
+Adds or updates a certificate in the `hdb_certificate` system table. 
+If a `private_key` is provided it will __not__ be stored in `hdb_certificate`, it will be written to file in `<ROOTPATH>/keys/`.
+If a `private_key` is not passed the operation will search for one that matches the certificate. If one is not found an error will be returned.
+
+_Operation is restricted to super_user roles only_
+
+* operation _(required)_ - must always be `add_certificate`
+* name _(required)_ - a unique name for the certificate
+* certificate _(required)_ - a PEM formatted certificate string
+* is_authority _(required)_ - a boolean indicating if the certificate is a certificate authority
+* hosts _(optional)_ - an array of hostnames that the certificate is valid for
+* private_key _(optional)_ - a PEM formatted private key string
+
+### Body
+```json
+{
+  "operation": "add_certificate",
+  "name": "my-cert",
+  "certificate": "-----BEGIN CERTIFICATE-----ZDFAay... -----END CERTIFICATE-----",
+  "is_authority": false,
+  "private_key": "-----BEGIN RSA PRIVATE KEY-----Y4dMpw5f... -----END RSA PRIVATE KEY-----"
+}
+```
+
+### Response: 200
+```json
+{
+  "message": "Successfully added certificate: my-cert"
+}
+```
+
+---
+
+## Remove Certificate
+
+Removes a certificate from the `hdb_certificate` system table and deletes the corresponding private key file.
+
+_Operation is restricted to super_user roles only_
+
+* operation _(required)_ - must always be `remove_certificate`
+* name _(required)_ - the name of the certificate
+
+### Body
+```json
+{
+  "operation": "remove_certificate",
+  "name": "my-cert"
+}
+```
+
+### Response: 200
+```json
+{
+  "message": "Successfully removed my-cert"
+}
+```
+
+---
+
+## List Certificates
+
+Lists all certificates in the `hdb_certificate` system table.
+
+_Operation is restricted to super_user roles only_
+
+* operation _(required)_ - must always be `list_certificates`
+
+### Body
+```json
+{
+  "operation": "list_certificates"
+}
+```
+
+### Response: 200
+```json
+[
+  {
+    "name": "HarperDB-Certificate-Authority-node1",
+    "certificate": "-----BEGIN CERTIFICATE-----\r\nTANBgkqhk... S34==\r\n-----END CERTIFICATE-----\r\n",
+    "private_key_name": "privateKey.pem",
+    "is_authority": true,
+    "details": {
+      "issuer": "CN=HarperDB-Certificate-Authority-node1 C=USA ST=Colorado L=Denver O=HarperDB\\, Inc.",
+      "subject": "CN=HarperDB-Certificate-Authority-node1 C=USA ST=Colorado L=Denver O=HarperDB\\, Inc.",
+      "serial_number": "5235345",
+      "valid_from": "Aug 27 15:00:00 2024 GMT",
+      "valid_to": "Aug 25 15:00:00 2034 GMT"
+    },
+    "is_self_signed": true,
+    "uses": [
+      "https",
+      "wss"
+    ]
+  },
+  {
+    "name": "node1",
+    "certificate": "-----BEGIN CERTIFICATE-----\r\ngIEcSR1M... 5bv==\r\n-----END CERTIFICATE-----\r\n",
+    "private_key_name": "privateKey.pem",
+    "is_authority": false,
+    "details": {
+      "issuer": "CN=HarperDB-Certificate-Authority-node1 C=USA ST=Colorado L=Denver O=HarperDB\\, Inc.",
+      "subject": "CN=node.1 C=USA ST=Colorado L=Denver O=HarperDB\\, Inc.",
+      "subject_alt_name": "IP Address:127.0.0.1, DNS:localhost, IP Address:0:0:0:0:0:0:0:1, DNS:node.1",
+      "serial_number": "5243646",
+      "valid_from": "Aug 27 15:00:00 2024 GMT",
+      "valid_to": "Aug 25 15:00:00 2034 GMT"
+    },
+    "is_self_signed": true,
+    "uses": [
+      "https",
+      "wss"
+    ]
+  }
+]
+```

docs/developers/operations-api/components.md

@@ -317,6 +317,8 @@ _Operation is restricted to super\_user roles only_
 }
 ```
 
+***
+
 ## Add SSH Key
 
 Adds an SSH key for deploying components from private repositories. This will also create an ssh config file that will be used when deploying the components.
@@ -379,6 +381,8 @@ Host harperdb-private-component.github.com
 
 Note that `deploy_component` with a package uses `npm install` so the url must be a valid npm format url. The above is an example of a url using a tag in the repo to install.
 
+***
+
 ## Update SSH Key
 
 Updates the private key contents of an existing SSH key.
@@ -436,6 +440,8 @@ _Operation is restricted to super\_user roles only_
 }
 ```
 
+***
+
 ## List SSH Keys
 
 List off the names of added SSH keys
@@ -463,6 +469,8 @@ _Operation is restricted to super\_user roles only_
 ]
 ```
 
+***
+
 ## Set SSH Known Hosts
 
 Sets the SSH known\_hosts file. This will overwrite the file.
@@ -513,3 +521,27 @@ _Operation is restricted to super\_user roles only_
     "known_hosts": "github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=\ngithub.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl\ngithub.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=\n"
 }
 ```
+
+***
+
+## Install Node Modules
+This operation is deprecated, as it is handled automatically by deploy_component and restart.
+Executes npm install against specified custom function projects.
+
+_Operation is restricted to super_user roles only_
+
+* operation _(required)_ - must always be `install_node_modules`
+* projects _(required)_ - must ba an array of custom functions projects.
+* dry_run _(optional)_ - refers to the npm --dry-run flag: [https://docs.npmjs.com/cli/v8/commands/npm-install#dry-run](https://docs.npmjs.com/cli/v8/commands/npm-install#dry-run). Defaults to false.
+
+### Body
+```json
+{
+  "operation": "install_node_modules",
+  "projects": [
+    "dogs",
+    "cats"
+  ],
+  "dry_run": true
+}
+```