|
1 | 1 | # Bulk Operations
|
2 | 2 |
|
| 3 | +## Export Local |
| 4 | +Exports data based on a given search operation to a local file in JSON or CSV format. |
| 5 | + |
| 6 | +* operation _(required)_ - must always be `export_local` |
| 7 | +* format _(required)_ - the format you wish to export the data, options are `json` & `csv` |
| 8 | +* path _(required)_ - path local to the server to export the data |
| 9 | +* search_operation _(required)_ - search_operation of `search_by_hash`, `search_by_value`, `search_by_conditions` or `sql` |
| 10 | +* filename _(optional)_ - the name of the file where your export will be written to (do not include extension in filename). If one is not provided it will be autogenerated based on the epoch. |
| 11 | + |
| 12 | +### Body |
| 13 | +```json |
| 14 | +{ |
| 15 | + "operation": "export_local", |
| 16 | + "format": "json", |
| 17 | + "path": "/data/", |
| 18 | + "search_operation": { |
| 19 | + "operation": "sql", |
| 20 | + "sql": "SELECT * FROM dev.breed" |
| 21 | + } |
| 22 | +} |
| 23 | +``` |
| 24 | + |
| 25 | +### Response: 200 |
| 26 | +```json |
| 27 | +{ |
| 28 | + "message": "Starting job with id 6fc18eaa-3504-4374-815c-44840a12e7e5" |
| 29 | +} |
| 30 | +``` |
| 31 | + |
| 32 | +--- |
| 33 | + |
3 | 34 | ## CSV Data Load
|
4 | 35 | Ingests CSV data, provided directly in the operation as an `insert`, `update` or `upsert` into the specified database table.
|
5 | 36 |
|
@@ -92,6 +123,43 @@ Ingests CSV data, provided via URL, as an `insert`, `update` or `upsert` into th
|
92 | 123 |
|
93 | 124 | ---
|
94 | 125 |
|
| 126 | +## Export To S3 |
| 127 | +Exports data based on a given search operation from table to AWS S3 in JSON or CSV format. |
| 128 | + |
| 129 | +* operation _(required)_ - must always be `export_to_s3` |
| 130 | +* format _(required)_ - the format you wish to export the data, options are `json` & `csv` |
| 131 | +* s3 _(required)_ - details your access keys, bucket, bucket region and key for saving the data to S3 |
| 132 | +* search_operation _(required)_ - search_operation of `search_by_hash`, `search_by_value`, `search_by_conditions` or `sql` |
| 133 | + |
| 134 | +### Body |
| 135 | +```json |
| 136 | +{ |
| 137 | + "operation": "export_to_s3", |
| 138 | + "format": "json", |
| 139 | + "s3": { |
| 140 | + "aws_access_key_id": "YOUR_KEY", |
| 141 | + "aws_secret_access_key": "YOUR_SECRET_KEY", |
| 142 | + "bucket": "BUCKET_NAME", |
| 143 | + "key": "OBJECT_NAME", |
| 144 | + "region": "BUCKET_REGION" |
| 145 | + }, |
| 146 | + "search_operation": { |
| 147 | + "operation": "sql", |
| 148 | + "sql": "SELECT * FROM dev.dog" |
| 149 | + } |
| 150 | +} |
| 151 | +``` |
| 152 | + |
| 153 | +### Response: 200 |
| 154 | +```json |
| 155 | +{ |
| 156 | + "message": "Starting job with id 9fa85968-4cb1-4008-976e-506c4b13fc4a", |
| 157 | + "job_id": "9fa85968-4cb1-4008-976e-506c4b13fc4a" |
| 158 | +} |
| 159 | +``` |
| 160 | + |
| 161 | +--- |
| 162 | + |
95 | 163 | ## Import from S3
|
96 | 164 | This operation allows users to import CSV or JSON files from an AWS S3 bucket as an `insert`, `update` or `upsert`.
|
97 | 165 |
|
@@ -129,4 +197,35 @@ This operation allows users to import CSV or JSON files from an AWS S3 bucket as
|
129 | 197 | "message": "Starting job with id 062a1892-6a0a-4282-9791-0f4c93b12e16",
|
130 | 198 | "job_id": "062a1892-6a0a-4282-9791-0f4c93b12e16"
|
131 | 199 | }
|
| 200 | +``` |
| 201 | + |
| 202 | +--- |
| 203 | + |
| 204 | +## Delete Records Before |
| 205 | + |
| 206 | +Delete data before the specified timestamp on the specified database table exclusively on the node where it is executed. Any clustered nodes with replicated data will retain that data. |
| 207 | + |
| 208 | +_Operation is restricted to super_user roles only_ |
| 209 | + |
| 210 | +* operation _(required)_ - must always be `delete_records_before` |
| 211 | +* date _(required)_ - records older than this date will be deleted. Supported format looks like: `YYYY-MM-DDThh:mm:ss.sZ` |
| 212 | +* schema _(required)_ - name of the schema where you are deleting your data |
| 213 | +* table _(required)_ - name of the table where you are deleting your data |
| 214 | + |
| 215 | +### Body |
| 216 | +```json |
| 217 | +{ |
| 218 | + "operation": "delete_records_before", |
| 219 | + "date": "2021-01-25T23:05:27.464", |
| 220 | + "schema": "dev", |
| 221 | + "table": "breed" |
| 222 | +} |
| 223 | +``` |
| 224 | + |
| 225 | +### Response: 200 |
| 226 | +```json |
| 227 | +{ |
| 228 | + "message": "Starting job with id d3aed926-e9fe-4ec1-aea7-0fb4451bd373", |
| 229 | + "job_id": "d3aed926-e9fe-4ec1-aea7-0fb4451bd373" |
| 230 | +} |
132 | 231 | ```
|
0 commit comments