Skip to content

Commit 3faea8c

Browse files
committed
Merge remote-tracking branch 'origin/release_4.4'
2 parents 8cdb1f0 + 6a0ab39 commit 3faea8c

29 files changed

+230
-30
lines changed

docs/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ Wondering what's new with HarperDB 4.4? Take a look at our latest [Release Notes
1313

1414
## Getting Started
1515

16-
<table data-column-title-hidden data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><a href="getting-started.md"><strong>Getting Started Guide</strong></a></td><td>Get up and running with HarperDB</td><td></td><td><a href="getting-started.md">getting-started.md</a></td></tr><tr><td><a href="deployments/install-harperdb/"><strong>Quick Install HarperDB</strong></a></td><td>Run HarperDB on your on hardware</td><td></td><td><a href="deployments/install-harperdb/">install-harperdb</a></td></tr><tr><td><a href="deployments/harperdb-cloud/"><strong>Try HarperDB Cloud</strong></a></td><td>Spin up an instance in minutes to going fast</td><td></td><td><a href="deployments/harperdb-cloud/">harperdb-cloud</a></td></tr></tbody></table>
16+
<table data-column-title-hidden data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><a href="getting-started.md"><strong>Getting Started Guide</strong></a></td><td>Get up and running with HarperDB</td><td></td><td><a href="getting-started.md">getting-started.md</a></td></tr><tr><td><a href="deployments/install-harperdb/"><strong>Quick Install HarperDB</strong></a></td><td>Run HarperDB on your on hardware</td><td></td><td><a href="deployments/install-harperdb/">install-harperdb</a></td></tr><tr><td><a href="deployments/harperdb-cloud/"><strong>Try HarperDB Cloud</strong></a></td><td>Spin up an instance in minutes to get going fast</td><td></td><td><a href="deployments/harperdb-cloud/">harperdb-cloud</a></td></tr></tbody></table>
1717

1818
## Building with HarperDB
1919

docs/SUMMARY.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,12 +122,27 @@
122122
* [Storage Algorithm](technical-details/reference/storage-algorithm.md)
123123
* [Release Notes](technical-details/release-notes/README.md)
124124
* [HarperDB Tucker (Version 4)](technical-details/release-notes/4.tucker/README.md)
125+
* [4.4.18](technical-details/release-notes/4.tucker/4.4.18.md)
126+
* [4.4.17](technical-details/release-notes/4.tucker/4.4.17.md)
127+
* [4.4.16](technical-details/release-notes/4.tucker/4.4.16.md)
128+
* [4.4.15](technical-details/release-notes/4.tucker/4.4.15.md)
129+
* [4.4.14](technical-details/release-notes/4.tucker/4.4.14.md)
130+
* [4.4.13](technical-details/release-notes/4.tucker/4.4.13.md)
131+
* [4.4.12](technical-details/release-notes/4.tucker/4.4.12.md)
132+
* [4.4.11](technical-details/release-notes/4.tucker/4.4.11.md)
133+
* [4.4.10](technical-details/release-notes/4.tucker/4.4.10.md)
134+
* [4.4.9](technical-details/release-notes/4.tucker/4.4.9.md)
135+
* [4.4.8](technical-details/release-notes/4.tucker/4.4.8.md)
136+
* [4.4.7](technical-details/release-notes/4.tucker/4.4.7.md)
137+
* [4.4.6](technical-details/release-notes/4.tucker/4.4.6.md)
125138
* [4.4.5](technical-details/release-notes/4.tucker/4.4.5.md)
126139
* [4.4.4](technical-details/release-notes/4.tucker/4.4.4.md)
127140
* [4.4.3](technical-details/release-notes/4.tucker/4.4.3.md)
128141
* [4.4.2](technical-details/release-notes/4.tucker/4.4.2.md)
129142
* [4.4.1](technical-details/release-notes/4.tucker/4.4.1.md)
130143
* [4.4.0](technical-details/release-notes/4.tucker/4.4.0.md)
144+
* [4.3.38](technical-details/release-notes/4.tucker/4.3.38.md)
145+
* [4.3.37](technical-details/release-notes/4.tucker/4.3.37.md)
131146
* [4.3.36](technical-details/release-notes/4.tucker/4.3.36.md)
132147
* [4.3.35](technical-details/release-notes/4.tucker/4.3.35.md)
133148
* [4.3.34](technical-details/release-notes/4.tucker/4.3.34.md)

docs/deployments/configuration.md

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -206,15 +206,21 @@ replication:
206206

207207
`routes` - _Type_: array;
208208

209-
An array of routes to connect to other nodes. Each element in the array can be either a string or an object with `hostname` and `port` properties.
209+
An array of routes to connect to other nodes. Each element in the array can be either a string or an object with `hostname`, `port` and optionally `startTime` properties.
210+
211+
`startTime` - _Type_: string; ISO formatted UTC date string.
212+
213+
Replication will attempt to catch up on all remote data upon setup. To start replication from a specific date, set this property.
210214

211215
```yaml
212216
replication:
217+
copyTablesToCatchUp: true
213218
hostname: server-one
214219
routes:
215220
- wss://server-two:9925 # URL based route
216221
- hostname: server-three # define a hostname and port
217222
port: 9930
223+
startTime: 2024-02-06T15:30:00Z
218224
```
219225

220226
`port` - _Type_: integer;
@@ -227,7 +233,11 @@ The port to use for secure replication connections.
227233

228234
`enableRootCAs` - _Type_: boolean; _Default_: true
229235

230-
When false, HarperDB will not verify certificates against the Node.js bundled CA store. The bundled CA store is a snapshot of the Mozilla CA store that is fixed at release time.
236+
When true, HarperDB will verify certificates against the Node.js bundled CA store. The bundled CA store is a snapshot of the Mozilla CA store that is fixed at release time.
237+
238+
`copyTablesToCatchUp` - _Type_: boolean; _Default_: true
239+
240+
Replication will first attempt to catch up using the audit log. If unsuccessful, it will perform a full table copy. When set to `false`, replication will only use the audit log.
231241

232242
***
233243

@@ -598,6 +608,27 @@ logging:
598608
stdStreams: false
599609
```
600610

611+
`auditAuthEvents`
612+
613+
`logFailed` - _Type_: boolean; _Default_: false
614+
615+
Log all failed authentication events.
616+
617+
_Example:_ `[error] [auth-event]: {"username":"admin","status":"failure","type":"authentication","originating_ip":"127.0.0.1","request_method":"POST","path":"/","auth_strategy":"Basic"}`
618+
619+
`logSuccessful` - _Type_: boolean; _Default_: false
620+
621+
Log all successful authentication events.
622+
623+
_Example:_ `[notify] [auth-event]: {"username":"admin","status":"success","type":"authentication","originating_ip":"127.0.0.1","request_method":"POST","path":"/","auth_strategy":"Basic"}`
624+
625+
```yaml
626+
logging:
627+
auditAuthEvents:
628+
logFailed: false
629+
logSuccessful: false
630+
```
631+
601632
***
602633

603634
### `authentication`

docs/deployments/harperdb-cli.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,3 +158,25 @@ last_updated_record: 1724483231970.9949
158158
`harperdb search_by_value table=dog search_attribute=name search_value=harper get_attributes='["id", "name"]'`
159159

160160
`harperdb sql sql='select * from dev.dog where id="1"'`
161+
162+
## Remote Operations
163+
164+
The CLI can also be used to run operations on remote HarperDB instances. To do this, pass the `target` parameter with the HTTP address of the remote instance. You generally will also need to provide credentials and specify the `username` and `password` parameters, or you can set environment variables `CLI_TARGET_USERNAME` and `CLI_TARGET_PASSWORD`, for example:
165+
```bash
166+
export CLI_TARGET_USERNAME=HDB_ADMIN
167+
export CLI_TARGET_PASSWORD=password
168+
harperdb describe_database database=dev target=https://server.com:9925
169+
```
170+
171+
The same set of operations API are available for remote operations as well.
172+
173+
### Remote Component Deployment
174+
175+
When using remote operations, you can deploy a local component to the remote instance. If you omit the `package` parameter, you can deploy the current directory. This will package the current directory and send it to the target server (also `deploy` is allowed as an alias to `deploy_component`):
176+
```bash
177+
harperdb deploy target=https://server.com:9925
178+
```
179+
If you are interacting with a cluster, you may wish to include the `replicated=true` parameter to ensure that the deployment operation is replicated to all nodes in the cluster. You will also need to restart afterwards to apply the changes (here seen with the replicated parameter):
180+
```bash
181+
harperdb restart target=https://server.com:9925 replicated=true
182+
```

docs/developers/operations-api/clustering.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ _Operation is restricted to super_user roles only_
1414
* hostname or url _(required)_ - one of these fields is required. You must provide either the `hostname` or the `url` of the node you want to add
1515
* verify_tls _(optional)_ - a boolean which determines if the TLS certificate should be verified. This will allow the HarperDB default self-signed certificates to be accepted. Defaults to `true`
1616
* authorization _(optional)_ - an object or a string which contains the authorization information for the node being added. If it is an object, it should contain `username` and `password` fields. If it is a string, it should use HTTP `Authorization` style credentials
17+
* retain_authorization _(optional)_ - a boolean which determines if the authorization credentials should be retained/stored and used everytime a connection is made to this node. If `true`, the authorization will be stored on the node record. Generally this should not be used, as mTLS/certificate based authorization is much more secure and safe, and avoids the need for storing credentials. Defaults to `false`.
1718
* subscriptions _(optional)_ - The relationship created between nodes. If not provided a fully replicated cluster will be setup. Must be an object array and include `database`, `table`, `subscribe` and `publish`:
1819
* database - the database to replicate
1920
* table - the table to replicate

docs/developers/operations-api/components.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -27,9 +27,9 @@ _Operation is restricted to super_user roles only_
2727
---
2828
## Deploy Component
2929

30-
Will deploy a component using either a base64-encoded string representation of a `.tar` file (the output from `package_component`) or a package value, which can be any valid NPM reference, such as a GitHub repo, an NPM package, a tarball, a local directory or a website.\
30+
Will deploy a component using either a base64-encoded string representation of a `.tar` file (the output from `package_component`) or a package value, which can be any valid NPM reference, such as a GitHub repo, an NPM package, a tarball, a local directory or a website.
3131

32-
If deploying with the `payload` option, HarperDB will decrypt the base64-encoded string, reconstitute the .tar file of your project folder, and extract it to the component root project directory.\
32+
If deploying with the `payload` option, HarperDB will decrypt the base64-encoded string, reconstitute the .tar file of your project folder, and extract it to the component root project directory.
3333

3434
If deploying with the `package` option, the package value will be written to `harperdb-config.yaml`. Then npm install will be utilized to install the component in the `node_modules` directory located in the hdb root. The value is a package reference, which should generally be a [URL reference, as described here](https://docs.npmjs.com/cli/v10/configuring-npm/package-json#urls-as-dependencies) (it is also possible to include NPM registerd packages and file paths). URL package references can directly reference tarballs that can be installed as a package. However, the most common and recommended usage is to install from a Git repository, which can be combined with a tag to deploy a specific version directly from versioned source control. When using tags, we highly recommend that you use the `semver` directive to ensure consistent and reliable installation by NPM. In addition to tags, you can also reference branches or commit numbers. Here is an example URL package reference to a (public) Git repository that doesn't require authentication:
3535
```
@@ -62,8 +62,9 @@ _Operation is restricted to super_user roles only_
6262
* project _(required)_ - the name of the project you wish to deploy
6363
* package _(optional)_ - this can be any valid GitHub or NPM reference
6464
* payload _(optional)_ - a base64-encoded string representation of the .tar file. Must be a string
65-
* restart _(optional)_ - if true, HarperDB will restart after deploying the component. Must be a boolean
65+
* restart _(optional)_ - must be either a boolean or the string `rolling`. If set to `rolling`, a rolling restart will be triggered after the component is deployed, meaning that each node in the cluster will be sequentially restarted (waiting for the last restart to start the next). If set to `true`, the restart will not be rolling, all nodes will be restarted in parallel. If `replicated` is `true`, the restart operations will be replicated across the cluster.
6666
* replicated _(optional)_ - if true, HarperDB will replicate the component to all nodes in the cluster. Must be a boolean.
67+
* install_command _(optional)_ - A command to use when installing the component. Must be a string. This can be used to install dependencies with pnpm or yarn, for example, like: `"install_command": "npm install -g pnpm && pnpm install"`
6768

6869
### Body
6970

@@ -133,6 +134,7 @@ _Operation is restricted to super_user roles only_
133134
* project _(required)_ - the name of the project you wish to delete or to delete from if using the `file` parameter
134135
* file _(optional)_ - the path relative to your project folder of the file you wish to delete
135136
* replicated _(optional)_ - if true, HarperDB will replicate the component deletion to all nodes in the cluster. Must be a boolean.
137+
* restart _(optional)_ - if true, HarperDB will restart after dropping the component. Must be a boolean.
136138

137139
### Body
138140

@@ -463,4 +465,4 @@ _Operation is restricted to super_user roles only_
463465
{
464466
"known_hosts": "github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=\ngithub.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl\ngithub.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=\n"
465467
}
466-
```
468+
```

docs/developers/operations-api/utilities.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ _Operation is restricted to super_user roles only_
2929

3030
* operation _(required)_ - must always be `restart_service`
3131
* service _(required)_ - must be one of: `http_workers`, `clustering_config` or `clustering`
32+
* replicated _(optional)_ - must be a boolean. If set to `true`, HarperDB will replicate the restart service operation across all nodes in the cluster. The restart will occur as a rolling restart, ensuring that each node is fully restarted before the next node begins restarting.
3233

3334
### Body
3435
```json
@@ -52,7 +53,7 @@ Returns detailed metrics on the host system.
5253
_Operation is restricted to super_user roles only_
5354

5455
* operation _(required)_ - must always be `system_information`
55-
* attributes _(optional)_ - string array of top level attributes desired in the response, if no value is supplied all attributes will be returned. Available attributes are: ['system', 'time', 'cpu', 'memory', 'disk', 'network', 'harperdb_processes', 'table_size', 'replication']
56+
* attributes _(optional)_ - string array of top level attributes desired in the response, if no value is supplied all attributes will be returned. Available attributes are: ['system', 'time', 'cpu', 'memory', 'disk', 'network', 'harperdb_processes', 'table_size', 'metrics', 'threads', 'replication']
5657

5758
### Body
5859
```json
@@ -364,7 +365,8 @@ _Operation is restricted to super_user roles only_
364365
### Body
365366
```json
366367
{
367-
"operation": "remove_certificate"
368+
"operation": "remove_certificate",
369+
"name": "my-cert"
368370
}
369371
```
370372

docs/developers/replication/README.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,13 @@ HarperDB supports the highest levels of security through public key infrastructu
7676

7777
#### Provide your own certificates
7878

79-
If you want to secure your HarperDB connections with your own signed certificates, you can easily do so. Whether you have certificates from a public authority (like Let's Encrypt or Digicert) or a corporate certificate authority, you can use them to authenticate nodes securely. Just make sure the certificate’s common name (CN) matches the node's hostname.
79+
If you want to secure your HarperDB connections with your own signed certificates, you can easily do so. Whether you have certificates from a public authority (like Let's Encrypt or Digicert) or a corporate certificate authority, you can use them to authenticate nodes securely. You can then allow nodes to authorize each other by checking the certificate against the standard list of root certificate authorities by enabling the `enableRootCAs` option in the config:
80+
```
81+
replication
82+
enableRootCAs: true
83+
```
84+
85+
And then just make sure the certificate’s common name (CN) matches the node's hostname.
8086

8187
#### Setting Up Custom Certificates
8288

docs/technical-details/reference/limits.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,4 +26,7 @@ Additionally, they cannot contain the first 31 non-printing characters. Spaces a
2626

2727
**Attribute Maximum**
2828

29-
HarperDB limits the number of total indexed attributes across tables (including the primary key of each table) to 10,000 per database.
29+
HarperDB limits the number of total indexed attributes across tables (including the primary key of each table) to 10,000 per database.
30+
31+
## Primary Keys
32+
The maximum length of a primary key is 1978 bytes or 659 characters (whichever is shortest).

0 commit comments

Comments
 (0)