Skip to content

Commit 7873b35

Browse files
committed
Merge remote-tracking branch 'origin/main' into vector-indexing
2 parents 50d22f6 + 0d3ec27 commit 7873b35

29 files changed

+771
-480
lines changed

.gitbook.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
root: ./docs/
22

33
redirects:
4+
developers/operations-api/utilities: developers/operations-api/system-operations.md
45
install-harperdb: deployments/install-harper/README.md
56
install-harperdb/linux: deployments/install-harper/linux.md
67
install-harperdb/other: deployments/install-harper/README.md

docs/SUMMARY.md

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,13 @@
2626
* [Users and Roles](developers/operations-api/users-and-roles.md)
2727
* [Clustering](developers/operations-api/clustering.md)
2828
* [Clustering with NATS](developers/operations-api/clustering-nats.md)
29-
* [Custom Functions](developers/operations-api/custom-functions.md)
3029
* [Components](developers/operations-api/components.md)
3130
* [Registration](developers/operations-api/registration.md)
3231
* [Jobs](developers/operations-api/jobs.md)
3332
* [Logs](developers/operations-api/logs.md)
34-
* [Utilities](developers/operations-api/utilities.md)
33+
* [System Operations](developers/operations-api/system-operations.md)
34+
* [Configuration](developers/operations-api/configuration.md)
35+
* [Certificate Management](developers/operations-api/certificate-management.md)
3536
* [Token Authentication](developers/operations-api/token-authentication.md)
3637
* [SQL Operations](developers/operations-api/sql-operations.md)
3738
* [Advanced JSON SQL Examples](developers/operations-api/advanced-json-sql-examples.md)
@@ -120,9 +121,18 @@
120121
* [Resource Class](technical-details/reference/resource.md)
121122
* [Transactions](technical-details/reference/transactions.md)
122123
* [Storage Algorithm](technical-details/reference/storage-algorithm.md)
124+
* [Blob](technical-details/reference/blob.md)
123125
* [Release Notes](technical-details/release-notes/README.md)
124126
* [Harper Tucker (Version 4)](technical-details/release-notes/4.tucker/README.md)
125127
* [4.6.0](technical-details/release-notes/4.tucker/4.6.0.md)
128+
* [4.5.10](technical-details/release-notes/4.tucker/4.5.10.md)
129+
* [4.5.9](technical-details/release-notes/4.tucker/4.5.9.md)
130+
* [4.5.8](technical-details/release-notes/4.tucker/4.5.8.md)
131+
* [4.5.7](technical-details/release-notes/4.tucker/4.5.7.md)
132+
* [4.5.6](technical-details/release-notes/4.tucker/4.5.6.md)
133+
* [4.5.5](technical-details/release-notes/4.tucker/4.5.5.md)
134+
* [4.5.4](technical-details/release-notes/4.tucker/4.5.4.md)
135+
* [4.5.3](technical-details/release-notes/4.tucker/4.5.3.md)
126136
* [4.5.2](technical-details/release-notes/4.tucker/4.5.2.md)
127137
* [4.5.1](technical-details/release-notes/4.tucker/4.5.1.md)
128138
* [4.5.0](technical-details/release-notes/4.tucker/4.5.0.md)

docs/developers/applications/defining-schemas.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,8 @@ By default the table name is inherited from the type name (in this case the tabl
4141
* `@table(expiration: 3600)` - Sets an expiration time on entries in the table before they are automatically cleared (primarily useful for caching tables). This is specified in seconds.
4242
* `@table(audit: true)` - This enables the audit log for the table so that a history of record changes are recorded. This defaults to [configuration file's setting for `auditLog`](../../deployments/configuration.md#logging).
4343

44+
Database naming: the default "data" database is generally a good default choice for tables in applications that will not be reused in other applications (and don't need to worry about staying in a separate namespace). Application with many tables may wish to organize the tables into separate databases (but remember that transactions do not preserve atomicity across different databases, only across tables in the same database). For components that are designed for re-use, it is recommended that you use a database name that is specific to the component (e.g. "my-component-data") to avoid name collisions with other components.
45+
4446
#### `@export`
4547

4648
This indicates that the specified table should be exported as a resource that is accessible as an externally available endpoints, through REST, MQTT, or any of the external resource APIs.

docs/developers/components/built-in.md

Lines changed: 36 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -20,13 +20,13 @@ Harper provides extended features using built-in components. They do **not** nee
2020

2121
Specify custom endpoints using [Fastify](https://fastify.dev/).
2222

23-
This component is a [Resource Extension](reference.md#resource-extension) and can be configured with the [`files`, `path`, and `root`](reference.md#resource-extension-configuration) configuration options.
23+
This component is a [Resource Extension](./reference.md#resource-extension) and can be configured with the [`files` and `urlPath`](./reference.md#resource-extension-configuration) configuration options.
2424

2525
Complete documentation for this feature is available here: [Define Fastify Routes](../applications/define-routes.md)
2626

2727
```yaml
2828
fastifyRoutes:
29-
files: './routes/*.js'
29+
files: 'routes/*.js'
3030
```
3131
3232
## graphql
@@ -45,13 +45,13 @@ graphql: true
4545

4646
Specify schemas for Harper tables and resources via GraphQL schema syntax.
4747

48-
This component is a [Resource Extension](reference.md#resource-extension) and can be configured with the [`files`, `path`, and `root`](reference.md#resource-extension-configuration) configuration options.
48+
This component is a [Resource Extension](./reference.md#resource-extension) and can be configured with the [`files` and `urlPath`](./reference.md#resource-extension-configuration) configuration options.
4949

5050
Complete documentation for this feature is available here: [Defining Schemas](../applications/defining-schemas.md)
5151

5252
```yaml
5353
graphqlSchema:
54-
files: './schemas.graphql'
54+
files: 'schemas.graphql'
5555
```
5656

5757
## jsResource
@@ -60,18 +60,18 @@ Specify custom, JavaScript based Harper resources.
6060

6161
Refer to the Application [Custom Functionality with JavaScript](../applications/#custom-functionality-with-javascript) guide, or [Resource Class](../../technical-details/reference/resource.md) reference documentation for more information on custom resources.
6262

63-
This component is a [Resource Extension](reference.md#resource-extension) and can be configured with the [`files`, `path`, and `root`](reference.md#resource-extension-configuration) configuration options.
63+
This component is a [Resource Extension](./reference.md#resource-extension) and can be configured with the [`files` and `urlPath`](./reference.md#resource-extension-configuration) configuration options.
6464

6565
```yaml
6666
jsResource:
67-
files: './resource.js'
67+
files: 'resource.js'
6868
```
6969

7070
## loadEnv
7171

7272
Load environment variables via files like `.env`.
7373

74-
This component is a [Resource Extension](./reference.md#resource-extension) and can be configured with the [`files`, `path`, and `root`](./reference.md#resource-extension-configuration) configuration options.
74+
This component is a [Resource Extension](./reference.md#resource-extension) and can be configured with the [`files` and `urlPath`](./reference.md#resource-extension-configuration) configuration options.
7575

7676
Ensure this component is specified first in `config.yaml` so that environment variables are loaded prior to loading any other components.
7777

@@ -128,22 +128,46 @@ rest:
128128

129129
Specify roles for Harper tables and resources.
130130

131-
This component is a [Resource Extension](reference.md#resource-extension) and can be configured with the [`files`, `path`, and `root`](reference.md#resource-extension-configuration) configuration options.
131+
This component is a [Resource Extension](./reference.md#resource-extension) and can be configured with the [`files` and `urlPath`](./reference.md#resource-extension-configuration) configuration options.
132132

133133
Complete documentation for this feature is available here: [Defining Roles](../applications/defining-roles.md)
134134

135135
```yaml
136136
roles:
137-
files: './roles.yaml'
137+
files: 'roles.yaml'
138138
```
139139

140140
## static
141141

142-
Specify which files to server statically from the Harper HTTP endpoint. Built using the [send](https://www.npmjs.com/package/send) and [serve-static](https://www.npmjs.com/package/serve-static) modules.
142+
Specify files to serve statically from the Harper HTTP endpoint.
143143

144-
This component is a [Resource Extension](reference.md#resource-extension) and can be configured with the [`files`, `path`, and `root`](reference.md#resource-extension-configuration) configuration options.
144+
Use the [Resource Extension](./reference.md#resource-extension) configuration options [`files` and `urlPath`](./reference.md#resource-extension-configuration) to specify the files to be served.
145+
146+
As specified by Harper's Resource Extension docs, the `files` option can be any glob pattern or a glob options object. This extension will serve all files matching the pattern, so make sure to be specific.
147+
148+
To serve the entire `web` directory, specify `files: 'web/**'`.
149+
150+
To serve only the html files within `web`, specify `files: 'web/*.html'` or `files: 'web/**/*.html'`.
151+
152+
The `urlPath` option is the base URL path entries will be resolved to. For example, a `urlPath: 'static'` will serve all files resolved from `files` to the URL path `localhost/static/`.
153+
154+
Given the `config.yaml`:
145155

146156
```yaml
147157
static:
148-
files: './web/*'
158+
files: 'web/*.html'
159+
urlPath: 'static'
149160
```
161+
162+
And the file directory structure:
163+
164+
```
165+
component/
166+
├─ web/
167+
│ ├─ index.html
168+
│ ├─ blog.html
169+
├─ config.yaml
170+
171+
```
172+
173+
The HTML files will be available at `localhost/static/index.html` and `localhost/static/blog.html` respectively.

docs/developers/components/reference.md

Lines changed: 49 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ And now in `config.yaml`:
6565
```yaml
6666
harper-nextjs-test-feature:
6767
package: '@harperdb/nextjs'
68-
files: '/*'
68+
files: './'
6969
# ...
7070
```
7171

@@ -84,7 +84,7 @@ jsResource:
8484
files: 'resources.js'
8585
fastifyRoutes:
8686
files: 'routes/*.js'
87-
path: '.'
87+
urlPath: '.'
8888
static:
8989
files: 'web/**'
9090
```
@@ -115,35 +115,64 @@ Furthermore, what defines an extension separately from a component is that it le
115115

116116
A Resource Extension is for processing a certain type of file or directory. For example, the built-in [jsResource](./built-in.md#jsresource) extension handles executing JavaScript files.
117117

118-
Resource Extensions are comprised of four distinct function exports, [`handleFile()`](#handlefilecontents-urlpath-path-resources-void--promisevoid), [`handleDirectory()`](#handledirectoryurlpath-path-resources-boolean--void--promiseboolean--void), [`setupFile()`](#setupfilecontents-urlpath-path-resources-void--promisevoid), and [`setupDirectory()`](#setupdirectoryurlpath-path-resources-boolean--void--promiseboolean--void). The `handleFile()` and `handleDirectory()` methods are executed on **all worker threads**, and are _executed again during restarts_. The `setupFile()` and `setupDirectory()` methods are only executed **once** on the **main thread** during the initial system start sequence.
118+
Resource Extensions are comprised of four distinct function exports, [`handleFile()`](#handlefilecontents-urlpath-absolutepath-resources-void--promisevoid), [`handleDirectory()`](#handledirectoryurlpath-absolutepath-resources-boolean--void--promiseboolean--void), [`setupFile()`](#setupfilecontents-urlpath-absolutepath-resources-void--promisevoid), and [`setupDirectory()`](#setupdirectoryurlpath-absolutepath-resources-boolean--void--promiseboolean--void). The `handleFile()` and `handleDirectory()` methods are executed on **all worker threads**, and are _executed again during restarts_. The `setupFile()` and `setupDirectory()` methods are only executed **once** on the **main thread** during the initial system start sequence.
119119

120120
> Keep in mind that the CLI command `harperdb restart` or CLI argument `restart=true` only restarts the worker threads. If a component is deployed using `harperdb deploy`, the code within the `setupFile()` and `setupDirectory()` methods will not be executed until the system is completely shutdown and turned back on.
121121

122122
Other than their execution behavior, the `handleFile()` and `setupFile()` methods, and `handleDirectory()` and `setupDirectory()` methods have identical function definitions (arguments and return value behavior).
123123

124124
#### Resource Extension Configuration
125125

126-
Any [Resource Extension](#resource-extension) can be configured with the `files`, `path`, and `root` options. These options control how _files_ and _directories_ are resolved in order to be passed to the extension's `handleFile()`, `setupFile()`, `handleDirectory()`, and `setupDirectory()` methods.
126+
Any [Resource Extension](#resource-extension) can be configured with the `files` and `path` options. These options control how _files_ and _directories_ are resolved in order to be passed to the extension's `handleFile()`, `setupFile()`, `handleDirectory()`, and `setupDirectory()` methods.
127127

128-
- **files** - `string` - *required* - Specifies the set of files and directories that should be handled by the component. Can be a glob pattern.
129-
- **path** - `string` - *optional* - Specifies the URL path to be handled by the component.
130-
- **root** - `string` - *optional* - Specifies the root directory for mapping file paths to the URLs.
128+
- **files** - `string | string[] | Object` - *required* - A [glob pattern](https://github.com/mrmlnc/fast-glob?tab=readme-ov-file#pattern-syntax) string, array of glob pattern strings, or a more expressive glob options object determining the set of files and directories to be resolved for the extension. If specified as an object, the `source` property is required. By default, Harper matches files *and directories*.
129+
- **source** - `string | string[]` - *required* - The glob pattern string or array of strings.
130+
- **onlyFiles** - `boolean` - *optional* - Defaults to `false`, meaning the glob pattern will match directories and files.
131+
- **ignore** - `string | string[]` - *optional* - A glob pattern string or array of strings for files to be ignored. Defaults to `[]`.
132+
- **urlPath** - `string` - *optional* - A base URL path to prepend to the resolved `files` entries.
133+
- If the value starts with `./`, such as `'./static/'`, the component name will be included in the base url path
134+
- If the value is `.`, then the component name will be the base url path
135+
- Note: `..` is an invalid pattern and will result in an error
136+
- Otherwise, the value here will be base url path. Leading and trailing `/` characters will be handled automatically (`/static/`, `/static`, and `static/` are all equivalent to `static`)
131137

132-
For example, to configure the [static](./built-in.md#static) component to server all files from `web` to the root URL path:
138+
For example, to configure the [static](./built-in.md#static) component to serve all HTML files from the `web` source directory on the `static` URL endpoint:
133139

134140
```yaml
135141
static:
136-
files: 'web/**'
137-
root: 'web'
142+
files: 'web/*.html'
143+
urlPath: 'static'
138144
```
139145

140-
Or, to configure the [graphqlSchema](./built-in.md#graphqlschema) component to load all schemas within the `src/schema` directory:
146+
If there are files such as `web/index.html` and `web/blog.html`, they would be available at `localhost/static/index.html` and `localhost/static/blog.html` respectively.
147+
148+
Furthermore, if the component is located in the `test-component` directory, and the `urlPath` was set to `'./static/'` instead, then the files would be served from `localhost/test-component/static/*` instead.
149+
150+
The `urlPath` is optional, for example to configure the [graphqlSchema](./built-in.md#graphqlschema) component to load all schemas within the `src/schema` directory, only specifying a `files` glob pattern is required:
141151

142152
```yaml
143153
graphqlSchema:
144154
files: 'src/schema/*.schema'
145155
```
146156

157+
The `files` option also supports a more complex options object. These additional fields enable finer control of the glob pattern matching.
158+
159+
For example, to match files within `web`, and omit any within the `web/images` directory, the configuration could be:
160+
```yaml
161+
static:
162+
files:
163+
source: 'web/**/*'
164+
ignore: 'web/images'
165+
```
166+
167+
Or to disable matching directories within the glob pattern:
168+
169+
```yaml
170+
test-component:
171+
files:
172+
source: 'dir/**/*'
173+
onlyFiles: true
174+
```
175+
147176
#### Resource Extension API
148177

149178
In order for an extension to be classified as a Resource Extension it must implement at least one of the `handleFile()`, `handleDirectory()`, `setupFile()`, or `setupDirectory()` methods. As a standalone extension, these methods should be named and exported directly. For example:
@@ -170,8 +199,8 @@ export function start() {
170199
}
171200
```
172201

173-
##### `handleFile(contents, urlPath, path, resources): void | Promise<void>`
174-
##### `setupFile(contents, urlPath, path, resources): void | Promise<void>`
202+
##### `handleFile(contents, urlPath, absolutePath, resources): void | Promise<void>`
203+
##### `setupFile(contents, urlPath, absolutePath, resources): void | Promise<void>`
175204

176205
These methods are for processing individual files. They can be async.
177206

@@ -185,14 +214,14 @@ Parameters:
185214

186215
- **contents** - `Buffer` - The contents of the file
187216
- **urlPath** - `string` - The recommended URL path of the file
188-
- **path** - `string` - The relative path of the file
217+
- **absolutePath** - `string` - The absolute path of the file
189218
<!-- TODO: Replace the Object type here with a more specific type representing the resources argument of loadComponent() -->
190219
- **resources** - `Object` - A collection of the currently loaded resources
191220

192221
Returns: `void | Promise<void>`
193222

194-
##### `handleDirectory(urlPath, path, resources): boolean | void | Promise<boolean | void>`
195-
##### `setupDirectory(urlPath, path, resources): boolean | void | Promise<boolean | void>`
223+
##### `handleDirectory(urlPath, absolutePath, resources): boolean | void | Promise<boolean | void>`
224+
##### `setupDirectory(urlPath, absolutePath, resources): boolean | void | Promise<boolean | void>`
196225

197226
These methods are for processing directories. They can be async.
198227

@@ -206,8 +235,8 @@ If the function returns or resolves a truthy value, then the component loading s
206235

207236
Parameters:
208237

209-
- **urlPath** - `string` - The recommended URL path of the file
210-
- **path** - `string` - The relative path of the directory
238+
- **urlPath** - `string` - The recommended URL path of the directory
239+
- **absolutePath** - `string` - The absolute path of the directory
211240
<!-- TODO: Replace the Object type here with a more specific type representing the resources argument of loadComponent() -->
212241
- **resources** - `Object` - A collection of the currently loaded resources
213242

@@ -219,14 +248,14 @@ A Protocol Extension is a more advanced form of a Resource Extension and is main
219248

220249
#### Protocol Extension Configuration
221250

222-
In addition to the `files`, `path`, and `root` [Resource Extension configuration](#resource-extension-configuration) options, and the `package` [Custom Component configuration](#custom-component-configuration) option, Protocol Extensions can also specify additional configuration options. Any options added to the extension configuration (in `config.yaml`), will be passed through to the `options` object of the `start()` and `startOnMainThread()` methods.
251+
In addition to the `files` and `urlPath` [Resource Extension configuration](#resource-extension-configuration) options, and the `package` [Custom Component configuration](#custom-component-configuration) option, Protocol Extensions can also specify additional configuration options. Any options added to the extension configuration (in `config.yaml`), will be passed through to the `options` object of the `start()` and `startOnMainThread()` methods.
223252

224253
For example, the [Harper Next.js Extension](https://github.com/HarperDB/nextjs#options) specifies multiple option that can be included in its configuration. For example, a Next.js app using `@harperdb/nextjs` may specify the following `config.yaml`:
225254

226255
```yaml
227256
'@harperdb/nextjs':
228257
package: '@harperdb/nextjs'
229-
files: '/*'
258+
files: './'
230259
prebuilt: true
231260
dev: false
232261
```

0 commit comments

Comments
 (0)