Merge branch 'release_4.6'

Author: kriszyp

docs/SUMMARY.md

@@ -536,7 +536,7 @@ localStudio:
 
 ### `logging`
 
-The `logging` section configures Harper logging across all Harper functionality. This includes standard text logging of application and database events as well as structured data logs of record changes. Logging of application/database events are logged in text format to the `~/hdb/log/hdb.log` file (or location specified by `logging.root`).
+The `logging` section configures Harper logging across all Harper functionality. This includes standard text logging of application and database events as well as structured data logs of record changes. Logging of application/database events are logged in text format to the `~/hdb/log/hdb.log` file (or location specified by `logging.root` or `logging.path`). Many of the logging configuration properties can be set and applied without a restart (are dynamically applied).
 
 In addition, structured logging of data changes are also available:
 
@@ -585,7 +585,7 @@ There exists a log level hierarchy in order as `trace`, `debug`, `info`, `warn`,
 
 `console` - _Type_: boolean; _Default_: true
 
-Controls whether console.log and other console.\* calls (as well as another JS components that writes to `process.stdout` and `process.stderr`) are logged to the log file. By default, these are logged to the log file, but this can be disabled.
+Controls whether console.log and other console.\* calls (as well as another JS components that writes to `process.stdout` and `process.stderr`) are logged to the log file. By default, these are not logged to the log file, but this can be enabled:
 
 ```yaml
 logging:
@@ -594,13 +594,22 @@ logging:
 
 `root` - _Type_: string; _Default_: \<ROOTPATH>/log
 
-The path where the log files will be written.
+The directory path where the log files will be written.
 
 ```yaml
 logging:
   root: ~/hdb/log
 ```
 
+`path` - _Type_: string; _Default_: \<ROOTPATH>/log/hdb.log
+
+The file path where the log file will be written.
+
+```yaml
+logging:
+  root: ~/hdb/log/hdb.log
+```
+
 `rotation`
 
 Rotation provides the ability for a user to systematically rotate and archive the `hdb.log` file. To enable `interval` and/or `maxSize` must be set.
@@ -667,7 +676,70 @@ logging:
     logSuccessful: false
 ```
 
----
+## Defining Separate Logging Configurations
+
+Harper's logger supports defining multiple logging configurations for different components in the system. Each logging configuration can be assigned its own `path` (or `root`), `level`, `tag`, and flag to enable/disable logging to `stdStreams`. All logging defaults to the configuration of the "main" logger as configured above, but when logging is configured for different loggers, they will use their own configuration. Separate loggers can be defined:
+
+`logging.external`
+
+The `logging.external` section can be used to define logging for all external components that use the [`logger` API](../technical-details/reference/globals.md). For example:
+```yaml
+logging:
+  external:
+    level: warn
+    path: ~/hdb/log/apps.log
+```
+
+`http.logging`
+
+This section defines log configuration for HTTP logging. By default, HTTP requests are not logged, but defining this section will enable HTTP logging. Note that there can be substantive overhead to logging all HTTP requests. In addition to the standard logging configuration, the `http.logging` section also allows the following configuration properties to be set:
+* `timing` - This will log timing information
+* `headers` - This will log the headers in each request (which can be very verbose)
+* `id` - This will assign a unique id to each request and log it in the entry for each request. This is assigned as the `request.requestId` property and can be used to by other logging to track a request.
+Note that the `level` will determine which HTTP requests are logged:
+* `info` (or more verbose) - All HTTP requests
+* `warn` - HTTP requests with a status code of 400 or above
+* `error` - HTTP requests with a status code of 500
+
+For example:
+```yaml
+http:
+  logging: 
+    timing: true
+    level: info
+    path: ~/hdb/log/http.log 
+  ... rest of http config
+```
+
+`authentication.logging`
+
+This section defines log configuration for authentication. This takes the standard logging configuration options of `path` (or `root`), `level`, `tag`, and flag to enable/disable logging to `stdStreams`.
+
+`mqtt.logging`
+
+This section defines log configuration for MQTT. This takes the standard logging configuration options of `path` (or `root`), `level`, `tag`, and flag to enable/disable logging to `stdStreams`.
+
+`replication.logging`
+
+This section defines log configuration for replication. This takes the standard logging configuration options of `path` (or `root`), `level`, `tag`, and flag to enable/disable logging to `stdStreams`.
+
+`tls.logging`
+
+This section defines log configuration for TLS. This takes the standard logging configuration options of `path` (or `root`), `level`, `tag`, and flag to enable/disable logging to `stdStreams`.
+
+`storage.logging`
+
+This section defines log configuration for setting up and reading the database files. This takes the standard logging configuration options of `path` (or `root`), `level`, `tag`, and flag to enable/disable logging to `stdStreams`.
+
+`storage.logging`
+
+This section defines log configuration for setting up and reading the database files. This takes the standard logging configuration options of `path` (or `root`), `level`, `tag`, and flag to enable/disable logging to `stdStreams`.
+
+`analytics.logging`
+
+This section defines log configuration for analytics. This takes the standard logging configuration options of `path` (or `root`), `level`, `tag`, and flag to enable/disable logging to `stdStreams`.
+
+***
 
 ### `authentication`
 

docs/deployments/configuration.md

@@ -37,8 +37,8 @@ class ThirdPartyAPI extends Resource {
 Next, we define this external data resource as the "source" for the caching table we defined above:
 
 ```javascript
-const { MyTable } = tables;
-MyTable.sourcedFrom(ThirdPartyAPI);
+const { MyCache } = tables;
+MyCache.sourcedFrom(ThirdPartyAPI);
 ```
 
 Now we have a fully configured and connected caching table. If you access data from `MyCache` (for example, through the REST API, like `/MyCache/some-id`), Harper will check to see if the requested entry is in the table and return it if it is available (and hasn't expired). If there is no entry, or it has expired (it is older than one hour in this case), it will go to the source, calling the `get()` method, which will then retrieve the requested entry. Once the entry is retrieved, it will be saved/cached in the caching table (for one hour based on our expiration time).

docs/developers/applications/caching.md

@@ -0,0 +1,172 @@
+# Data Loader
+
+The Data Loader is a built-in component that provides a reliable mechanism for loading data from JSON or YAML files into Harper tables as part of component deployment. This feature is particularly useful for ensuring specific records exist in your database when deploying components, such as seed data, configuration records, or initial application data.
+
+## Configuration
+
+To use the Data Loader, first specify your data files in the `config.yaml` in your component directory:
+
+```yaml
+dataLoader:
+  files: 'data/*.json'
+```
+
+The Data Loader is an [Extension](../components/reference.md#extensions) and supports the standard `files` configuration option.
+
+## Data File Format
+
+Data files can be structured as either JSON or YAML files containing the records you want to load. Each data file must specify records for a single table - if you need to load data into multiple tables, create separate data files for each table.
+
+### Basic Example
+
+Create a data file in your component's data directory (one table per file):
+
+```json
+{
+  "database": "myapp",
+  "table": "users",
+  "records": [
+    {
+      "id": 1,
+      "username": "admin",
+      "email": "[email protected]",
+      "role": "administrator"
+    },
+    {
+      "id": 2,
+      "username": "user1",
+      "email": "[email protected]",
+      "role": "standard"
+    }
+  ]
+}
+```
+
+### Multiple Tables
+
+To load data into multiple tables, create separate data files for each table:
+
+**users.json:**
+```json
+{
+  "database": "myapp",
+  "table": "users",
+  "records": [
+    {
+      "id": 1,
+      "username": "admin",
+      "email": "[email protected]"
+    }
+  ]
+}
+```
+
+**settings.yaml:**
+```yaml
+database: myapp
+table: settings
+records:
+  - id: 1
+    setting_name: app_name
+    setting_value: My Application
+  - id: 2
+    setting_name: version
+    setting_value: "1.0.0"
+```
+
+## File Organization
+
+You can organize your data files in various ways:
+
+### Single File Pattern
+```yaml
+dataLoader:
+  files: 'data/seed-data.json'
+```
+
+### Multiple Files Pattern
+```yaml
+dataLoader:
+  files: 
+    - 'data/users.json'
+    - 'data/settings.yaml'
+    - 'data/initial-products.json'
+```
+
+### Glob Pattern
+```yaml
+dataLoader:
+  files: 'data/**/*.{json,yaml,yml}'
+```
+
+## Loading Behavior
+
+When Harper starts up with a component that includes the Data Loader:
+
+1. The Data Loader reads all specified data files (JSON or YAML)
+2. For each file, it validates that a single table is specified
+3. Records are inserted or updated based on timestamp comparison:
+   - New records are inserted if they don't exist
+   - Existing records are updated only if the data file's modification time is newer than the record's updated time
+   - This ensures data files can be safely reloaded without overwriting newer changes
+4. If records with the same primary key already exist, updates occur only when the file is newer
+
+Note: While the Data Loader can create tables automatically by inferring the schema from the provided records, it's recommended to define your table schemas explicitly using the [graphqlSchema](../applications/defining-schemas.md) component for better control and type safety.
+
+## Best Practices
+
+1. **Define Schemas First**: While the Data Loader can infer schemas, it's strongly recommended to define your table schemas and relations explicitly using the [graphqlSchema](../applications/defining-schemas.md) component before loading data. This ensures proper data types, constraints, and relationships between tables.
+
+2. **One Table Per File**: Remember that each data file can only load records into a single table. Organize your files accordingly.
+
+3. **Idempotency**: Design your data files to be idempotent - they should be safe to load multiple times without creating duplicate or conflicting data.
+
+4. **Version Control**: Include your data files in version control to ensure consistency across deployments.
+
+5. **Environment-Specific Data**: Consider using different data files for different environments (development, staging, production).
+
+6. **Data Validation**: Ensure your data files are valid JSON or YAML and match your table schemas before deployment.
+
+7. **Sensitive Data**: Avoid including sensitive data like passwords or API keys directly in data files. Use environment variables or secure configuration management instead.
+
+## Example Component Structure
+
+```
+my-component/
+├── config.yaml
+├── data/
+│   ├── users.json
+│   ├── roles.json
+│   └── settings.json
+├── schemas.graphql
+└── roles.yaml
+```
+
+With this structure, your `config.yaml` might look like:
+
+```yaml
+# Load environment variables first
+loadEnv:
+  files: '.env'
+
+# Define schemas
+graphqlSchema:
+  files: 'schemas.graphql'
+
+# Define roles
+roles:
+  files: 'roles.yaml'
+
+# Load initial data
+dataLoader:
+  files: 'data/*.json'
+
+# Enable REST endpoints
+rest: true
+```
+
+## Related Documentation
+
+- [Built-In Components](../components/built-in.md)
+- [Extensions](../components/reference.md#extensions)
+- [Bulk Operations](../operations-api/bulk-operations.md) - For loading data via the Operations API

docs/developers/applications/data-loader.md

@@ -15,23 +15,24 @@ Content-Type: application/json
 
 The operations API reference is available below and categorized by topic:
 
-- [Quick Start Examples](quickstart-examples.md)
-- [Databases and Tables](databases-and-tables.md)
-- [NoSQL Operations](nosql-operations.md)
-- [Bulk Operations](bulk-operations.md)
-- [Users and Roles](users-and-roles.md)
-- [Clustering](clustering.md)
-- [Clustering with NATS](clustering-nats.md)
-- [Components](components.md)
-- [Registration](registration.md)
-- [Jobs](jobs.md)
-- [Logs](logs.md)
-- [System Operations](system-operations.md)
-- [Configuration](configuration.md)
-- [Certificate Management](certificate-management.md)
-- [Token Authentication](token-authentication.md)
-- [SQL Operations](sql-operations.md)
-- [Advanced JSON SQL Examples](advanced-json-sql-examples.md)
+* [Quick Start Examples](quickstart-examples.md)
+* [Databases and Tables](databases-and-tables.md)
+* [NoSQL Operations](nosql-operations.md)
+* [Bulk Operations](bulk-operations.md)
+* [Users and Roles](users-and-roles.md)
+* [Clustering](clustering.md)
+* [Clustering with NATS](clustering-nats.md)
+* [Components](components.md)
+* [Registration](registration.md)
+* [Jobs](jobs.md)
+* [Logs](logs.md)
+* [System Operations](system-operations.md)
+* [Configuration](configuration.md)
+* [Certificate Management](certificate-management.md)
+* [Token Authentication](token-authentication.md)
+* [SQL Operations](sql-operations.md)
+* [Advanced JSON SQL Examples](advanced-json-sql-examples.md)
+* [Analytics](analytics.md)
 
 • [Past Release API Documentation](https://olddocs.harperdb.io)