Skip to content

Add table describing known analytics #281

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cap10morgan merged 8 commits into from
Sep 2, 2025
Merged

Add table describing known analytics #281

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
9d71f8e
Add table describing known analytics
kriszyp Aug 22, 2025
ecef5a2
More table updates
kriszyp Aug 22, 2025
1b834d6
Automated analytics table formatting fixes
cap10morgan Aug 25, 2025
dc20de7
Document built-in analytics
cap10morgan Aug 25, 2025
8301503
Add missing units to some metrics
cap10morgan Sep 2, 2025
6b85368
Fix formatting
cap10morgan Sep 2, 2025
a299e1d
Copy new analytics docs to 4.6
cap10morgan Sep 2, 2025
0bc9fab
Shorten header separators
cap10morgan Sep 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 67 additions & 11 deletions docs/technical-details/reference/analytics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,16 +102,72 @@ And a summary record looks like:
}
```

The following are general resource usage statistics that are tracked:

- `memory` - This includes RSS, heap, buffer and external data usage.
- `utilization` - How much of the time the worker was processing requests.
- `mqtt-connections` - The number of MQTT connections.
# Standard Analytics Metrics

While applications can define their own metrics, Harper provides a set of standard metrics that are tracked for all services:

## HTTP

The following metrics are tracked for all HTTP requests:

| `metric` | `path` | `method` | `type` | Unit | Description |
| ------------------ | ------------- | -------------- | ---------------------------------------------- | ------------ | ------------------------------------------------------- |
| `duration` | resource path | request method | `cache-hit` or `cache-miss` if a caching table | milliseconds | Duration of request handler |
| `duration` | route path | request method | fastify-route | milliseconds | |
| `duration` | operation | | operation | milliseconds | |
| `success` | resource path | request method | | % | |
| `success` | route path | request method | fastify-route | % | |
| `success` | operation | | operation | % | |
| `bytes-sent` | resource path | request method | | bytes | |
| `bytes-sent` | route path | request method | fastify-route | bytes | |
| `bytes-sent` | operation | | operation | bytes | |
| `transfer` | resource path | request method | operation | milliseconds | duration of transfer |
| `transfer` | route path | request method | fastify-route | milliseconds | duration of transfer |
| `transfer` | operation | | operation | milliseconds | duration of transfer |
| `socket-routed` | | | | % | percentage of sockets that could be immediately routed |
| `tls-handshake` | | | | milliseconds | |
| `tls-reused` | | | | % | percentage of TLS that reuses sessions |
| `cache-hit` | table name | | | % | The percentage of cache hits |
| `cache-resolution` | table name | | | milliseconds | The duration of resolving requests for uncached entries |

The following are metrics for real-time MQTT connections:
| `metric` | `path` | `method` | `type` | Unit | Description |
|---|---|---|---|---|---|
| `mqtt-connections` | | | | count | The number of open direct MQTT connections |
| `ws-connections` | | | | count | number of open WS connections|
| `connection` | `mqtt` | `connect` | | % | percentage of successful direct MQTT connections |
| `connection` | `mqtt` | `disconnect` | | % | percentage of explicit direct MQTT disconnects |
| `connection` | `ws` | `connect` | | % | percentage of successful WS connections |
| `connection` | `ws` | `disconnect` | | % | percentage of explicit WS disconnects |
| `bytes-sent` | topic | mqtt command | `mqtt` | bytes | The number of bytes sent for a given command and topic |

The following are metrics for replication:

| `metric` | `path` | `method` | `type` | Unit | Description |
| ---------------- | ------------- | ------------- | --------- | ----- | ----------------------------------------------------- |
| `bytes-sent` | node.database | `replication` | `egress` | bytes | The number of bytes sent for replication |
| `bytes-sent` | node.database | `replication` | `blob` | bytes | The number of bytes sent for replication of blobs |
| `bytes-received` | node.database | `replication` | `ingress` | bytes | The number of bytes received for replication |
| `bytes-received` | node.database | `replication` | `blob` | bytes | The number of bytes received for replication of blobs |

The following types of information is tracked for each HTTP request:

- `success` - How many requests returned a successful response (20x response code). TTFB - Time to first byte in the response to the client.
- `transfer` - Time to finish the transfer of the data to the client.
- `bytes-sent` - How many bytes of data were sent to the client.
The following are general resource usage statistics that are tracked:

Requests are categorized by operation name, for the operations API, by the resource (name) with the REST API, and by command for the MQTT interface.
| `metric` | primary attribute(s) | other attribute(s) | Unit | Description |
| ------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | The size of the database in bytes |
| `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage; including idle time, active time, task queue latency, RSS, heap, buffer and external memory usage |
| `resource-usage` | | | various | [See breakout below](#resource-usage) |
| `storage-volume` | `available`, `free`, `size` | `database` | bytes | The size of the storage volume in bytes |
| `table-size` | `size` | `database`, `table` | bytes | The size of the table in bytes |
| `utilization` | | | % | How much of the time the worker was processing requests |

<a id="resource-usage"></a>
`resource-usage` metrics are everything returned by [node:process.resourceUsage()](https://nodejs.org/api/process.html#processresourceusage)[^1] plus the following additional metrics:

| `metric` | Unit | Description |
| ---------------- | ---- | ----------------------------------------------------- |
| `time` | ms | Current time when metric was recorded (Unix time) |
| `period` | ms | Duration of the metric period |
| `cpuUtilization` | % | CPU utilization percentage (user and system combined) |

[^1]: The `userCPUTime` and `systemCPUTime` metrics are converted to milliseconds to match the other time-related metrics.
78 changes: 67 additions & 11 deletions versioned_docs/version-4.6/technical-details/reference/analytics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,16 +102,72 @@ And a summary record looks like:
}
```

The following are general resource usage statistics that are tracked:

- `memory` - This includes RSS, heap, buffer and external data usage.
- `utilization` - How much of the time the worker was processing requests.
- mqtt-connections - The number of MQTT connections.
# Standard Analytics Metrics

While applications can define their own metrics, Harper provides a set of standard metrics that are tracked for all services:

## HTTP

The following metrics are tracked for all HTTP requests:

| `metric` | `path` | `method` | `type` | Unit | Description |
| ------------------ | ------------- | -------------- | ---------------------------------------------- | ------------ | ------------------------------------------------------- |
| `duration` | resource path | request method | `cache-hit` or `cache-miss` if a caching table | milliseconds | Duration of request handler |
| `duration` | route path | request method | fastify-route | milliseconds | |
| `duration` | operation | | operation | milliseconds | |
| `success` | resource path | request method | | % | |
| `success` | route path | request method | fastify-route | % | |
| `success` | operation | | operation | % | |
| `bytes-sent` | resource path | request method | | bytes | |
| `bytes-sent` | route path | request method | fastify-route | bytes | |
| `bytes-sent` | operation | | operation | bytes | |
| `transfer` | resource path | request method | operation | milliseconds | duration of transfer |
| `transfer` | route path | request method | fastify-route | milliseconds | duration of transfer |
| `transfer` | operation | | operation | milliseconds | duration of transfer |
| `socket-routed` | | | | % | percentage of sockets that could be immediately routed |
| `tls-handshake` | | | | milliseconds | |
| `tls-reused` | | | | % | percentage of TLS that reuses sessions |
| `cache-hit` | table name | | | % | The percentage of cache hits |
| `cache-resolution` | table name | | | milliseconds | The duration of resolving requests for uncached entries |

The following are metrics for real-time MQTT connections:
| `metric` | `path` | `method` | `type` | Unit | Description |
|--------------------|---------------|----------------|--------------------------------------------|--------------------------------------------------------|---------------------------------------------------------|
| `mqtt-connections` | | | | count | The number of open direct MQTT connections |
| `ws-connections` | | | | count | number of open WS connections|
| `connection` | `mqtt` | `connect` | | % | percentage of successful direct MQTT connections |
| `connection` | `mqtt` | `disconnect` | | % | percentage of explicit direct MQTT disconnects |
| `connection` | `ws` | `connect` | | % | percentage of successful WS connections |
| `connection` | `ws` | `disconnect` | | % | percentage of explicit WS disconnects |
| `bytes-sent` | topic | mqtt command | `mqtt` | bytes | The number of bytes sent for a given command and topic |

The following are metrics for replication:

| `metric` | `path` | `method` | `type` | Unit | Description |
| ---------------- | ------------- | ------------- | --------- | ----- | ----------------------------------------------------- |
| `bytes-sent` | node.database | `replication` | `egress` | bytes | The number of bytes sent for replication |
| `bytes-sent` | node.database | `replication` | `blob` | bytes | The number of bytes sent for replication of blobs |
| `bytes-received` | node.database | `replication` | `ingress` | bytes | The number of bytes received for replication |
| `bytes-received` | node.database | `replication` | `blob` | bytes | The number of bytes received for replication of blobs |

The following types of information is tracked for each HTTP request:

- `success` - How many requests returned a successful response (20x response code). TTFB - Time to first byte in the response to the client.
- `transfer` - Time to finish the transfer of the data to the client.
- bytes-sent - How many bytes of data were sent to the client.
The following are general resource usage statistics that are tracked:

Requests are categorized by operation name, for the operations API, by the resource (name) with the REST API, and by command for the MQTT interface.
| `metric` | primary attribute(s) | other attribute(s) | Unit | Description |
| ------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | The size of the database in bytes |
| `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage; including idle time, active time, task queue latency, RSS, heap, buffer and external memory usage |
| `resource-usage` | | | various | [See breakout below](#resource-usage) |
| `storage-volume` | `available`, `free`, `size` | `database` | bytes | The size of the storage volume in bytes |
| `table-size` | `size` | `database`, `table` | bytes | The size of the table in bytes |
| `utilization` | | | % | How much of the time the worker was processing requests |

<a id="resource-usage"></a>
`resource-usage` metrics are everything returned by [node:process.resourceUsage()](https://nodejs.org/api/process.html#processresourceusage)[^1] plus the following additional metrics:

| `metric` | Unit | Description |
| ---------------- | ---- | ----------------------------------------------------- |
| `time` | ms | Current time when metric was recorded (Unix time) |
| `period` | ms | Duration of the metric period |
| `cpuUtilization` | % | CPU utilization percentage (user and system combined) |

[^1]: The `userCPUTime` and `systemCPUTime` metrics are converted to milliseconds to match the other time-related metrics.