Merge #71

71: docs: Add monitoring section to node onboarding docs r=jwolski2 a=jwolski2

<!--
Thank you for your Pull Request. Please provide a description above and review
the requirements below.

Bug fixes and new features should include tests.

Contributors guide: https://github.com/NillionNetwork/nillion/blob/master/CONTRIBUTING.md

-->

## Motivation

<!--
Explain the context and why you're making that change. What is the problem
you're trying to solve? In some cases there is not a problem and this can be
thought of as being the motivation for your change.
-->

Node operators have asked for advice/recommendations on which nilvm metrics to monitor and set alerts on.

## Solution

<!--
Summarize the solution and provide any necessary context needed to understand
the code change.
-->

This commit adds a "Monitor Your Node" section to the onboarding documentation which provides the advice, taken from our (internal) Prometheus / Alert Manager rules.


Fixes #
Design discussion issue (if applicable) #

## Merge requirement checklist

* [ ] [CONTRIBUTING](https://github.com/NillionNetwork/nillion/blob/main/CONTRIBUTING.md) guidelines followed
* [ ] Unit tests added/updated (if applicable)
* [ ] Breaking change analysis completed (if applicable). "Will this change require all network cluster operators to update? Does it break public APIs?"
* [ ] For new features or breaking changes, created a documentation issue in [nillion-docs](https://github.com/NillionNetwork/nillion-docs/issues/new/choose)


Co-authored-by: Jeff Wolski <[email protected]>

Author: nilogy-bors[bot]

docs/node_onboarding/README.md

@@ -115,6 +115,32 @@ docker run -e CONFIG_PATH=/etc/nillion/node.yaml public.ecr.aws/k5d9x2g2/nilvm:$
 
 All nodes present in `cluster.members` must be available before network functions can be validated.
 
+## Monitor Your Node
+
+Set alerts on the following metrics and Prometheus expressions; it is recommended to use a `5m` duration for alerts.
+
+### Error Rate Metrics
+
+| Metric                            | Prometheus Expression                                                            | Description                             |
+| --------------------------------- | -------------------------------------------------------------------------------- | --------------------------------------- |
+| High blob operation error rate    | `sum by (operation) (increase(blob_operation_errors_total[5m])) > 0`             | Blob operation error rate is above 0    |
+| High error rate                   | `sum(rate(grpc_request_duration_seconds_count{status_code="Internal"}[1m])) > 0` | gRPC error rate is above 0              |
+| High token price query error rate | `sum by (env) (increase(token_price_errors_total[5m])) > 0`                      | Token price query error rate is above 0 |
+
+### Latency Metrics
+
+| Metric                         | Prometheus Expression                                                                                    | Description                                           |
+| ------------------------------ | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
+| High blob latency              | `histogram_quantile(0.99, sum(rate(blob_operation_duration_seconds_bucket[5m])) by (operation, le)) > 1` | 99th percentile blob operation latency is above 1s    |
+| High latency                   | `histogram_quantile(0.99, sum(rate(grpc_request_duration_seconds_bucket[1m])) by (le)) > 5`              | 99th percentile gRPC latency is above 5s              |
+| High token price query latency | `histogram_quantile(0.99, sum(rate(token_price_duration_seconds_bucket[5m])) by (le)) > 5`               | 99th percentile token price query latency is above 5s |
+
+### Other Metrics
+
+| Metric                     | Prometheus Expression                                                                                                   | Description                                   |
+| -------------------------- | ----------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- |
+| Low preprocessing elements | `max(preprocessing_offsets{offset="latest"} - on(element) preprocessing_offsets{offset="committed"}) by (element) < 32` | Amount of preprocessing elements is below 32  |
+
 [nillion-sdk]: https://docs.nillion.com/nillion-sdk-and-tools
 [node-yaml-mainnet-1]: ./networks/nilvm-mainnet-1.yaml
 [node-yaml-testnet-1]: ./networks/nilvm-testnet-1.yaml