Skip to content

Commit dbe7658

Browse files
bruce-rileybruce-riley
authored
docs: Query proxy server operations manual (wormhole-foundation#3916)
* Doc: Query proxy server operations manual * Review rework
1 parent 21e2c87 commit dbe7658

File tree

2 files changed

+202
-0
lines changed

2 files changed

+202
-0
lines changed

‎cspell-custom-words.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,7 @@ optin
120120
Optin
121121
parachain
122122
pdas
123+
permissioned
123124
permissionless
124125
permissionlessly
125126
Polkachu

‎docs/query_proxy.md‎

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
# Operating the Wormhole Queries Proxy Server
2+
3+
The Wormhole queries proxy server (sometimes referred to as the CCQ proxy) is a server that listens on a
4+
REST endpoint for Wormhole query requests. It validates those requests and forwards them to the guardian
5+
CCQ P2P network for processing by the guardians. It then accumulates the responses from the guardians,
6+
verifies quorum and forwards the response to the client.
7+
8+
## Building the Proxy Server
9+
10+
The proxy server runs as another instance of the `guardiand` process, similar to the spy. It is built exactly
11+
the same as the spy, and requires the same dependencies. Please see the [Operations Guide](operations.md#building-guardiand) for
12+
details on how to build `guardiand`.
13+
14+
## Deploying the Proxy Server
15+
16+
The proxy server can be deployed just like the spy, including potentially running in a container. Note that it
17+
requires a public IP address to listen for REST requests, and it needs to be able to reach the guardian P2P network.
18+
19+
The proxy is not particularly resource intensive, so should run successfully on a reasonable size VM.
20+
21+
## Configuring the Proxy Server
22+
23+
There are two main parts to configuring the proxy server. The first is setting up the command line arguments,
24+
which generally will not change after initial setup. The second part of the configuration is the permissions file,
25+
which will change as the requirements of integrators change.
26+
27+
### Proxy Server Command Line Arguments
28+
29+
The following is a sample command line for running the proxy server in mainnet.
30+
31+
```shell
32+
wormhole $build/bin/guardiand query-server \
33+
--env "mainnet" \
34+
--nodeKey /home/ccq/data/ccq_server.nodeKey \
35+
--permFile "/home/ccq/data/ccq_server.perms.json" \
36+
--signerKey "/home/ccq/data/ccq_server.signerKey" \
37+
--listenAddr "[::]:8080" \
38+
--ethRPC https://eth.drpc.org \
39+
--ethContract "0x98f3c9e6E3fAce36bAAd05FE09d375Ef1464288B" \
40+
--logLevel=info \
41+
--telemetryLokiURL $LOKI_URL \
42+
--telemetryNodeName "Mainnet CCQ server 1" \
43+
--promRemoteURL $PROM_URL
44+
```
45+
46+
- The `env` can be mainnet, testnet or devnet.
47+
- The `nodeKey` should point to the file containing the P2P key. The first time the proxy runs, if the
48+
file does not exist, it will be created. You can look in the proxy server logs to get the generated key.
49+
- The `permFile` is the JSON permissions file, which is documented below.
50+
- The `signerKey` should point to an armored file containing a key that will be used to sign requests received
51+
from integrators who are configured to support auto signing and opt not to sign a request. Please see below
52+
for how to generate this file.
53+
- The `listenAddr` specifies the port on which the proxy listens for REST requests.
54+
- The `ethRPC` and `ethContract` are used to read the wormhole guardian set on start up. The address
55+
above is for mainnet. If you are running in testnet, you should point to Holesky and use `0xa10f2eF61dE1f19f586ab8B6F2EbA89bACE63F7a`.
56+
(You can confirm these addresses [here](https://docs.wormhole.com/wormhole/reference/constants#contract-addresses).)
57+
Note that using a public endpoint should be fine, since the proxy only does a single read of the guardian set.
58+
- The `telemetryLokiURL`, `telemetryNodeName` and `promRemoteURL` are used for telemetry purposes and
59+
the values will be provided by Wormhole Foundation personnel if appropriate.
60+
61+
#### Creating the Signing Key File
62+
63+
Do the following to create the signing key file. Note that the `block-type` must exactly match what is specified below,
64+
but the `desc` can be anything you want.
65+
66+
```shell
67+
wormhole$ build/bin/guardiand keygen --desc "Your CCQ proxy server" --block-type "CCQ SERVER SIGNING KEY" /home/ccq/data/ccq_server.signerKey
68+
```
69+
70+
### Guardian Support for a New Proxy Server
71+
72+
The Queries P2P network is permissioned. The guardians will ignore P2P traffic from sources that are not in their configuration.
73+
Additionally, they will only honor query requests signed using a key in their configured list. Before you can begin publishing
74+
requests from your proxy, you must get a quorum (preferably all) of the guardians to add your values for the following to their
75+
configurations:
76+
77+
- P2P key (the value from `nodeKey` file, logged on proxy start up).
78+
- The public key associated with the signing key. See the `signerKey` file.
79+
80+
Please work with foundation personnel to get your proxy server added to the guardian configurations.
81+
82+
### Permissions Configuration
83+
84+
The file specified by the `permFile` parameter contains JSON that defines the set of allowed queries users, along with the
85+
sets of requests they are allowed to make.
86+
87+
#### File Format
88+
89+
The simplest file would look something like this
90+
91+
```json
92+
{
93+
"permissions": [
94+
{
95+
"userName": "Monitor",
96+
"apiKey": "insert_generated_api_key_here",
97+
"allowUnsigned": true,
98+
"allowedCalls": [
99+
{
100+
"ethCall": {
101+
"note:": "Name of WETH on Ethereum",
102+
"chain": 2,
103+
"contractAddress": "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2",
104+
"call": "0x06fdde03"
105+
}
106+
}
107+
]
108+
}
109+
}
110+
```
111+
112+
This creates a single user called "Monitor", who will use the specified API key (more on API keys below).
113+
This user is allowed to submit unsigned requests (which will be signed using the configured signing key).
114+
115+
This sample user is only allowed to make a single `ethCall` request on Ethereum (Wormhole chain ID 2),
116+
which allows them to call the `name` method on the contract that resides at `0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2`.
117+
The `call` parameter is the first four bytes of the hash of the ABI encoded function call to be allowed.
118+
119+
A given user can have any number of allowed calls (at least one), but they can only make calls that are configured here.
120+
121+
#### Supported Call Types
122+
123+
The proxy server supports all of the query types supported by the Wormhole Queries protocol. For details on those calls,
124+
please see the [Wormhole Queries Whitepaper](../whitepapers/0013_ccq.md).
125+
126+
The following are the EVM call types, all of which require the `chain`, `contractAddress` and `call` arguments.
127+
128+
- `ethCall`
129+
- `ethCallByTimestamp`
130+
- `ethCallWithFinality`
131+
132+
The following are the Solana call types. Both require the `chain` parameter plus the extra parameter listed below.
133+
134+
- `solAccount`, requires the `account` parameter.
135+
- `solPDA`, requires the `programAddress` parameter.
136+
137+
The Solana account and and program address can be expressed as either a 32 byte hex string starting with "0x" or as a base 58 value.
138+
139+
#### Creating New API Keys
140+
141+
Each user must have an API key. These keys only have meaning to the proxy server. They are not passed to the guardians.
142+
The proxy requires that a key be present in each query request, and that the specified key exists in the permissions file.
143+
Beyond that, the API keys have no special meaning. They can be generated using a site like [this](https://www.uuidgenerator.net/version4).
144+
145+
#### Updating the Permissions File
146+
147+
The proxy server monitors the permissions file for changes. Whenever a change is detected, it reads the file, validates it, and if
148+
it passes validation, switches to the new version. Care should be taken when editing the file while the proxy server is running, because
149+
as soon as you save the file, the changes will be picked up (whether they are logically complete or not).
150+
151+
## Telemetry
152+
153+
The proxy server provides two types of telemetry data, logs and metrics.
154+
155+
### Logging
156+
157+
The proxy server uses the same logging mechanism as the guardian. It will write to a local file, but can also be configured to
158+
publish logs to Grafana using the Loki protocol. If you will be running your proxy server in mainnet, you should contact foundation
159+
personnel about getting a Grafana ID to be used for logging and use it to set the `--telemetryLokiURL` command line argument.
160+
161+
If you set the log level to `info`, the proxy server logs information on all incoming requests and output bound responses. This can
162+
be helpful for determining when requests reach quorum, but may be too chatty as the level of queries traffic grows. If that is the
163+
case, you can set the log level to `warn`.
164+
165+
### Metrics
166+
167+
The proxy server uses Prometheus to track various activity and can publish them to Grafana. If you will be running your proxy server in mainnet,
168+
you should contact foundation personnel about getting a Grafana ID and use it to set the `--promRemoteURL` command line argument.
169+
170+
For the set of available metrics, see [here](../node/cmd/ccq/metrics.go).
171+
172+
## Troubleshooting
173+
174+
### P2P Health
175+
176+
If you think you are having trouble with your access to the P2P network, you can add `--monitorPeers` to the command line arguments,
177+
which will cause the proxy server to periodically check its connectivity to the P2P bootstrap peers, and attempt to reconnect if necessary.
178+
179+
### Invalid Requests
180+
181+
If the proxy server determines that a request is invalid, it does the following:
182+
183+
- Logs an error message using the user name (not the API Key).
184+
- Increments the appropriate Prometheus metric.
185+
- Sends a failure response to the user.
186+
187+
Note that if the proxy server thinks a request is valid, but the guardians do not, the guardians silently drop the request, so it will look
188+
like a timeout. This is to avoid a denial of service attack on the guardians. This can happen if the proxy server is not properly permissioned
189+
on the guardians.
190+
191+
### Logging Request Detail.
192+
193+
If a given integrator is reporting problems with their queries, you may find it useful to add the following to their permissions config
194+
(at the same level as the API Key, etc).
195+
196+
```json
197+
"logResponses": true,
198+
```
199+
200+
This will cause the proxy server to log every response received for that user, along with the number of responses and how many are
201+
still needed to meet quorum.

0 commit comments

Comments
 (0)