Skip to content

Commit 7573d37

Browse files
authored
Outbound Webhooks (matrix-org#945)
* Initial support for outbound webhooks. * Refactor outbound into it's own connection type. * Add support for media / encrypted media. * Ensure we configure a sensible User Agent * Add a test for outbound webhooks * Checkpoint for feature completeness. * Lint tidy * Finish up media tests. * changelog * Add outbound documentation * update default config * fix tests
1 parent 60ccc04 commit 7573d37

File tree

18 files changed

+840
-42
lines changed

18 files changed

+840
-42
lines changed

changelog.d/945.feature

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Add support for new connection type "Outgoing Webhooks". This feature allows you to send outgoing HTTP requests to other services
2+
when a message appears in a Matrix room. See [the documentation](https://matrix-org.github.io/matrix-hookshot/latest/setup/webhooks.html)
3+
for help with this feature.

config.sample.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,7 @@ listeners:
100100
# #'allowJsTransformationFunctions' will allow users to write short transformation snippets in code, and thus is unsafe in untrusted environments
101101

102102
# enabled: false
103+
# outbound: false
103104
# enableHttpGet: false
104105
# urlPrefix: https://example.com/webhook/
105106
# userIdPrefix: _webhooks_

docs/setup/webhooks.md

Lines changed: 50 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Webhooks
22

3-
Hookshot supports generic webhook support so that services can send messages into Matrix rooms without being aware of the Matrix protocol. This works
4-
by having services hit a unique URL that then transforms a HTTP payload into a Matrix message.
3+
Hookshot supports two kinds of webhooks, inbound (previously known as Generic Webhooks) and outbound.
4+
55

66
## Configuration
77

@@ -10,13 +10,19 @@ You will need to add the following configuration to the config file.
1010
```yaml
1111
generic:
1212
enabled: true
13+
outbound: true # For outbound webhook support
1314
urlPrefix: https://example.com/mywebhookspath/
1415
allowJsTransformationFunctions: false
1516
waitForComplete: false
1617
enableHttpGet: false
1718
# userIdPrefix: webhook_
1819
```
1920

21+
## Inbound Webhooks
22+
23+
Hookshot supports generic webhook support so that services can send messages into Matrix rooms without being aware of the Matrix protocol. This works
24+
by having services hit a unique URL that then transforms a HTTP payload into a Matrix message.
25+
2026
<section class="notice">
2127
Previous versions of the bridge listened for requests on `/` rather than `/webhook`. While this behaviour will continue to work,
2228
administators are advised to use `/webhook`.
@@ -50,15 +56,15 @@ namespaces:
5056
exclusive: true
5157
```
5258
53-
## Adding a webhook
59+
### Adding a webhook
5460
5561
To add a webhook to your room:
5662
- Invite the bot user to the room.
5763
- Make sure the bot able to send state events (usually the Moderator power level in clients)
5864
- Say `!hookshot webhook example` where `example` is a name for your hook.
5965
- The bot will respond with the webhook URL to be sent to services.
6066

61-
## Webhook Handling
67+
### Webhook Handling
6268

6369
Hookshot handles `POST` and `PUT` HTTP requests by default.
6470

@@ -76,7 +82,7 @@ If the body *also* contains a `username` key, then the message will be prepended
7682
If the body does NOT contain a `text` field, the full payload will be sent to the room. This can be adapted into a message by creating a **JavaScript transformation function**.
7783

7884

79-
### Payload formats
85+
#### Payload formats
8086

8187
If the request is a `POST`/`PUT`, the body of the request will be decoded and stored inside the event. Currently, Hookshot supports:
8288

@@ -88,7 +94,7 @@ If the request is a `POST`/`PUT`, the body of the request will be decoded and st
8894
Decoding is done in the order given above. E.g. `text/xml` would be parsed as XML. Any formats not described above are not
8995
decoded.
9096

91-
### GET requests
97+
#### GET requests
9298

9399
In previous versions of hookshot, it would also handle the `GET` HTTP method. This was disabled due to concerns that it was too easy for the webhook to be
94100
inadvertently triggered by URL preview features in clients and servers. If you still need this functionality, you can enable it in the config.
@@ -102,7 +108,7 @@ to a string representation of that value. This change is <strong>not applied</st
102108
variable, so it will contain proper float values.
103109
</section>
104110

105-
### Wait for complete
111+
#### Wait for complete
106112

107113
It is possible to choose whether a webhook response should be instant, or after hookshot has handled the message. The reason
108114
for this is that some services expect a quick response time (like Slack) whereas others will wait for the request to complete. You
@@ -111,7 +117,7 @@ can specify this either globally in your config, or on the widget with `waitForC
111117
If you make use of the `webhookResponse` feature, you will need to enable `waitForComplete` as otherwise hookshot will
112118
immeditately respond with it's default response values.
113119

114-
## JavaScript Transformations
120+
### JavaScript Transformations
115121

116122
<section class="notice">
117123
Although every effort has been made to securely sandbox scripts, running untrusted code from users is always risky. Ensure safe permissions
@@ -130,7 +136,7 @@ Please seek out documentation from your client on how to achieve this.
130136

131137
The script string should be set within the state event under the `transformationFunction` key.
132138

133-
### Script API
139+
#### Script API
134140

135141
Transformation scripts have a versioned API. You can check the version of the API that the hookshot instance supports
136142
at runtime by checking the `HookshotApiVersion` variable. If the variable is undefined, it should be considered `v1`.
@@ -141,7 +147,7 @@ Scripts are executed synchronously and expect the `result` variable to be set.
141147
If the script contains errors or is otherwise unable to work, the bridge will send an error to the room. You can check the logs of the bridge
142148
for a more precise error.
143149

144-
### V2 API
150+
#### V2 API
145151

146152
The `v2` api expects an object to be returned from the `result` variable.
147153

@@ -176,7 +182,7 @@ if (data.counter === undefined) {
176182
```
177183

178184

179-
### V1 API
185+
#### V1 API
180186

181187
The v1 API expects `result` to be a string. The string will be automatically interpreted as Markdown and transformed into HTML. All webhook messages
182188
will be prefixed with `Received webhook:`. If `result` is falsey (undefined, false or null) then the message will be `No content`.
@@ -192,3 +198,36 @@ if (data.counter > data.maxValue) {
192198
result = `*Everything is fine*, the counter is under by ${data.maxValue - data.counter}`
193199
}
194200
```
201+
202+
## Outbound webhooks
203+
204+
You can also configure Hookshot to send outgoing requests to other services when a message appears
205+
on Matrix. To do so, you need to configure hookshot to enable outgoing messages with:
206+
207+
```yaml
208+
generic:
209+
outbound: true
210+
```
211+
212+
### Request format
213+
214+
Requests can be sent to any service that accepts HTTP requests. You may configure Hookshot to either use the HTTP `PUT` (default)
215+
or `POST` methods.
216+
217+
Each request will contain 3 headers which you may use to authenticate and direct traffic:
218+
219+
- 'X-Matrix-Hookshot-EventId' contains the event's ID.
220+
- 'X-Matrix-Hookshot-RoomId' contains the room ID where the message was sent.
221+
- 'X-Matrix-Hookshot-Token' is the unique authentication token provided when you created the webhook. Use this
222+
to verify that the message came from Hookshot.
223+
224+
The payloads are formatted as `multipart/form-data`.
225+
226+
The first file contains the event JSON data, proviced as the `event` file. This is a raw representation of the Matrix event data. If the
227+
event was encrypted, this will be the **decrypted** body.
228+
229+
If any media is linked to in the event, then a second file will be present named `media` which will contain the media referenced in
230+
the event.
231+
232+
All events that occur in the room will be sent to the outbound URL, so be careful to ensure your remote service can filter the
233+
traffic appropriately (e.g. check the `type` in the event JSON)

package.json

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -48,8 +48,8 @@
4848
"@octokit/rest": "^20.0.2",
4949
"@octokit/webhooks": "^12.0.10",
5050
"@sentry/node": "^7.52.1",
51-
"@vector-im/compound-design-tokens": "^0.1.0",
52-
"@vector-im/compound-web": "^0.9.4",
51+
"@vector-im/compound-design-tokens": "^1.3.0",
52+
"@vector-im/compound-web": "^4.8.0",
5353
"ajv": "^8.11.0",
5454
"axios": "^1.6.3",
5555
"cors": "^2.8.5",
@@ -86,6 +86,7 @@
8686
"@rollup/plugin-alias": "^5.1.0",
8787
"@tsconfig/node18": "^18.2.2",
8888
"@types/ajv": "^1.0.0",
89+
"@types/busboy": "^1.5.4",
8990
"@types/chai": "^4.2.22",
9091
"@types/cors": "^2.8.12",
9192
"@types/express": "^4.17.14",
@@ -100,6 +101,7 @@
100101
"@typescript-eslint/eslint-plugin": "^6.17.0",
101102
"@typescript-eslint/parser": "^6.17.0",
102103
"@uiw/react-codemirror": "^4.12.3",
104+
"busboy": "^1.6.0",
103105
"chai": "^4.3.4",
104106
"eslint": "^8.49.0",
105107
"eslint-config-preact": "^1.3.0",
@@ -116,5 +118,6 @@
116118
"ts-node": "^10.9.1",
117119
"typescript": "^5.3.3",
118120
"vite": "^5.0.13"
119-
}
121+
},
122+
"packageManager": "[email protected]+sha512.a6b2f7906b721bba3d67d4aff083df04dad64c399707841b7acf00f6b133b7ac24255f2652fa22ae3534329dc6180534e98d17432037ff6fd140556e2bb3137e"
120123
}

spec/util/fixtures.ts

Lines changed: 1 addition & 0 deletions
Large diffs are not rendered by default.

spec/webhooks.spec.ts

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
import { E2ESetupTestTimeout, E2ETestEnv, E2ETestMatrixClient } from "./util/e2e-test";
2+
import { describe, it, beforeEach, afterEach } from "@jest/globals";
3+
import { OutboundHookConnection } from "../src/Connections";
4+
import { TextualMessageEventContent } from "matrix-bot-sdk";
5+
import { IncomingHttpHeaders, createServer } from "http";
6+
import busboy, { FileInfo } from "busboy";
7+
import { TEST_FILE } from "./util/fixtures";
8+
9+
async function createOutboundConnection(user: E2ETestMatrixClient, botMxid: string, roomId: string) {
10+
const join = user.waitForRoomJoin({ sender: botMxid, roomId });
11+
const connectionEvent = user.waitForRoomEvent({
12+
eventType: OutboundHookConnection.CanonicalEventType,
13+
stateKey: 'test',
14+
sender: botMxid
15+
});
16+
await user.inviteUser(botMxid, roomId);
17+
await user.setUserPowerLevel(botMxid, roomId, 50);
18+
await join;
19+
20+
// Note: Here we create the DM proactively so this works across multiple
21+
// tests.
22+
// Get the DM room so we can get the token.
23+
const dmRoomId = await user.dms.getOrCreateDm(botMxid);
24+
25+
await user.sendText(roomId, '!hookshot outbound-hook test http://localhost:8111/test-path');
26+
// Test the contents of this.
27+
await connectionEvent;
28+
29+
const msgPromise = user.waitForRoomEvent({ sender: botMxid, eventType: "m.room.message", roomId: dmRoomId });
30+
const { data: msgData } = await msgPromise;
31+
32+
const [_match, token ] = /<code>(.+)<\/code>/.exec((msgData.content as unknown as TextualMessageEventContent).formatted_body ?? "") ?? [];
33+
return token;
34+
}
35+
36+
/**
37+
*
38+
* @returns
39+
*/
40+
function awaitOutboundWebhook() {
41+
return new Promise<{headers: IncomingHttpHeaders, files: {name: string, file: Buffer, info: FileInfo}[]}>((resolve, reject) => {
42+
const server = createServer((req, res) => {
43+
const bb = busboy({headers: req.headers});
44+
const files: {name: string, file: Buffer, info: FileInfo}[] = [];
45+
bb.on('file', (name, stream, info) => {
46+
const buffers: Buffer[] = [];
47+
stream.on('data', d => {
48+
buffers.push(d)
49+
});
50+
stream.once('close', () => {
51+
files.push({name, info, file: Buffer.concat(buffers)})
52+
});
53+
});
54+
55+
bb.once('close', () => {
56+
res.writeHead(200, { 'Content-Type': 'text/plain' });
57+
res.end('OK');
58+
resolve({
59+
headers: req.headers,
60+
files,
61+
});
62+
clearTimeout(timer);
63+
server.close();
64+
});
65+
66+
req.pipe(bb);
67+
});
68+
server.listen(8111);
69+
let timer: NodeJS.Timeout;
70+
timer = setTimeout(() => {
71+
reject(new Error("Request did not arrive"));
72+
server.close();
73+
}, 10000);
74+
75+
});
76+
}
77+
78+
describe('OutboundHooks', () => {
79+
let testEnv: E2ETestEnv;
80+
81+
beforeAll(async () => {
82+
const webhooksPort = 9500 + E2ETestEnv.workerId;
83+
testEnv = await E2ETestEnv.createTestEnv({
84+
matrixLocalparts: ['user'],
85+
config: {
86+
generic: {
87+
enabled: true,
88+
outbound: true,
89+
urlPrefix: `http://localhost:${webhooksPort}`
90+
},
91+
listeners: [{
92+
port: webhooksPort,
93+
bindAddress: '0.0.0.0',
94+
// Bind to the SAME listener to ensure we don't have conflicts.
95+
resources: ['webhooks'],
96+
}],
97+
}
98+
});
99+
await testEnv.setUp();
100+
}, E2ESetupTestTimeout);
101+
102+
afterAll(() => {
103+
return testEnv?.tearDown();
104+
});
105+
106+
it('should be able to create a new webhook and push an event.', async () => {
107+
const user = testEnv.getUser('user');
108+
const roomId = await user.createRoom({ name: 'My Test Webhooks room'});
109+
const token = await createOutboundConnection(user, testEnv.botMxid, roomId);
110+
const gotWebhookRequest = awaitOutboundWebhook();
111+
112+
const eventId = await user.sendText(roomId, 'hello!');
113+
const { headers, files } = await gotWebhookRequest;
114+
expect(headers['x-matrix-hookshot-roomid']).toEqual(roomId);
115+
expect(headers['x-matrix-hookshot-eventid']).toEqual(eventId);
116+
expect(headers['x-matrix-hookshot-token']).toEqual(token);
117+
118+
// And check the JSON payload
119+
const [event, media] = files;
120+
expect(event.name).toEqual('event');
121+
expect(event.info.mimeType).toEqual('application/json');
122+
expect(event.info.filename).toEqual('event_data.json');
123+
const eventJson = JSON.parse(event.file.toString('utf-8'));
124+
125+
// Check that the content looks sane.
126+
expect(eventJson.room_id).toEqual(roomId);
127+
expect(eventJson.event_id).toEqual(eventId);
128+
expect(eventJson.sender).toEqual(await user.getUserId());
129+
expect(eventJson.content.body).toEqual('hello!');
130+
131+
// No media should be present.
132+
expect(media).toBeUndefined();
133+
});
134+
135+
it('should be able to create a new webhook and push a media attachment.', async () => {
136+
const user = testEnv.getUser('user');
137+
const roomId = await user.createRoom({ name: 'My Test Webhooks room'});
138+
await createOutboundConnection(user, testEnv.botMxid, roomId);
139+
const gotWebhookRequest = awaitOutboundWebhook();
140+
141+
const mxcUrl = await user.uploadContent(TEST_FILE, 'image/svg+xml', "matrix.svg");
142+
await user.sendMessage(roomId, {
143+
url: mxcUrl,
144+
msgtype: "m.file",
145+
body: "matrix.svg",
146+
})
147+
const { files } = await gotWebhookRequest;
148+
const [event, media] = files;
149+
expect(event.info.mimeType).toEqual('application/json');
150+
expect(event.info.filename).toEqual('event_data.json');
151+
const eventJson = JSON.parse(event.file.toString('utf-8'));
152+
expect(eventJson.content.body).toEqual('matrix.svg');
153+
154+
155+
expect(media.info.mimeType).toEqual('image/svg+xml');
156+
expect(media.info.filename).toEqual('matrix.svg');
157+
expect(media.file).toEqual(TEST_FILE);
158+
});
159+
160+
// TODO: This requires us to support Redis in test conditions, as encryption is not possible
161+
// in hookshot without it at the moment.
162+
163+
// it.only('should be able to create a new webhook and push an encrypted media attachment.', async () => {
164+
// const user = testEnv.getUser('user');
165+
// const roomId = await user.createRoom({ name: 'My Test Webhooks room', initial_state: [{
166+
// content: {
167+
// "algorithm": "m.megolm.v1.aes-sha2"
168+
// },
169+
// state_key: "",
170+
// type: "m.room.encryption"
171+
// }]});
172+
// await createOutboundConnection(user, testEnv.botMxid, roomId);
173+
// const gotWebhookRequest = awaitOutboundWebhook();
174+
175+
// const encrypted = await user.crypto.encryptMedia(Buffer.from(TEST_FILE));
176+
// const mxc = await user.uploadContent(TEST_FILE);
177+
// await user.sendMessage(roomId, {
178+
// msgtype: "m.image",
179+
// body: "matrix.svg",
180+
// info: {
181+
// mimetype: "image/svg+xml",
182+
// },
183+
// file: {
184+
// url: mxc,
185+
// ...encrypted.file,
186+
// },
187+
// });
188+
189+
// const { headers, files } = await gotWebhookRequest;
190+
// const [event, media] = files;
191+
// expect(event.info.mimeType).toEqual('application/json');
192+
// expect(event.info.filename).toEqual('event_data.json');
193+
// const eventJson = JSON.parse(event.file.toString('utf-8'));
194+
// expect(eventJson.content.body).toEqual('matrix.svg');
195+
196+
197+
// expect(media.info.mimeType).toEqual('image/svg+xml');
198+
// expect(media.info.filename).toEqual('matrix.svg');
199+
// expect(media.file).toEqual(TEST_FILE);
200+
// });
201+
});

0 commit comments

Comments
 (0)