Skip to content

Commit 50f80f2

Browse files
jesopoGnuxie
andauthored
manymjolnir appservice (#364)
Mjolnir can now be run as an application service, meaning it will host multiple independent mjolnirs that can be requested by users. If the user is on the same homeserver as the appservice is deployed on, then they can provision a mjolnir via a widget https://github.com/matrix-org/mjolnir-widget. Otherwise they can invite the appservice bot to a room they want to protect. This will create them a mjolnir, a management room and a policy list. The appservice shares the same docker image as the bot, but is started slightly differently by specifying "appservice" as the first argument to docker run (this s managed by `mjolnir-entrypoint.sh`. We could have used another Dockerfile for the appservice, extending the existing one but we decided not to because there would have been lots of fiddling around the entrypoint and logistics involved around adding a tag for it via github actions. Not to mention that this would be duplicating the image just to run it with a different binary. A list of followup issues can be found here https://github.com/issues?q=is%3Aopen+is%3Aissue+author%3AGnuxie+archived%3Afalse+label%3AA-Appservice. Somewhat relevant and squashed commit messages(regrettably squashing because frankly these won't make sense in isolation): * draft widget backend * add `managementRoomId` to `provisionNewMjolnir` * remove ratelimits from appservice mjolnirs * add /join endpoint to api backend * tighter guard around room type in PolicyList matrix-bot-sdk imporved the types for this * enable esModuleInterop * launch and use postgres in a container whilst using mx-tester * limited access control policy list used for access control * Redesign initialization API of many mjolnir. It's much harder to forget to initialize the components now that you have to in order to construct them in the first place. * Ammend config not to clash with existing CI this means that the appsrvice bot is now called 'mjolnir-bot' by default which was easier than going through old code base and renaming * Change entrypoint in Dockerfile so that we can start the appservice. We could have used another Dockerfile for the appservice, extending the exising one but we decided not to because there would have been lots of fiddling around the entrypoint and logistics involved around adding a tag for it via github actions. Not to mention that this would be duplicating the image just to run it with a different binary. This solution is much simpler, backwards compatible, and conscious about the future. Co-authored-by: gnuxie <[email protected]>
1 parent 81cd91c commit 50f80f2

24 files changed

+2438
-443
lines changed

.github/workflows/mjolnir.yml

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,3 +56,25 @@ jobs:
5656
run: RUST_LOG=debug,hyper=info,rusttls=info mx-tester run
5757
- name: Cleanup
5858
run: mx-tester down
59+
appservice-integration:
60+
name: Application Service Integration tests
61+
runs-on: ubuntu-latest
62+
timeout-minutes: 30
63+
steps:
64+
- uses: actions/checkout@v3
65+
- uses: actions/setup-node@v3
66+
with:
67+
node-version: '16'
68+
- name: Fetch and build mx-tester (cached across runs)
69+
uses: baptiste0928/cargo-install@v1
70+
with:
71+
crate: mx-tester
72+
version: "0.3.3"
73+
- name: Setup image
74+
run: RUST_LOG=debug,hyper=info,rusttls=info mx-tester build up
75+
- name: Setup dependencies
76+
run: yarn install
77+
- name: Run tests
78+
run: yarn test:appservice:integration
79+
- name: Cleanup
80+
run: mx-tester down

Dockerfile

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,18 @@
1-
FROM node:16-alpine
1+
# We can't use alpine anymore because crypto has rust deps.
2+
FROM node:16-slim
23
COPY . /tmp/src
34
RUN cd /tmp/src \
45
&& yarn install \
56
&& yarn build \
67
&& mv lib/ /mjolnir/ \
78
&& mv node_modules / \
9+
&& mv mjolnir-entrypoint.sh / \
810
&& cd / \
911
&& rm -rf /tmp/*
1012

1113
ENV NODE_ENV=production
1214
ENV NODE_CONFIG_DIR=/data/config
1315

14-
CMD node /mjolnir/index.js
16+
CMD ["bot"]
17+
ENTRYPOINT ["./mjolnir-entrypoint.sh"]
1518
VOLUME ["/data"]

docs/appservice.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
Mjolnir can be run as an appservice, allowing users you trust or on your homeserver to run their own Mjolnir without hosting anything themselves.
2+
This module is currently alpha quality and is subject to rapid changes,
3+
it is not recommended currently and support will be limited.
4+
5+
# Prerequisites
6+
7+
This guide assumes you will be using Docker and that you are able to provide a postgres database for Mjolnir to connect to in application service mode.
8+
9+
# Setup
10+
11+
1. Create a new Matrix room that will act as a policy list for who can use the appservice.
12+
FIXME: Currently required to be aliased.
13+
FIXME: Should really be created and managed by the admin room, but waiting for command refactor before doing that.
14+
15+
2. Decide on a spare local TCP port number to use that will listen for messages from the matrix homeserver. Take care to configure firewalls appropriately. We will call this port `$MATRIX_PORT` in the remaining instructions.
16+
17+
3. Create a `config/config.appservice.yaml` file that can be copied from the example in `src/appservice/config/config.example.yaml`. Your config file needs to be accessible to the docker container later on. To do this you could create a directory called `mjolnir-data` so we can map it to a volume when we launch the container later on.
18+
19+
4. Generate the appservice registration file. This will be used by both the appservice and your homeserver.
20+
Here, you must specify the direct link the Matrix Homeserver can use to access the appservice, including the Matrix port it will send messages through (if this bridge runs on the same machine you can use `localhost` as the `$HOST` name):
21+
22+
`docker run -rm -v /your/path/to/mjolnir-data:/data matrixdotorg/mjolnir appservice -r -u "http://$HOST:$MATRIX_PORT" -f /data/config/mjolnir-registration.yaml`
23+
24+
5. Step 4 created an application service bot. This will be a bot iwth the mxid specified in `mjolnir-registration.yaml` under `sender_localpart`. You now need to invite it in the access control room that you have created in Step 1.
25+
26+
6. Start the application service `docker run -v /your/path/to/mjolnir-data/:/data/ matrixdotorg/mjolnir appservice -c /data/config/config.appservice.yaml -f /data/config/mjolnir-registration.yaml -p $MATRIX_PORT`
27+
28+
7. Copy the `mjolnir-registration.yaml` to your matrix homeserver and refer to it in `homeserver.yaml` like so:
29+
```
30+
app_service_config_files:
31+
- "/data/mjolnir-registration.yaml"
32+
```

mjolnir-entrypoint.sh

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#!/bin/sh
2+
3+
# This is used as the entrypoint in the mjolnir Dockerfile.
4+
# We want to transition away form people running the image without specifying `bot` or `appservice`.
5+
# So if eventually cli arguments are provided for the bot version, we want this to be the opportunity to move to `bot`.
6+
# Therefore using arguments without specifying `bot` (or appservice) is unsupported.
7+
# We maintain the behaviour where if it looks like someone is providing an executable to `docker run`, then we will execute that instead.
8+
# This aids configuration and debugging of the image if for example node needed to be started via another method.
9+
case "$1" in
10+
bot) shift; set -- node /mjolnir/index.js "$@";;
11+
appservice) shift; set -- node /mjolnir/appservice/cli.js "$@";;
12+
esac
13+
14+
exec "$@";

mx-tester.yml

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,16 @@ name: mjolnir
22

33
up:
44
before:
5+
- docker run --rm --network $MX_TEST_NETWORK_NAME --name mjolnir-test-postgres --domainname mjolnir-test-postgres -e POSTGRES_PASSWORD=mjolnir-test -e POSTGRES_USER=mjolnir-tester -e POSTGRES_DB=mjolnir-test-db -d -p 127.0.0.1:8083:5432 postgres
6+
# Wait until postgresql is ready
7+
- until psql postgres://mjolnir-tester:mjolnir-test@localhost:8083/mjolnir-test-db -c ""; do echo "Waiting for psql..."; sleep 1s; done
8+
# Make table in postgres
9+
- psql postgres://mjolnir-tester:mjolnir-test@localhost:8083/mjolnir-test-db -c "CREATE TABLE mjolnir (local_part VARCHAR(255), owner VARCHAR(255), management_room TEXT)"
510
# Launch the reverse proxy, listening for connections *only* on the local host.
611
- docker run --rm --network host --name mjolnir-test-reverse-proxy -p 127.0.0.1:8081:80 -v $MX_TEST_CWD/test/nginx.conf:/etc/nginx/nginx.conf:ro -d nginx
12+
- yarn install
13+
- npx ts-node src/appservice/cli.ts -r -u "http://host.docker.internal:9000"
14+
- cp mjolnir-registration.yaml $MX_TEST_SYNAPSE_DIR/data/
715
after:
816
# Wait until Synapse is ready
917
- until curl localhost:9999 --stderr /dev/null > /dev/null; do echo "Waiting for Synapse..."; sleep 1s; done
@@ -14,12 +22,13 @@ run:
1422

1523
down:
1624
finally:
25+
- docker stop mjolnir-test-postgres || true
1726
- docker stop mjolnir-test-reverse-proxy || true
1827

1928
modules:
2029
- name: mjolnir
2130
build:
22-
- cp -r synapse_antispam $MX_TEST_MODULE_DIR
31+
- cp -r synapse_antispam $MX_TEST_MODULE_DIR/
2332
config:
2433
module: mjolnir.Module
2534
config: {}
@@ -34,6 +43,9 @@ homeserver:
3443
enable_registration: true
3544
enable_registration_without_verification: true
3645

46+
app_service_config_files:
47+
- "/data/mjolnir-registration.yaml"
48+
3749
# We remove rc_message so we can test rate limiting,
3850
# but we keep the others because of https://github.com/matrix-org/synapse/issues/11785
3951
# and we don't want to slow integration tests down.

package.json

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,37 +14,44 @@
1414
"start:dev": "yarn build && node --async-stack-traces lib/index.js",
1515
"test": "ts-mocha --project ./tsconfig.json test/commands/**/*.ts",
1616
"test:integration": "NODE_ENV=harness ts-mocha --async-stack-traces --require test/integration/fixtures.ts --timeout 300000 --project ./tsconfig.json \"test/integration/**/*Test.ts\"",
17+
"test:appservice:integration": "NODE_ENV=harness ts-mocha --async-stack-traces --timeout 300000 --project ./tsconfig.json \"test/appservice/integration/**/*Test.ts\"",
1718
"test:manual": "NODE_ENV=harness ts-node test/integration/manualLaunchScript.ts",
1819
"version": "sed -i '/# version automated/s/[0-9][0-9]*\\.[0-9][0-9]*\\.[0-9][0-9]*/'$npm_package_version'/' synapse_antispam/setup.py && git add synapse_antispam/setup.py && cat synapse_antispam/setup.py"
1920
},
2021
"devDependencies": {
2122
"@types/crypto-js": "^4.0.2",
23+
"@types/express": "^4.17.13",
2224
"@types/html-to-text": "^8.0.1",
2325
"@types/humanize-duration": "^3.27.1",
2426
"@types/js-yaml": "^4.0.5",
2527
"@types/jsdom": "^16.2.11",
2628
"@types/mocha": "^9.0.0",
29+
"@types/nedb": "^1.8.12",
2730
"@types/node": "^16.7.10",
31+
"@types/pg": "^8.6.5",
32+
"@types/request": "^2.48.8",
2833
"@types/shell-quote": "1.7.1",
2934
"crypto-js": "^4.1.1",
3035
"eslint": "^7.32",
3136
"expect": "^27.0.6",
3237
"mocha": "^9.0.1",
3338
"ts-mocha": "^9.0.2",
3439
"tslint": "^6.1.3",
35-
"typescript": "^4.3.5",
40+
"typescript": "^4.8.4",
3641
"typescript-formatter": "^7.2"
3742
},
3843
"dependencies": {
3944
"await-lock": "^2.2.2",
45+
"body-parser": "^1.20.1",
4046
"express": "^4.17",
4147
"html-to-text": "^8.0.0",
4248
"humanize-duration": "^3.27.1",
4349
"humanize-duration-ts": "^2.1.1",
4450
"js-yaml": "^4.1.0",
4551
"jsdom": "^16.6.0",
46-
"matrix-bot-sdk": "^0.5.19",
52+
"matrix-appservice-bridge": "^5.0.0",
4753
"parse-duration": "^1.0.2",
54+
"pg": "^8.8.0",
4855
"shell-quote": "^1.7.3",
4956
"yaml": "^2.1.1"
5057
},

src/appservice/AccessControl.ts

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
/*
2+
Copyright 2022 The Matrix.org Foundation C.I.C.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
import { Bridge } from "matrix-appservice-bridge";
18+
import AccessControlUnit, { EntityAccess } from "../models/AccessControlUnit";
19+
import PolicyList from "../models/PolicyList";
20+
import { Permalinks } from "matrix-bot-sdk";
21+
22+
/**
23+
* Utility to manage which users have access to the application service,
24+
* meaning whether a user is able to provision a mjolnir or continue to use one.
25+
* Internally we use a policy list within matrix to determine who has access via the `AccessControlUnit`.
26+
*/
27+
export class AccessControl {
28+
29+
private constructor(
30+
private readonly accessControlList: PolicyList,
31+
private readonly accessControlUnit: AccessControlUnit
32+
) {
33+
}
34+
35+
/**
36+
* Construct and initialize access control for the `MjolnirAppService`.
37+
* @param accessControlListId The room id of a policy list used to manage access to the appservice (who can provision & use mjolniren)
38+
* @param bridge The matrix-appservice-bridge, used to get the appservice bot.
39+
* @returns A new instance of `AccessControl` to be used by `MjolnirAppService`.
40+
*/
41+
public static async setupAccessControl(
42+
/** The room id for the access control list. */
43+
accessControlListId: string,
44+
bridge: Bridge,
45+
): Promise<AccessControl> {
46+
await bridge.getBot().getClient().joinRoom(accessControlListId);
47+
const accessControlList = new PolicyList(
48+
accessControlListId,
49+
Permalinks.forRoom(accessControlListId),
50+
bridge.getBot().getClient()
51+
);
52+
const accessControlUnit = new AccessControlUnit([accessControlList]);
53+
await accessControlList.updateList();
54+
return new AccessControl(accessControlList, accessControlUnit);
55+
}
56+
57+
public handleEvent(roomId: string, event: any) {
58+
if (roomId === this.accessControlList.roomId) {
59+
this.accessControlList.updateForEvent(event);
60+
}
61+
}
62+
63+
public getUserAccess(mxid: string): EntityAccess {
64+
return this.accessControlUnit.getAccessForUser(mxid, "CHECK_SERVER");
65+
}
66+
}

0 commit comments

Comments
 (0)