diff --git a/.buildkite/pipeline.yaml b/.buildkite/pipeline.yaml index 6243482c16..0028d3cacc 100644 --- a/.buildkite/pipeline.yaml +++ b/.buildkite/pipeline.yaml @@ -56,6 +56,7 @@ steps: - wget https://github.com/rust-lang/mdBook/releases/download/v0.4.1/mdbook-v0.4.1-x86_64-unknown-linux-gnu.tar.gz - tar -xf mdbook-v0.4.1-x86_64-unknown-linux-gnu.tar.gz - ./mdbook build server -d /workdir/implementation-guides/implementation-guides/server + - ./mdbook build widgets -d /workdir/implementation-guides/implementation-guides/widgets - tar -czf implementation-guides.tar.gz implementation-guides artifact_paths: - implementation-guides/implementation-guides.tar.gz diff --git a/implementation-guides/widgets/book.toml b/implementation-guides/widgets/book.toml new file mode 100644 index 0000000000..4a75e91b3b --- /dev/null +++ b/implementation-guides/widgets/book.toml @@ -0,0 +1,14 @@ +[book] +authors = ["The Matrix.org Foundation C.I.C."] +language = "en" +multilingual = false +src = "src" +title = "Matrix Widget Implementors Guide" + +[output.html] +theme = "../theme" +git-repository-url = "https://github.com/matrix-org/matrix.org/tree/master/implementation-guides/widgets" + +[preprocessor.links] + +[preprocessor.index] diff --git a/implementation-guides/widgets/src/SUMMARY.md b/implementation-guides/widgets/src/SUMMARY.md new file mode 100644 index 0000000000..5347af5be9 --- /dev/null +++ b/implementation-guides/widgets/src/SUMMARY.md @@ -0,0 +1,14 @@ +# Summary + +[Introduction](./intro.md) + +- [Widget basics](./basics/readme.md) + - [URL templating](./basics/url-templating.md) +- [Communicating with clients](./communication/readme.md) + - [Requests/Responses](./communication/requests-responses.md) + - [Error handling](./communication/errors.md) + - [Capabilities](./communication/capabilities.md) +- [A simple stickerpicker](./example-stickerpicker/readme.md) + - [Defining our stickers](./example-stickerpicker/send-behaviour.md) + - [Communicating with the client](./example-stickerpicker/communication.md) + - [Using the stickerpicker in Element](./example-stickerpicker/usage-element-web.md) diff --git a/implementation-guides/widgets/src/basics/readme.md b/implementation-guides/widgets/src/basics/readme.md new file mode 100644 index 0000000000..32a14d4f2e --- /dev/null +++ b/implementation-guides/widgets/src/basics/readme.md @@ -0,0 +1,77 @@ +# Widget basics + +Widgets exist in two places currently: rooms and on a user's account. Room widgets are accessible +to anyone who can see the room while account widgets are only accessible to that user. + +Both forms of widget have the same general structure: + +```json +{ + "id": "20200827_WidgetExample", + "type": "m.custom", + "name": "My Cool Widget", + "url": "https://example.org/my/widget.html?roomId=$matrix_room_id", + "creatorUserId": "@alice:example.org", + "data": { + "custom-key": "This is a custom key", + "title": "This is a witty description for the widget" + } +} +``` + +The `id` is the widget's ID, which must be unique to the room/account where the widget will be +located. The `type` is almost always going to be `m.custom` to indicate it is a generic widget, +though other types are available. The `name` is simply what the widget should be called, and the +`url` is where the widget is located. + +The `creatorUserId` is the user ID of who added the widget. For account widgets this should be +the user's own ID, though for rooms it should be whoever originally added the widget. Room widgets +can be edited over time by other members of the room, so this indicates who was responsible for +the widget's construction rather than who edited it last. + +`data` has a special meaning for the `url` in that the keys of the the object can be used as variables +to the `url`. This is most useful when using a custom `type` so variables can be provided to the +client when they are using purpose-built UI. An optional `title` can be specified in the `data` +to give a short summary of what the widget is representing alongside the `name`. + +## Room widgets + +Widgets at the room level are stored as state events in the room with an event type of `m.widget` +and a state key matching the widget's `id`. The state event's content is the same as the object +described above. + +State events that are missing a `url` or `type` in the event content will not be rendered by clients. + +## Account widgets + +Widgets at the user/account level are stored in that user's account data under a single `m.widgets` +type. This type has keys which are each widget's `id` and a value consisting of a minimal room widget. + +For example: + +```json +{ + "20200827_WidgetExample": { + "content": { + "id": "20200827_WidgetExample", + "type": "m.custom", + "name": "My Cool Widget", + "url": "https://example.org/my/widget.html?roomId=$matrix_room_id", + "creatorUserId": "@alice:example.org", + "data": { + "custom-key": "This is a custom key", + "title": "This is a witty description for the widget" + } + }, + "sender": "@alice:example.org", + "state_key": "20200827_WidgetExample", + "type": "m.widget" + } +} +``` + +This looks a bit confusing, though the idea is that clients can use this similarity to render +widgets mroe easily. + +To remove a widget from the user's account, simply remove all references to it from the `m.widgets` +object. diff --git a/implementation-guides/widgets/src/basics/url-templating.md b/implementation-guides/widgets/src/basics/url-templating.md new file mode 100644 index 0000000000..b5642bd810 --- /dev/null +++ b/implementation-guides/widgets/src/basics/url-templating.md @@ -0,0 +1,14 @@ +# URL templating + +To render a widget by URL, clients will first replace any keys from `data` with their associated +values in the `url`. For example, if the `data` object was +`{"custom-key": 1234, "another-key": "hello!"}` then the client would replace `$custom-key` with +`1234` and `$another-key` with `hello!`, wherever those appear in the URL. + +Some additional variables are defined by the specification for use in the widget URL: + +* `$matrix_user_id` - The current user's ID. +* `$matrix_room_id` - The room ID the user is currently viewing, or an empty string if none + applicable. +* `$matrix_display_name` - The current user's display name, or user ID if not set. +* `$matrix_avatar_url` - An HTTP URL to the current user's avatar, or an empty string if not set. diff --git a/implementation-guides/widgets/src/communication/capabilities.md b/implementation-guides/widgets/src/communication/capabilities.md new file mode 100644 index 0000000000..a29dd8dc69 --- /dev/null +++ b/implementation-guides/widgets/src/communication/capabilities.md @@ -0,0 +1,8 @@ +# Capabilities + +As part of establishing a session for widgets and clients to communicate with each other, a +capabilities negotation happens where the client asks the widget what permissions it wants and +the widget replies with its ideal set. The client is then supposed to ask the user what permissions +to grant, or if it's obvious for certain widgets, approve them implicitly. + +The widget specification has a list of available capabilities. diff --git a/implementation-guides/widgets/src/communication/errors.md b/implementation-guides/widgets/src/communication/errors.md new file mode 100644 index 0000000000..48b42e1cb4 --- /dev/null +++ b/implementation-guides/widgets/src/communication/errors.md @@ -0,0 +1,23 @@ +# Error handling + +All requests made by clients/widgets over the Widget API should have a timeout of about 10 seconds. +If after 10 seconds no response was received, the application should either try again or give up. + +Errors related to executing a request should be sent as a `response` looking like the following: + +```json +{ + "api": "fromWidget", + "requestId": "generated-id-1234", + "widgetId": "20200827_WidgetExample", + "action": "com.example.say_hello", + "data": { + "request-param": "value" + }, + "response": { + "error": { + "message": "Failed to process request: Server returned 500 error" + } + } +} +``` diff --git a/implementation-guides/widgets/src/communication/readme.md b/implementation-guides/widgets/src/communication/readme.md new file mode 100644 index 0000000000..a2825e5abf --- /dev/null +++ b/implementation-guides/widgets/src/communication/readme.md @@ -0,0 +1,9 @@ +# Communicating with clients + +Widgets can communicate with their host client over a `postMessage` API known as the Widget API. +This API provides the widget with some abilities, such as being able to send stickers or ask +to be left on screen, and allows the client to more directly integrate with the widget. + +The API is split into two parts: the `toWidget` API and `fromWidget` API, where the difference +is where a request over the API comes from. The communication channel in which widgets and clients +speak to each other is known as a "session". diff --git a/implementation-guides/widgets/src/communication/requests-responses.md b/implementation-guides/widgets/src/communication/requests-responses.md new file mode 100644 index 0000000000..1028b086c4 --- /dev/null +++ b/implementation-guides/widgets/src/communication/requests-responses.md @@ -0,0 +1,30 @@ +# Requests/Responses + +Requests are structured objects sent over `postMessage` to the other side, where they then +respond with responses. All requests must have a response, or time out. + +An example request is: + +```json +{ + "api": "fromWidget", + "widgetId": "20200827_WidgetExample", + "requestId": "generated-id-1234", + "action": "com.example.say_hello", + "data": { + "request-param": "value" + } +} +``` + +The `api` is either `fromWidget` or `toWidget` depending on where the request is originating from. +If it's from the widget, `fromWidget`. If it's from the client, `toWidget`. + +The `widgetId` is the widget's ID and should be used to ensure requests are coming from a valid +source. The `requestId` is generated and should be unique within the session. + +The `action` is what describes the request, and what `data` attributes to include. A full description +of all actions available can be found in the widget specification. + +Responses are simply a copy of the request object with an added `response` field, where the attributes +are defined by the `action` being executed. diff --git a/implementation-guides/widgets/src/example-stickerpicker/communication.md b/implementation-guides/widgets/src/example-stickerpicker/communication.md new file mode 100644 index 0000000000..4776773f26 --- /dev/null +++ b/implementation-guides/widgets/src/example-stickerpicker/communication.md @@ -0,0 +1,89 @@ +# Communicating with the client + +To recap, at this point we have a stickerpicker which tries to send stickers to the client +but instead just encounters errors. Most importantly, we need to know our `widgetId` so we can +send a valid request and we need to ask for permission to send stickers. + +Let's set up some basic request handling and validation at the top of our `stickerpicker.js` +file: + +```javascript +let widgetId = null; // to be populated on the first `toWidget` request. + +// First we need to set up a listener to ensure we're able to hear the client's requests +window.onmessage = function(event) { + // First make sure we are roughly in shape to be a widget: we need a parent window to + // make sure it's not another tab trying to contact us. + if (!window.parent) return; + + // Next we validate to make sure the request is a valid shape + const request = event.data; + if (!request) return; + if (!request['requestId'] || !request['widgetId'] || !request['action']) return; + if (request['api'] !== "toWidget") return; + + // Now we see if it is for us, if we know what our widget ID is + if (widgetId) { + if (widgetId !== request['widgetId']) return; + } else { + widgetId = request['widgetId']; + } + + // We'll finish this function in a moment. +}; +``` + +We define a variable outside our function so we can use it wherever we want, which in our case will +be the `sendSticker` function where we currently have `widgetId: null`. Change that to +`widgetId: widgetId` so we can make sure we have a valid request. + +A lot of the `onmessage` event handler function is just making sure that the client is sending a +valid request. We don't want the user's browser extensions to interfere with our stickerpicker, so +we try and filter them out. We also filter out any requests that are not sent over the `toWidget` +API because we're not supposed to receive any other kind of request as a widget. + +Once we know we have a valid request, we'll capture the `widgetId` sent by the client or, if we +already know the widget ID, we'll make sure the client is talking to us. + +What we're looking to handle with this function is a capabilities request from the client so we can +claim our required permissions. If we set up Element Web correctly later in this guide, it will +auto-accept the permissions we need as long as we're not trying to ask for too much. This means +we need to watch out for an `action` of `capabilities` and reply to it, like the following. We'll +also respond with an error to all other actions because we aren't worried about handling them in +this proof of concept. + +```javascript +// Finally, we can get on to the action handling. +if (request['action'] === 'capabilities') { + // We're going to respond with the capabilities we want: m.sticker + window.parent.postMessage({ + ...request, // include the original request + response: { + capabilities: ['m.sticker'], + }, + }, event.origin); +} else { + // We'll send an error response for this. Ideally we'd do a full implementation + // of the widget API, but that is out of scope for this tutorial. + window.parent.postMessage({ + ...request, // include the original request + response: { + error: { + message: "Action not supported", + }, + }, + }, event.origin); +} +``` + +Both for the `capabilities` response and the error response we have `...request` which just includes +the original request object in the response for us. This ensures that we don't miss any detail when +replying, and the client will know what we're replying to. + +The `response` object for the `capabilities` request is simply the capabilities we want. We only +want to send stickers, so that's all we'll ask for. + +**Note**: This is where we diverge from the spec: we're supposed to handle a lot more request actions, +but that would make this guide complicated so we've excluded them here. + +We should now have enough to try out our widget, but we need to add it to our account first. diff --git a/implementation-guides/widgets/src/example-stickerpicker/readme.md b/implementation-guides/widgets/src/example-stickerpicker/readme.md new file mode 100644 index 0000000000..ecadeb464f --- /dev/null +++ b/implementation-guides/widgets/src/example-stickerpicker/readme.md @@ -0,0 +1,81 @@ +# A simple stickerpicker + +This guide covers writing your very own simple stickerpicker to explore some of the widget API +and gain a better understanding of the protocol. The end result for this is available on +[GitHub](https://github.com/matrix-org/simple-stickerpicker-widget) for ease of reference. + +If all goes according to plan, your widget will look like this in Element Web/Desktop: + +![stickerpicker](https://raw.githubusercontent.com/matrix-org/simple-stickerpicker-widget/master/stickerpicker.png) + +**Note**: The widget that is created this way is not fully compliant with the spec. It is for +demonstration and educational purposes only. + +## The HTML + +Luckily for this kind of widget the amount of HTML is relatively small. We're using hardcoded +stickers here, however this is just meant to teach the basics. + +To get started, create a `stickerpicker.html` with the following HTML in it: + +```html + + + + + + Example Stickerpicker + + + + + + +

Sample Stickers

+

Click a sticker to send it.

+
+ +
+
+ +
+ + + + +``` + +*Full source: https://github.com/matrix-org/simple-stickerpicker-widget/blob/master/stickerpicker.html* + +The CSS is also minimal and is just there to make it look relatively okay compared to the default +browser styling. Put this in a file named `stickerpicker.css` next to your `stickerpicker.html`: + +```css +* { + font-family: Inter, Arial, Helvetica, sans-serif; + color: #333333; + background-color: #ffffff; +} + +.sticker { + display: inline-block; + margin-right: 8px; + padding: 8px; + border-radius: 4px; + background-color: #f4f4f4; + cursor: pointer; + position: relative; +} + +.sticker:hover { + border-bottom: 5px solid #0098d4; +} + +h1 { + font-size: 1.5em; +} +``` + +*Full source: https://github.com/matrix-org/simple-stickerpicker-widget/blob/master/stickerpicker.css* + +All we have to do now is create and populate `stickerpicker.js` so the buttons work! diff --git a/implementation-guides/widgets/src/example-stickerpicker/send-behaviour.md b/implementation-guides/widgets/src/example-stickerpicker/send-behaviour.md new file mode 100644 index 0000000000..e873c9bc4e --- /dev/null +++ b/implementation-guides/widgets/src/example-stickerpicker/send-behaviour.md @@ -0,0 +1,88 @@ +# Defining our stickers + +Create a new `stickerpicker.js` file next to your `stickerpicker.html` file and put the following +in it: + +```javascript +// This is where we register our stickers. The HTML calls sendSticker() with the ID +// of the sticker to send, which is the key of this object. The value of the object +// is simply StickerActionRequestData which gets placed in the request. +const stickers = { + 'normal': { + name: "Made for Matrix badge", + content: { + url: "mxc://matrix.org/AFdISYOGCRXUIJejgxeRxaEg", + info: { + w: 256, + h: 256, + mimetype: "image/png", + }, + }, + }, + 'inverted': { + name: "Made for Matrix badge (inverted)", + content: { + url: "mxc://matrix.org/wKGtfcEVxrUFXbbipBzfvfpD", + info: { + w: 256, + h: 256, + mimetype: "image/png", + }, + }, + }, +}; +``` + +This object is used to define or "register" our stickers so we can send the right sticker to the +client. The key name is what is referenced by our HTML and the value is what we'll be sending +to the client. There are more options available, see +[the specification for StickerActionRequest](https://matrix.org/docs/spec/widgets/latest#stickeractionrequest-schema). + +**Tip**: For the best quality stickers, upload them to your server as 512x512 but say they are 256x256 +in the `info`, as we've done here. This helps Retina displays come to a less blurry result. + +We also need to define the `sendSticker` function referenced by the `onclick` handlers in the HTML, so +let's do that: + +```javascript +function sendSticker(id) { + // First, see if we forgot to register the sticker + const sticker = stickers[id]; + if (!sticker) { + alert("Error: unknown sticker"); + return; + } + + // Now create and send a request to the client to send a sticker + window.parent.postMessage({ + api: "fromWidget", // because we're sending from the widget + requestId: "sticker-" + Date.now(), // we'll use the current time to make a unique request ID + action: "m.sticker", // we want to send a sticker + widgetId: null, // we'll figure this out in a moment + data: sticker, // send the sticker request body + }, '*'); + + // Note: We post to '*' as an origin because we don't have a reliable origin to + // get access to (browsers think that `window.parent.location.origin` is cross-origin and do + // not let us see it). +} +``` + +The first half of the function ensures that we've properly defined our sticker in the `stickers` +object, allowing us to move on to sending it. + +The second half is where we send the request to the client, so let's break that down. We're using +the `window.parent` to ensure that we contact the client and not accidentally send a request to +ourselves. We're also telling the browser that we want to `postMessage` our object to all origins, +as denoted by the `*`. We have to do this because we don't really know what origin, if any, the +client is on so can't scope it to that particular location. This does mean that the user's extensions +could potentially see the message, though for this proof of concept we aren't concerned with this. + +The object itself is a [StickerActionRequest](https://matrix.org/docs/spec/widgets/latest#stickeractionrequest-schema) +originating from us (the widget), so we use `api: "fromWidget"`. Because we're starting the request, +we also need to define a `requestId` that is unique. The thing we're trying to do is send a sticker, +so say that with `action: "m.sticker"`. We don't have a `widgetId` to supply yet, but we'll get there. +Finally, as mentioned above, our `data` is simply the object defined by `stickers`, so put that there. + +Now if we were to use this stickerpicker as-is in Element Web/Desktop, we'd see a lot of errors in +the JavaScript console and nothing would work. Let's fix that. diff --git a/implementation-guides/widgets/src/example-stickerpicker/usage-element-web.md b/implementation-guides/widgets/src/example-stickerpicker/usage-element-web.md new file mode 100644 index 0000000000..4a749c2072 --- /dev/null +++ b/implementation-guides/widgets/src/example-stickerpicker/usage-element-web.md @@ -0,0 +1,51 @@ +# Using the stickerpicker in Element + +Our widget is finally ready to be used. All we have to do is upload it to an SSL-protected website +to ensure we don't run into mixed content problems. Many hosting providers, like GitHub Pages, offer +free SSL certificates and can easily deploy this widget. + +**Note**: At this point in the tutorial you'll want to use a test account to avoid causing problems +on your main account. + +In Element Web or Desktop, open up a room and type `/devtools` to open up the Developer Tools. After +clicking 'Send Account Data' you should see a form to fill out. For the Event Type, put `m.widgets` +and in the Event Content put something similar to the following: + +```json +{ + "my-stickerpicker": { + "type": "m.widget", + "state_key": "my-stickerpicker", + "creatorUserId": "@yourusername:example.org", + "content": { + "type": "m.stickerpicker", + "url": "https://yourdomain.com/stickerpicker.html", + "name": "Stickerpicker", + "data": {} + } + } +} + +``` + +The `url` and `creatorUserId` will need updating to match your particular setup. + +If everything looks good, click 'Send' and wait a few moments. Then you should be able to click the +'Show Stickers' button near the voice/video call options and see your stickerpicker. If all went +according to plan, clicking the stickers should send them into the room you're looking at. + +If for some reason your stickerpicker isn't working, visit +[#matrix-dev:matrix.org](https://matrix.to/#/#matrix-dev:matrix.org) and other members of the +community should be able to help out. + +## Next steps + +As mentioned, this stickerpicker is technically not spec compliant. Getting it up to spec is +difficult, so we've done the hard work for you in [matrix-widget-api](https://github.com/matrix-org/matrix-widget-api). Using matrix-widget-api will take care of all of the things we've covered in this +guide and make the API a bit easier to use. + +If you've been spying on the JavaScript console while trying out the sticker picker you might have +noticed some errors about widget visibility requests. This is the client trying to tell the +stickerpicker that it is visible or not visible to the user (you) - we don't need this functionality +in this guide, but if a server-side component was added to the stickerpicker to let users add or +remove their own sticker packs, it would be a good opportunity to reload data from the server. diff --git a/implementation-guides/widgets/src/intro.md b/implementation-guides/widgets/src/intro.md new file mode 100644 index 0000000000..a8f012bf11 --- /dev/null +++ b/implementation-guides/widgets/src/intro.md @@ -0,0 +1,14 @@ +# Introduction + +Widgets are embedded applications in Matrix which clients use to add functionality to their user +experience. Currently only rooms and users can have widgets associated with them, though in the +future it is planned to be possible to have "inline widgets" - widgets that can be sent as events +in a room. + +The most common widgets are video conferencing, sticker pickers, and notepads though anything that +can be represented as a website can be embedded as a widget. Clients are also able to present +widgets with purpose-built UI if they recognize the widget type. + +This implementation guide focuses on what widget and client authors need to know for enabling or +making widgets. The [specification](https://matrix.org/docs/spec/widgets) has many more details +about widgets, such as the security considerations clients and widgets should take into account.