Skip to content

Commit 3409ade

Browse files
Add custom sources documentation (#114)
## Description Closes FCE-1496 Added documentation regarding custom sources in react-client
1 parent a945525 commit 3409ade

File tree

5 files changed

+167
-3
lines changed

5 files changed

+167
-3
lines changed

.spelling

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,3 +83,8 @@ js
8383
visioncamera
8484
frameprocessorplugin
8585
Javascript
86+
WebGL
87+
WebGPU
88+
Three.js
89+
PixiJS
90+
MediaStream

docs/react/custom-sources.mdx

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
---
2+
sidebar_position: 8
3+
---
4+
5+
import Tabs from "@theme/Tabs";
6+
import TabItem from "@theme/TabItem";
7+
8+
# Custom sources
9+
10+
:::important
11+
12+
If you only wish to send camera, microphone or screen share output through Fishjam, then you most likely should refer to the documentation in [Streaming media](/react/start-streaming) and [Managing devices](/react/managing-devices) instead of this page.
13+
14+
:::
15+
16+
This section demonstrates how to stream non-standard video or audio to other peers in your web app.
17+
The utilities in this section allow you to integrate Fishjam with powerful browser APIs such as [WebGL](https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API) and [WebGPU](https://developer.mozilla.org/en-US/docs/Web/API/WebGPU_API),
18+
or higher level libraries, which leverage these APIs, such as [Three.js](https://threejs.org/), [Smelter](https://smelter.dev) or [PixiJS](https://pixijs.com/)
19+
20+
## Creating a custom source - [`useCustomSource()`](/api/web/functions/useCustomSource)
21+
22+
To create a custom source, you only need to do two things:
23+
24+
1. Call the `useCustomSource` hook.
25+
2. Call the [`setStream`](/api/web/functions/useCustomSource#setstream) callback provided by the `useCustomSource` hook with a [MediaStream](#how-to-get-a-mediastream-object) object.
26+
27+
#### Usage Example
28+
29+
```tsx
30+
import { useCustomSource } from "@fishjam-cloud/react-client";
31+
32+
export function CameraControl() {
33+
const stream: MediaStream = ...
34+
const { setStream } = useCustomSource("my-custom-source");
35+
36+
useEffect(() => {
37+
setStream(stream);
38+
}, [stream, setStream]);
39+
40+
...
41+
}
42+
```
43+
44+
### Using a created custom source
45+
46+
Once you have called [`setStream`](/api/web/functions/useCustomSource#setstream) for a given source ID (in the above example, `"my-custom-source"`), any subsequent calls to `useCustomSource` with the same ID will return the same state.
47+
This enables multiple components to control and read a shared custom source.
48+
49+
### Deleting a custom source
50+
51+
If you wish to remove a custom source, then you should call the [`setStream`](/api/web/functions/useCustomSource#setstream) callback with `null` as its argument.
52+
53+
#### Usage Example
54+
55+
```tsx
56+
const { setStream } = useCustomSource("my-custom-source");
57+
...
58+
await setStream(null);
59+
```
60+
61+
## How to get a MediaStream object?
62+
63+
Depending on your use-case, the way you can obtain a [MediaStream](https://developer.mozilla.org/en-US/docs/Web/API/MediaStream) will vary.
64+
Below are some common examples of how to obtain one from an existing video or audio source.
65+
66+
### Canvas
67+
68+
If you have a `<canvas>` HTML element, then you will need to call the [`captureStream`](https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/captureStream) method of the given canvas.
69+
This is particularly useful if you are using [Three.js](https://threejs.org/) or [PixiJS](https://pixijs.com/) to render complex animations and more.
70+
71+
#### Usage Example
72+
73+
```tsx
74+
import { useCallback, useState } from "react";
75+
76+
export function CanvasExample() {
77+
const [canvasStream, setCanvasStream] = useState<MediaStream>();
78+
79+
const streamFromCanvas = useCallback((canvas: HTMLCanvasElement | null) => {
80+
if (!canvas) return;
81+
setCanvasStream(canvas.captureStream());
82+
}, []);
83+
84+
// use canvasStream as desired
85+
86+
return <canvas ref={streamFromCanvas} />;
87+
}
88+
```
89+
90+
### Smelter
91+
92+
If you are using [Smelter](https://smelter.dev), then all you need to do is register an output via [`registerOutput`](https://smelter.dev/ts-sdk/outputs/wasm-stream/).
93+
94+
#### Usage Example
95+
96+
:::tip
97+
98+
If you want to see a full example React app which uses Fishjam with Smelter, then you can check out our [full example on GitHub](https://github.com/fishjam-cloud/web-client-sdk/tree/main/examples/react-client/minimal-smelter).
99+
100+
:::
101+
102+
```tsx
103+
const { stream } = await smelter.registerOutput("example-output", <View />, {
104+
type: "stream",
105+
video: {
106+
resolution: { width: 1920, height: 1080 },
107+
},
108+
});
109+
```
110+
111+
### Video/Audio HTML tag
112+
113+
If you have a `<video>` or `<audio>` element, then you need to call their [`captureStream`](https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/captureStream) method, analogously as you would with a [`canvas`](#canvas).
114+
115+
#### Usage Example
116+
117+
<Tabs groupId="language">
118+
<TabItem value="video" label="Video">
119+
120+
```tsx
121+
import { useCallback, useState } from "react";
122+
123+
export function CanvasExample() {
124+
const [stream, setStream] = useState<MediaStream>();
125+
126+
const streamFromVideo = useCallback((video: HTMLVideoElement | null) => {
127+
if (!video) return;
128+
setStream(video.captureStream());
129+
}, []);
130+
131+
// use stream as desired
132+
133+
return <video ref={streamFromVideo} ... />;
134+
}
135+
```
136+
137+
</TabItem>
138+
139+
<TabItem value="audio" label="Audio">
140+
141+
```tsx
142+
import { useCallback, useState } from "react";
143+
144+
export function CanvasExample() {
145+
const [stream, setStream] = useState<MediaStream>();
146+
147+
const streamFromAudio = useCallback((audio: HTMLAudioElement | null) => {
148+
if (!audio) return;
149+
setStream(audio.captureStream());
150+
}, []);
151+
152+
// use stream as desired
153+
154+
return <audio ref={streamFromAudio} ... />;
155+
}
156+
```
157+
158+
</TabItem>
159+
</Tabs>

packages/web-client-sdk

Submodule web-client-sdk updated 80 files

0 commit comments

Comments
 (0)