Skip to content

Commit 85be3c9

Browse files
author
Chris Lorenzo
committed
add announcer plugin
1 parent 401a9c0 commit 85be3c9

File tree

8 files changed

+564
-7
lines changed

8 files changed

+564
-7
lines changed

docs/_sidebar.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
- [Getting started](/getting-started.md)
22

33
- Plugins
4-
- [Accessibility](/plugins/accessibility.md)
4+
- [Accessibility](/plugins/accessibility/index.md)
55
- [App](/plugins/app.md)
66
- [Utils](/plugins/utils.md)
77
- [Storage](/plugins/storage.md)
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
# Announcer Mixin
2+
3+
Extend your app with the `Announcer` class, that when enabled, allows for relevant information to be voiced along the focus path of the application. On Focus Change events, the `Announcer` class traverses the `_focusPath` property collecting strings or promises of strings to announce to the user. The array of information is passed to a speak function which is responsible for converting the text to speech. We include a default speak function
4+
which uses the [speechSynthesis API](https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesis), but you can replace this with your own implementation by passing a speak function as the second argument to `Announcer`.
5+
6+
Note: The speechSynth api has some known problems:
7+
https://stackoverflow.com/questions/39391502/js-speechsynthesis-problems-with-the-cancel-method
8+
https://stackoverflow.com/questions/23483990/speechsynthesis-api-onend-callback-not-working
9+
10+
This class does its best to work around these issues, but speech synth api can randomly fail.
11+
12+
## Usage
13+
14+
Extend your application with `Announcer` before boot:
15+
16+
```js
17+
import { Router, Accessibility } from '@lightningjs/sdk';
18+
const Base = Announcer(Router.App)
19+
export default class App extends Base {
20+
```
21+
22+
Set `announcerEnabled` to true in your app and optionally `debug` to true to see console tables of the output as shown below.
23+
24+
| Index | Component | Property | Value |
25+
| ----- | ------------ | ---------- | ------------------------------------------------ |
26+
| 0 | BrowsePage-1 | Title | Free to Me |
27+
| 1 | Grid | Title | |
28+
| 2 | Rows | Title | |
29+
| 3 | Row | Title | Popular Movies - Free to Me |
30+
| 4 | Items | Title | |
31+
| 5 | TileItem | Title | Teenage Mutant Ninja Turtles: Out of the Shadows |
32+
| 6 | Metadata | Announce | Promise |
33+
| 7 | Metadata | No Context | |
34+
| 8 | TileItem | Context | 1 of 5 |
35+
| 9 | Items | No Context | |
36+
| 10 | Row | No Context | |
37+
| 11 | Rows | No Context | |
38+
| 12 | Grid | No Context | |
39+
| 13 | BrowsePage-1 | Context | PAUSE-2 Press LEFT or RIGHT to |
40+
41+
The `Announcer` will travel through the `_focusPath` looking for `component.announce` then `component.title` properties. After collecting those properties it reverses the `_focusPath` looking for `component.announceContext` properties.
42+
43+
### SpeechType
44+
45+
All of the properties may return values compatible with the following recursive type definition:
46+
47+
```
48+
SpeechType = string | Array<SpeechType> | Promise<SpeechType> | () => SpeechType
49+
```
50+
51+
At its simplest, you may return a string or an array of strings. Promises and functions that return SpeechType values may be used as necessary for advanced/asyncronous use cases.
52+
53+
#### Examples
54+
55+
```js
56+
get announce() {
57+
return [
58+
'Despicable Me',
59+
Promise.resolve([
60+
['2020', 'Rated PG'],
61+
Promise.resolve('Steve Carell, Miranda Cosgrove, Kristen Wiig, Pierre Coffin'),
62+
() => 'A description of the movie'
63+
])
64+
];
65+
}
66+
67+
get announceContext() {
68+
return 'Press LEFT or RIGHT to review items';
69+
}
70+
```
71+
72+
## Inserting a pause
73+
74+
You may also use `PAUSE-#` to pause speech for # seconds before saying the next string.
75+
76+
```js
77+
get announceContext() {
78+
return ['PAUSE-2.5', 'Hello there!'];
79+
}
80+
```
81+
82+
## API
83+
84+
### Options
85+
86+
| name | type | readonly | default | description |
87+
| ------------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
88+
| voiceOutDelay | integer | false | 500ms | time in ms to determine when voice out is read. | |
89+
| announcerFocusDebounce | integer | false | 400ms | time in ms to determine when voice out is read. | |
90+
| announcerTimeout | integer | false | 30000ms | time in ms till focus history is reset causing full readout | |
91+
### Properties
92+
93+
| name | type | readonly | description |
94+
| ---------------- | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
95+
| announcerEnabled | boolean | false | flag to turn on or off Announcer |
96+
| announcerTimeout | number | false | By default the announcer only gets information about what changed between focus paths. The announcerTimeout resets the cache to announce the full focus path when the user has been inactive a certain amount of time. Default value is 5 minutes. |
97+
98+
### Signals
99+
100+
| name | args | description |
101+
| ----------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
102+
| $announce | | Performs a manual announce |
103+
| &nbsp; | `announcement` | See _SpeechType_ above |
104+
| &nbsp; | `options` | Object containing one or more boolean flags: <br/><ul><li>append - Appends announcement to the currently announcing series.</li><li>notification - Speaks out notification and then performs $announcerRefresh.</li></ul> |
105+
| $announcerRefresh | depth | Performs an announce using the focusPath - depth can trim known focusPath |
106+
| $announcerCancel | none | Cancels current speaking |
107+
108+
### Events
109+
110+
These stage level events can be listened to using the syntax `this.stage.on('EVENT_NAME', callback);`
111+
112+
| name | args | description |
113+
| ------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------------- |
114+
| announceEnded | - | emitted when all contents of the announce array have finished being read out |
115+
| announceTimoutEnded | - | emitted after the amount of time of announded word count \* 500ms, used to account for the known speechSynth api issues above |

docs/plugins/accessibility.md renamed to docs/plugins/accessibility/colorshift.md

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,3 @@
1-
# Accessibility
2-
3-
The Accessibility plugin provides functionality to easily make your App more accessible.
4-
5-
Currently it provides functionality to apply a **Colorshifting filter** to your App, helping people with different types of color blindness to properly use your App.
6-
71
## Usage
82

93
The Accessibility plugin is automatically included in the root of your App. In most cases there is no need to specifically import the Accessibility plugin into your App code.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Accessibility
2+
3+
The Accessibility plugins provide functionality to make your App more accessible.
4+
5+
[Announcer](announcer.md)
6+
Add voice guidance for visually impaired users to provide auditory information about current focus path.
7+
8+
[Colorshift](colorshift.md)
9+
Currently it provides functionality to apply a **Colorshifting filter** to your App, helping people with different types of color blindness to properly use your App.
10+
11+

src/Accessibility/Announcer.js

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
/*
2+
* If not stated otherwise in this file or this component's LICENSE file the
3+
* following copyright and licenses apply:
4+
*
5+
* Copyright 2020 Metrological
6+
*
7+
* Licensed under the Apache License, Version 2.0 (the License);
8+
* you may not use this file except in compliance with the License.
9+
* You may obtain a copy of the License at
10+
*
11+
* http://www.apache.org/licenses/LICENSE-2.0
12+
*
13+
* Unless required by applicable law or agreed to in writing, software
14+
* distributed under the License is distributed on an "AS IS" BASIS,
15+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
* See the License for the specific language governing permissions and
17+
* limitations under the License.
18+
*/
19+
20+
import Speech from './Speech.js'
21+
import { debounce, getElmName } from './utils.js'
22+
23+
const defaultOptions = {
24+
voiceOutDelay: 500,
25+
announcerFocusDebounce: 400,
26+
announcerTimeout: 5 * 60 * 1000, //Five Minutes
27+
}
28+
29+
export default function Announcer(Base, speak = Speech, announcerOptions = {}) {
30+
const options = { ...defaultOptions, ...announcerOptions }
31+
32+
return class extends Base {
33+
_construct() {
34+
this._debounceAnnounceFocusChanges = debounce(
35+
this._announceFocusChanges.bind(this),
36+
options.announcerFocusDebounce
37+
)
38+
39+
this._resetFocusTimer = debounce(() => {
40+
// Reset focus path for full announce
41+
this._lastFocusPath = undefined
42+
}, options.announcerTimeout)
43+
44+
// Lightning only calls Focus Change on second focus
45+
this._focusChange()
46+
}
47+
48+
_voiceOut(toSpeak) {
49+
if (this._voiceOutDisabled) {
50+
return
51+
}
52+
53+
const speech = speak(toSpeak)
54+
// event using speech synthesis api promise
55+
if (speech && speech.series) {
56+
speech.series.then(() => {
57+
this.stage.emit('announceEnded')
58+
})
59+
}
60+
61+
// event in case speech synthesis api is flakey,
62+
// assume the ammount of time it takes to read each word
63+
const toAnnounceStr = Array.isArray(toSpeak) ? toSpeak.concat().join(' ') : toSpeak
64+
const toAnnounceWords = toAnnounceStr.split(' ')
65+
const timeoutDelay = toAnnounceWords.length * options.voiceOutDelay
66+
clearTimeout(this._announceEndedTimeout)
67+
this._announceEndedTimeout = setTimeout(() => {
68+
this.stage.emit('announceTimeoutEnded')
69+
}, timeoutDelay)
70+
71+
return speech
72+
}
73+
74+
_disable() {
75+
clearTimeout(this._announceEndedTimeout)
76+
this.stage.emit('announceEnded')
77+
this.stage.emit('announceTimeoutEnded')
78+
}
79+
80+
set announcerEnabled(val) {
81+
this._announcerEnabled = val
82+
this._focusChange()
83+
}
84+
85+
get announcerEnabled() {
86+
return this._announcerEnabled
87+
}
88+
89+
_focusChange() {
90+
if (!this._resetFocusTimer) {
91+
return
92+
}
93+
94+
this._resetFocusTimer()
95+
this.$announcerCancel()
96+
this._debounceAnnounceFocusChanges()
97+
}
98+
99+
_announceFocusChanges() {
100+
const focusPath = this.application.focusPath || []
101+
const lastFocusPath = this._lastFocusPath || []
102+
const loaded = focusPath.every(elm => !elm.loading)
103+
const focusDiff = focusPath.filter(elm => !lastFocusPath.includes(elm))
104+
105+
if (!loaded) {
106+
this._debounceAnnounceFocusChanges()
107+
return
108+
}
109+
110+
this._lastFocusPath = focusPath.slice(0)
111+
// Provide hook for focus diff for things like TextBanner
112+
this.focusDiffHook = focusDiff
113+
114+
if (!this.announcerEnabled) {
115+
return
116+
}
117+
118+
let toAnnounce = focusDiff.reduce((acc, elm) => {
119+
if (elm.announce) {
120+
acc.push([getElmName(elm), 'Announce', elm.announce])
121+
} else if (elm.title) {
122+
acc.push([getElmName(elm), 'Title', elm.title || ''])
123+
}
124+
return acc
125+
}, [])
126+
127+
focusDiff.reverse().reduce((acc, elm) => {
128+
if (elm.announceContext) {
129+
acc.push([getElmName(elm), 'Context', elm.announceContext])
130+
} else {
131+
acc.push([getElmName(elm), 'No Context', ''])
132+
}
133+
return acc
134+
}, toAnnounce)
135+
136+
if (this.debug) {
137+
console.table(toAnnounce)
138+
}
139+
140+
toAnnounce = toAnnounce.reduce((acc, a) => {
141+
const txt = a[2]
142+
txt && acc.push(txt)
143+
return acc
144+
}, [])
145+
146+
if (toAnnounce.length) {
147+
this.$announcerCancel()
148+
this._currentlySpeaking = this._voiceOut(
149+
toAnnounce.reduce((acc, val) => acc.concat(val), [])
150+
)
151+
}
152+
}
153+
154+
$announce(toAnnounce, { append = false, notification = false } = {}) {
155+
if (this.announcerEnabled) {
156+
this._debounceAnnounceFocusChanges.flush()
157+
if (append && this._currentlySpeaking && this._currentlySpeaking.active) {
158+
this._currentlySpeaking.append(toAnnounce)
159+
} else {
160+
this.$announcerCancel()
161+
this._currentlySpeaking = this._voiceOut(toAnnounce)
162+
}
163+
164+
if (notification) {
165+
this._voiceOutDisabled = true
166+
this._currentlySpeaking.series.finally(() => {
167+
this._voiceOutDisabled = false
168+
this.$announcerRefresh()
169+
})
170+
}
171+
}
172+
}
173+
174+
$announcerCancel() {
175+
this._currentlySpeaking && this._currentlySpeaking.cancel()
176+
}
177+
178+
$announcerRefresh(depth = 0) {
179+
this._lastFocusPath = this._lastFocusPath.slice(0, depth)
180+
this._resetFocusTimer()
181+
this._focusChange()
182+
}
183+
}
184+
}

0 commit comments

Comments
 (0)