Skip to content

Commit 7b26d58

Browse files
Merge pull request #384 from chiefcll/feat/announcer
Announcer plugin
2 parents 79466c1 + 09e8399 commit 7b26d58

File tree

10 files changed

+512
-7
lines changed

10 files changed

+512
-7
lines changed

LICENSE

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,3 +201,10 @@
201201
limitations under the License.
202202

203203

204+
Copyright <YEAR> <COPYRIGHT HOLDER>
205+
206+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
207+
208+
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
209+
210+
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

NOTICE

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,10 @@ listed below. Your use of this material within the component is also subject to
77
conditions of these licenses. The LICENSE file contains the text of all the licenses which apply
88
within this component.
99

10+
Code from: https://github.com/jashkenas/underscore is
11+
Copyright (c) 2009-2022 Jeremy Ashkenas, Julian Gonggrijp, and DocumentCloud and Investigative Reporters & Editors
12+
Licensed under the MIT License
13+
based off:
14+
http://unscriptable.com/2009/03/20/debouncing-javascript-methods/ which is:
15+
Copyright (c) 2007-2009 unscriptable.com and John M. Hann
16+
Licensed under the MIT License (with X11 advertising exception)

docs/_sidebar.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
- [Getting started](/getting-started.md)
22

33
- Plugins
4-
- [Accessibility](/plugins/accessibility.md)
4+
- [Accessibility](/plugins/accessibility/index.md)
55
- [App](/plugins/app.md)
66
- [Utils](/plugins/utils.md)
77
- [Storage](/plugins/storage.md)
Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Announcer Library
2+
3+
The `Announcer` library allows for relevant information to be voiced along the focus path of the application. By passing the focus path to `Announcer.onFocusChange` the function traverses the `_focusPath` property collecting strings or promises of strings to announce to the user. The array of information is passed to a SpeechEngine which is responsible for converting the text to speech. By default we use the [speechSynthesis API](https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesis), but you can replace this by overwriting `Announcer._textToSpeech`.
4+
5+
Note: The speechSynth api has some known problems:<br />
6+
https://stackoverflow.com/questions/39391502/js-speechsynthesis-problems-with-the-cancel-method<br />
7+
https://stackoverflow.com/questions/23483990/speechsynthesis-api-onend-callback-not-working<br />
8+
9+
This class does its best to work around these issues, but speech synth api can randomly fail.
10+
11+
## Usage
12+
13+
```js
14+
import { Router, {Announcer} = Accessibility } from '@lightningjs/sdk';
15+
export default class App extends Router.App {
16+
...
17+
_focusChange() {
18+
Announcer.onFocusChange(this.application._focusPath);
19+
}
20+
}
21+
```
22+
23+
Set `Announcer.debug = true` to see console tables of the output as shown below.
24+
25+
| Index | Component | Property | Value |
26+
| ----- | ------------ | ---------- | ------------------------------------------------ |
27+
| 0 | BrowsePage-1 | Title | Free to Me |
28+
| 1 | Grid | Title | |
29+
| 2 | Rows | Title | |
30+
| 3 | Row | Title | Popular Movies - Free to Me |
31+
| 4 | Items | Title | |
32+
| 5 | TileItem | Title | Teenage Mutant Ninja Turtles: Out of the Shadows |
33+
| 6 | Metadata | Announce | Promise |
34+
| 7 | Metadata | No Context | |
35+
| 8 | TileItem | Context | 1 of 5 |
36+
| 9 | Items | No Context | |
37+
| 10 | Row | No Context | |
38+
| 11 | Rows | No Context | |
39+
| 12 | Grid | No Context | |
40+
| 13 | BrowsePage-1 | Context | PAUSE-2 Press LEFT or RIGHT to |
41+
42+
The `Announcer` will travel through the `_focusPath` looking for `component.announce` then `component.title` properties. After collecting those properties it reverses the `_focusPath` looking for `component.announceContext`.
43+
44+
### SpeechType
45+
46+
All of the properties may return values compatible with the following recursive type definition:
47+
48+
```
49+
SpeechType = string | Array<SpeechType> | Promise<SpeechType> | () => SpeechType
50+
```
51+
52+
At its simplest, you may return a string or an array of strings. Promises and functions that return SpeechType values may be used as necessary for advanced/asyncronous use cases.
53+
54+
#### Examples
55+
56+
```js
57+
get announce() {
58+
return [
59+
'Despicable Me',
60+
Promise.resolve([
61+
['2020', 'Rated PG'],
62+
Promise.resolve('Steve Carell, Miranda Cosgrove, Kristen Wiig, Pierre Coffin'),
63+
() => 'A description of the movie'
64+
])
65+
];
66+
}
67+
68+
get announceContext() {
69+
return 'Press LEFT or RIGHT to review items';
70+
}
71+
```
72+
73+
## Inserting a pause
74+
75+
You may also use `PAUSE-#` to pause speech for # seconds before saying the next string.
76+
77+
```js
78+
get announceContext() {
79+
return ['PAUSE-2.5', 'Hello there!'];
80+
}
81+
```
82+
83+
## API
84+
### Properties
85+
86+
| name | type | readonly | description |
87+
| ---------------- | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
88+
| enabled | boolean | false | default true - flag to turn on or off Announcer |
89+
| announcerTimeout | number | false | By default the announcer only gets information about what changed between focus paths. The announcerTimeout resets the cache to announce the full focus path when the user has been inactive a certain amount of time. Default value is 5 minutes. |
90+
91+
### Methods
92+
93+
| name | args | description |
94+
| ----------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
95+
| speak | | Performs a manual announce |
96+
| &nbsp; | `announcement` | See _SpeechType_ above |
97+
| &nbsp; | `options` | Object containing one or more boolean flags: <br/><ul><li>append - Appends announcement to the currently announcing series.</li><li>notification - Speaks out notification and then performs $announcerRefresh.</li></ul> |
98+
| clearPrevFocus | `depth` | Clears the last known focusPath - depth can trim known focusPath |
99+
| cancel | none | Cancels current speaking |
100+
| setupTimers | `options` | Object containing: <br/><ul><li>focusDebounce - default amount of time to wait after last input before focus change announcing will occur.</li><li>focusChangeTimeout - Amount of time with no input before full announce will occur on next focusChange</li></ul> |

docs/plugins/accessibility.md renamed to docs/plugins/accessibility/colorshift.md

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,3 @@
1-
# Accessibility
2-
3-
The Accessibility plugin provides functionality to easily make your App more accessible.
4-
5-
Currently it provides functionality to apply a **Colorshifting filter** to your App, helping people with different types of color blindness to properly use your App.
6-
71
## Usage
82

93
The Accessibility plugin is automatically included in the root of your App. In most cases there is no need to specifically import the Accessibility plugin into your App code.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Accessibility
2+
3+
The Accessibility plugins provide functionality to make your App more accessible.
4+
5+
[Announcer](announcer.md)
6+
Add voice guidance for visually impaired users to provide auditory information about current focus path.
7+
8+
[Colorshift](colorshift.md)
9+
Currently it provides functionality to apply a **Colorshifting filter** to your App, helping people with different types of color blindness to properly use your App.
10+
11+
Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
/*
2+
* If not stated otherwise in this file or this component's LICENSE file the
3+
* following copyright and licenses apply:
4+
*
5+
* Copyright 2020 Metrological
6+
*
7+
* Licensed under the Apache License, Version 2.0 (the License);
8+
* you may not use this file except in compliance with the License.
9+
* You may obtain a copy of the License at
10+
*
11+
* http://www.apache.org/licenses/LICENSE-2.0
12+
*
13+
* Unless required by applicable law or agreed to in writing, software
14+
* distributed under the License is distributed on an "AS IS" BASIS,
15+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
* See the License for the specific language governing permissions and
17+
* limitations under the License.
18+
*/
19+
20+
/* global SpeechSynthesisErrorEvent */
21+
function flattenStrings(series = []) {
22+
const flattenedSeries = []
23+
24+
for (var i = 0; i < series.length; i++) {
25+
if (typeof series[i] === 'string' && !series[i].includes('PAUSE-')) {
26+
flattenedSeries.push(series[i])
27+
} else {
28+
break
29+
}
30+
}
31+
// add a "word boundary" to ensure the Announcer doesn't automatically try to
32+
// interpret strings that look like dates but are not actually dates
33+
// for example, if "Rising Sun" and "1993" are meant to be two separate lines,
34+
// when read together, "Sun 1993" is interpretted as "Sunday 1993"
35+
return [flattenedSeries.join(',\b ')].concat(series.slice(i))
36+
}
37+
38+
function delay(pause) {
39+
return new Promise(resolve => {
40+
setTimeout(resolve, pause)
41+
})
42+
}
43+
44+
/**
45+
* Speak a string
46+
*
47+
* @param {string} phrase Phrase to speak
48+
* @param {SpeechSynthesisUtterance[]} utterances An array which the new SpeechSynthesisUtterance instance representing this utterance will be appended
49+
* @return {Promise<void>} Promise resolved when the utterance has finished speaking, and rejected if there's an error
50+
*/
51+
function speak(phrase, utterances, lang = 'en-US') {
52+
const synth = window.speechSynthesis
53+
return new Promise((resolve, reject) => {
54+
const utterance = new SpeechSynthesisUtterance(phrase)
55+
utterance.lang = lang
56+
utterance.onend = () => {
57+
resolve()
58+
}
59+
utterance.onerror = e => {
60+
reject(e)
61+
}
62+
utterances.push(utterance)
63+
synth.speak(utterance)
64+
})
65+
}
66+
67+
function speakSeries(series, lang, root = true) {
68+
const synth = window.speechSynthesis
69+
const remainingPhrases = flattenStrings(Array.isArray(series) ? series : [series])
70+
const nestedSeriesResults = []
71+
/*
72+
We hold this array of SpeechSynthesisUtterances in order to prevent them from being
73+
garbage collected prematurely on STB hardware which can cause the 'onend' events of
74+
utterances to not fire consistently.
75+
*/
76+
const utterances = []
77+
let active = true
78+
79+
const seriesChain = (async () => {
80+
try {
81+
while (active && remainingPhrases.length) {
82+
const phrase = await Promise.resolve(remainingPhrases.shift())
83+
if (!active) {
84+
// Exit
85+
// Need to check this after the await in case it was cancelled in between
86+
break
87+
} else if (typeof phrase === 'string' && phrase.includes('PAUSE-')) {
88+
// Pause it
89+
let pause = phrase.split('PAUSE-')[1] * 1000
90+
if (isNaN(pause)) {
91+
pause = 0
92+
}
93+
await delay(pause)
94+
} else if (typeof phrase === 'string' && phrase.length) {
95+
// Speak it
96+
const totalRetries = 3
97+
let retriesLeft = totalRetries
98+
while (active && retriesLeft > 0) {
99+
try {
100+
await speak(phrase, utterances, lang)
101+
retriesLeft = 0
102+
} catch (e) {
103+
// eslint-disable-next-line no-undef
104+
if (e instanceof SpeechSynthesisErrorEvent) {
105+
if (e.error === 'network') {
106+
retriesLeft--
107+
console.warn(`Speech synthesis network error. Retries left: ${retriesLeft}`)
108+
await delay(500 * (totalRetries - retriesLeft))
109+
} else if (e.error === 'canceled' || e.error === 'interrupted') {
110+
// Cancel or interrupt error (ignore)
111+
retriesLeft = 0
112+
} else {
113+
throw new Error(`SpeechSynthesisErrorEvent: ${e.error}`)
114+
}
115+
} else {
116+
throw e
117+
}
118+
}
119+
}
120+
} else if (typeof phrase === 'function') {
121+
const seriesResult = speakSeries(phrase(), lang, false)
122+
nestedSeriesResults.push(seriesResult)
123+
await seriesResult.series
124+
} else if (Array.isArray(phrase)) {
125+
// Speak it (recursively)
126+
const seriesResult = speakSeries(phrase, lang, false)
127+
nestedSeriesResults.push(seriesResult)
128+
await seriesResult.series
129+
}
130+
}
131+
} finally {
132+
active = false
133+
}
134+
})()
135+
return {
136+
series: seriesChain,
137+
get active() {
138+
return active
139+
},
140+
append: toSpeak => {
141+
remainingPhrases.push(toSpeak)
142+
},
143+
cancel: () => {
144+
if (!active) {
145+
return
146+
}
147+
if (root) {
148+
synth.cancel()
149+
}
150+
nestedSeriesResults.forEach(nestedSeriesResults => {
151+
nestedSeriesResults.cancel()
152+
})
153+
active = false
154+
},
155+
}
156+
}
157+
158+
let currentSeries
159+
export default function(toSpeak, lang) {
160+
currentSeries && currentSeries.cancel()
161+
currentSeries = speakSeries(toSpeak, lang)
162+
return currentSeries
163+
}

0 commit comments

Comments
 (0)