design suggestions for sonification library

[venkateshpotluri]

@make4all/sensor-accessibility I am looking for input on a few design decisions for the library before we start coding.

Usecase

First, I would like to re-iterate on the usecase for this library. The primary usecase for this is to be consumed by interfaces that run on a browser, with eventual usecases being sonification inside IDEs e.g. VSCode, and other prototypes that do not necessarily run on the browser.

API overview

After looking at several examples such as Apple's Audio Graph API (video introduction), their spatial audio API, and Vega-Lite specification, and reading up on different API designs, I feel the best approach for us would be to expose a set of functions (for consumers that have Web Audio support) and a JSON format (for consumers that do not have a web audio context). Consumers that do not have Web Audio context will need to run chromium in a separate process to play audio. We will need to investigate if this can be a headless instance.

Data structure

The primary class for sonification is the AudioSource. The AudioSource will contain the following variables.

• SourceType:Enum {TTS,AUDIO} Speaker for Aria live region or screen reader-based alerts and Audio for sonification.
• AnnouncementSeverity:Enum {LOW, hIGH} This maps to aria-polite and aria-assertive for web, and equivalent speech canceling mechanism should be developed if screen reader announcements are being made.
• Announcer: Enum {WEB,SR_SPEECH,SELF_VOICE} if we plan to incorporate screen reader announcements into our sonification library. Web will need us to induce an aria-live region into the DOM, SR_SPEECH will require messaging to be implemented into a screen reader, and SELF_VOICE support will be implemented through Web Speech.
• Mapping Function: [ value [min-max data value] => value [min-max pitch]] Function that determines how a particular property should change.
• text:String a text (format string) representation of how to speak a particular value in the data stream.
• DataStream: event look at Web Audio events for inspiration.
• Feature:[[regionEnterTrigger:Filter, regionEntercallback:Function],[inRegionTrigger:Filter,inRegionCallback:Function],[endRegionTrigger:Filter,endRegionCallback:Function][...]] a set of filter-callback combinations to denote regions of interest.

request for advice on

• API design -- is there something I am missing, based on the above list?
• Handling data: is there a better way than an event-driven approach to do this? -- will look at Web Audio events based on suggestion by @jamesadevine.

next steps

design

• update spec to support formatting for the string.
• Design methods for the library once we decide on the above list.
• draft a JSON spec

build

• setup a development environment and stub code.
• CI/CD integration.
• development begins!

[jamesadevine]

Hi @venkateshpotluri!

Couple of thoughts:

The primary class for sonification is the AudioSource. The AudioSource will contain the following variables.

• Firstly, I'm not clear on what AudioSource actually means in terms of the software architecture. Is an AudioSource the code that maps events from the UI to WebAudio/TTS? Or is AudioSource the event itself?
• If AudioSource performs the mapping and indicates what types of sonification the mapper can perform, could we please change the name of SourceType:Enum {Speaker,Audio} to make the meaning immediately clear? I think the two options should be called TextToSpeech and AudioStream. Disclaimer: I may not have full context for choosing Speaker,Audio.
• Again, if AudioSource performs the mapping, then as shown in the Audio Graph API intro video from apple, it is often desireable to have both TTS and WebAudio occur at the same time. Do you envisage the enums could be combined to configure the mapper to do both types of sonification?

Mapping Function: [ value [min-max data value] => value [min-max pitch]] Function that determines how a particular property should change.

• It seems like we need to do some pre-processing on the data beforehand and extract the min and max values. We can certainly pass min/max via this callback function, yes.

handling data: is there a better way than an event-driven approach to do this?

• Event-driven is the best way to do this. WebAudio has some really nice event-based APIs.

text:String a text representation of how to speak a particular value in the data stream.

• We probably want this to be some kind of format string. We'd need to handle cases where the number is a float with 16 decimal places, and a format string would allow the developer to pick a reasonable amount of resolution without overwhelming the listener.

[jmankoff]

One concern about min and max is that they are to some extent unknowable for streaming data. Should we dynamically update the mapping? Or should they be specified a priori?

[venkateshpotluri]

@jamesadevine Thank you for the feedback!

Is an AudioSource the code that maps events from the UI to WebAudio/TTS? Or is AudioSource the event itself?
AudioSource is the code that maps the data from the UI or hardware to Web Audio or TTS.
could we please change the name of SourceType:Enum {Speaker,Audio} to make the meaning immediately clear?
Updated. What do you think of these?
Do you envisage the enums could be combined to configure the mapper to do both types of sonification?
I was thinking that text:String could help with the kind of TTS and Audio that apple demonstrated in their video. However, I don't see a reason why these enums could not be combined to support both.
We probably want this to be some kind of format string.
Thank you for catching this! I added a to-do on me in the updated description and will post a comment once I have an update on the spec for that.

[venkateshpotluri]

Sorry about the poor formatting in the above comment. I will fix that as soon as I can get the screen reader to allow me to select the edit option.

[pelikhan]

You probably want to use another name than AudioSource since it is one of the base class in Web Audio and it will lead to some confusion.

The plan looks like a great start. If you define your JSON spec as TypeScrip types, this would provide a very clear structure of the declarative data needed to sonify a data source.

It's hard to get these kind of libraries right on the first shot, so we'll probably have to iterate the concepts as we try to sonify various libraries. What are the key scenario you want to be able to sonify?

[jmankoff]

I think the state of a button, the accelerometer, and a light sensor makes a good initial set.

[venkateshpotluri]

I think the state of a button, the accelerometer, and a light sensor makes a good initial set.

Yes. This gives us a variety of discrete and continuous signals, with single values (light sensors) and multiple axes (accelerometer).