Feature request: Play audio with download support

[johnhtodd]

General notes:

• sound events should be ignored in any counts
• it is possible for multiple sounds to be played over each other if timing is not managed
• a new option of "mute all sounds" should be available in the configuration menu/URL specification with obvious effect
• what formats should be supported? gsm? wav? mp3? aif? others? I have no idea what standard codecs come included in browsers these days. I'd suggest that anything that can be digested without loading other libraries should be supported.
• does object caching automatically work here? Is a sound file requested every time it's referenced or can it be cached during the lifespan of a GoW window?

Introduces three new objects/concepts:

• sound-link objects that can be downloaded from remote servers. sound-link objects have a "volume" specifier which regulates how loudly they are played (1-10, default is 10) File must complete download before playback.
• sound-pause, which is (milliseconds) a delay in between playback of files in a sound-set. No effect if not used in a sound-set.
• sound-set - an array of sound-links or sound-pause objects. All files are downloaded before the sound-set starts to play, in order to prevent choppy playback. If a file fails to download, it is skipped during playback of any other files in the sound-set. Failed files will cause the playback to be significantly delayed.

Simple single sound file:

  {
    "type": "sound-link",
    "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/warning.aif",
    "volume": 10
  }

More complex sound-set:

{
  "sound-set": [
    {
        "type": "sound-link",
        "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/warning.aif",
        "volume": 10  
    },
    {
        "type": "sound-pause",
        "sound-pause-milliseconds": "100"
    },
    {
        "type": "sound-link",
        "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/maximum.aif",
        "volume": 10  
    }
 ]
}

[esensar]

General notes:

* sound events should be ignored in any counts

This sounds like it would make more sense to be a command, rather than event, so that should cover the above too.

* it is possible for multiple sounds to be played over each other if timing is not managed

Sounds good. This should be easier (for me) to do rather than having to handle overlaps and preventing them - this will move the burden over to the user.

* a new option of "mute all sounds" should be available in the configuration menu/URL specification with obvious effect

I planned on adding a setting similar to all other event types/commands. This will just prevent these commands from being processed - if one is already running it will not be stopped, but most browser offer the option to mute a tab. Do we still want an additional setting (or changes in this setting) to also mute sounds if they are already being played?

* what formats should be supported?  gsm? wav? mp3? aif? others?  I have no idea what standard codecs come included in browsers these days. I'd suggest that anything that can be digested without loading other libraries should be supported.

Since I will be using browsers audio support, it will be whatever the browser supports. Here is a list from MDN: https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats/Audio_codecs#common_codecs but I haven't been able to find lists for specific browsers. I agree, I would rather offload this to the browser rather than adding some additional dependencies.

* does object caching automatically work here?  Is a sound file requested every time it's referenced or can it be cached during the lifespan of a GoW `window?`

I think the browser handles it automatically.

Introduces three new objects/concepts:

* sound-link objects that can be downloaded from remote servers. sound-link objects have a "volume" specifier which regulates how loudly they are played (1-10, default is 10)  File must complete download before playback.

* sound-pause, which is (milliseconds) a delay in between playback of files in a sound-set. No effect if not used in a sound-set.

* sound-set - an array of sound-links or sound-pause objects. All files are downloaded before the sound-set starts to play, in order to prevent choppy playback. If a file fails to download, it is skipped during playback of any other files in the sound-set.  Failed files will cause the playback to be significantly delayed.

Simple single sound file:

  {
    "type": "sound-link",
    "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/warning.aif",
    "volume": 10
  }

More complex sound-set:

{
  "sound-set": [
    {
        "type": "sound-link",
        "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/warning.aif",
        "volume": 10  
    },
    {
        "type": "sound-pause",
        "sound-pause-milliseconds": "100"
    },
    {
        "type": "sound-link",
        "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/maximum.aif",
        "volume": 10  
    }
 ]
}

I have changed this slightly, partly to match other objects naming conventions and to make it act like other commands (so it supports command_delay too). I have added information about it in the README, but here are the examples from above changed into the new format:

Simple single sound file:

  {
    "type": "play_sound_command",
    "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/warning.aif",
    "volume": 10
  }

More complex sound-set:

{
  "type": "play_soundset_command,"
  "soundset": [
    {
        "type": "sound_link",
        "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/warning.aif",
        "volume": 10
    },
    {
        "type": "sound_pause",
        "sound_pause_milliseconds": "100"
    },
    {
        "type": "sound_link",
        "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/maximum.aif",
        "volume": 10
    }
 ]
}

[esensar]

If needed, it would be possible to add sound support for any event that can be displayed on the globe - it would appear as it was emitted from that point on the globe, but not sure if we need that.

[johnhtodd]

If needed, it would be possible to add sound support for any event that can be displayed on the globe - it would appear as it was emitted from that point on the globe, but not sure if we need that.

I think that spatial sounds are probably overkill; no need for that.

I'm not sure what you mean by "add sound support for any event that can be displayed on the globe" - do you mean some JSON fields that indicate a sound resource (file, link, etc.) would be associated with the primary object? This is interesting and useful, and if it's easy to do I would say it's worth adding so that sounds and object displays are connected. I assume if there is a delay on the display of the item that the sound would start at the moment the object started to be displayed.

It would also make sense to stop playing the sound if the object has a very short lifespan but the sound is longer than that display lifespan. It would be great to have diminishing volume of an item at the same rate as "fade" happens on a displayed item ,but that sounds really complicated and if it takes more than 20 minutes to do then I'd say don't invest the time.

[esensar]

I think that spatial sounds are probably overkill; no need for that.

Agreed.

I'm not sure what you mean by "add sound support for any event that can be displayed on the globe" - do you mean some JSON fields that indicate a sound resource (file, link, etc.) would be associated with the primary object? This is interesting and useful, and if it's easy to do I would say it's worth adding so that sounds and object displays are connected. I assume if there is a delay on the display of the item that the sound would start at the moment the object started to be displayed.

Yes, similar to how we can add labels and similar to any object on the globe. Yes, it would make sense to sync it to the delay of the object.

It would also make sense to stop playing the sound if the object has a very short lifespan but the sound is longer than that display lifespan. It would be great to have diminishing volume of an item at the same rate as "fade" happens on a displayed item ,but that sounds really complicated and if it takes more than 20 minutes to do then I'd say don't invest the time.

Sounds good, I think both of these would be quite simple to do.

Do you still think it makes sense to also have the play sound command that would just play sounds without being connected to a specific event (like implemented in #45)?

[johnhtodd]

General notes:

* sound events should be ignored in any counts

This sounds like it would make more sense to be a command, rather than event, so that should cover the above too.

OK.

* it is possible for multiple sounds to be played over each other if timing is not managed

Sounds good. This should be easier (for me) to do rather than having to handle overlaps and preventing them - this will move the burden over to the user.

* a new option of "mute all sounds" should be available in the configuration menu/URL specification with obvious effect

I planned on adding a setting similar to all other event types/commands. This will just prevent these commands from being processed - if one is already running it will not be stopped, but most browser offer the option to mute a tab. Do we still want an additional setting (or changes in this setting) to also mute sounds if they are already being played?

I think it's fine to trust the browser to be able to mute long-playing sounds if they are already in process, and the "mute" feature in the interface will only prevent new sounds from being started.

* what formats should be supported?  gsm? wav? mp3? aif? others?  I have no idea what standard codecs come included in browsers these days. I'd suggest that anything that can be digested without loading other libraries should be supported.

Since I will be using browsers audio support, it will be whatever the browser supports. Here is a list from MDN: https://developer.mozilla.org/en-US/docs/Web/Media/Guides/Formats/Audio_codecs#common_codecs but I haven't been able to find lists for specific browsers. I agree, I would rather offload this to the browser rather than adding some additional dependencies.

Sounds good.

* does object caching automatically work here?  Is a sound file requested every time it's referenced or can it be cached during the lifespan of a GoW `window?`

I think the browser handles it automatically.

OK.

Introduces three new objects/concepts:

* sound-link objects that can be downloaded from remote servers. sound-link objects have a "volume" specifier which regulates how loudly they are played (1-10, default is 10)  File must complete download before playback.

* sound-pause, which is (milliseconds) a delay in between playback of files in a sound-set. No effect if not used in a sound-set.

* sound-set - an array of sound-links or sound-pause objects. All files are downloaded before the sound-set starts to play, in order to prevent choppy playback. If a file fails to download, it is skipped during playback of any other files in the sound-set.  Failed files will cause the playback to be significantly delayed.

This looks fine. Is there any way to shorten the timeout for failed file TCP connection attempts, or is that buried too deeply in the browser code to influence? I think a 3 second timeout would be sufficient.

Simple single sound file:

  {
    "type": "sound-link",
    "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/warning.aif",
    "volume": 10
  }

More complex sound-set:

{
  "sound-set": [
    {
        "type": "sound-link",
        "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/warning.aif",
        "volume": 10  
    },
    {
        "type": "sound-pause",
        "sound-pause-milliseconds": "100"
    },
    {
        "type": "sound-link",
        "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/maximum.aif",
        "volume": 10  
    }
 ]
}

I have changed this slightly, partly to match other objects naming conventions and to make it act like other commands (so it supports command_delay too). I have added information about it in the README, but here are the examples from above changed into the new format:

Simple single sound file:

  {
    "type": "play_sound_command",
    "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/warning.aif",
    "volume": 10
  }

More complex sound-set:

{
  "type": "play_soundset_command,"
  "soundset": [
    {
        "type": "sound_link",
        "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/warning.aif",
        "volume": 10
    },
    {
        "type": "sound_pause",
        "sound_pause_milliseconds": "100"
    },
    {
        "type": "sound_link",
        "downloaded_object_url": "http://www.loligo.com/asterisk/sounds/AIF/maximum.aif",
        "volume": 10
    }
 ]
}

Looks good!

[johnhtodd]

Do you still think it makes sense to also have the play sound command that would just play sounds without being connected to a specific event (like implemented in #45)?

I do think is useful to have a sound without an event, though perhaps this would be creation of a "null" event with which a sound could be associated? I don't know what method would be more consistent with the method you've built.

[esensar]

This looks fine. Is there any way to shorten the timeout for failed file TCP connection attempts, or is that buried too deeply in the browser code to influence? I think a 3 second timeout would be sufficient.

I don't think we can influence that - at least not while using these high-level APIs.

[esensar]

Do you still think it makes sense to also have the play sound command that would just play sounds without being connected to a specific event (like implemented in #45)?

I do think is useful to have a sound without an event, though perhaps this would be creation of a "null" event with which a sound could be associated? I don't know what method would be more consistent with the method you've built.

Yeah, I think the null event or something like that would be better, so that we don't have duplicate concepts for playing sounds. Although, that breaks some other things then - we don't really need lat/lon for sounds, and we want them to not be included in the counter by default, so it would either require users to define lat/lon even though it is not needed and explicitly disable counter, or we would treat these events differently, but I am afraid that would introduce many issues that we didn't think about before, because we made assumptions about these events.

Maybe better to keep these commands separate for when sounds without events are required.

[johnhtodd]

This looks fine. Is there any way to shorten the timeout for failed file TCP connection attempts, or is that buried too deeply in the browser code to influence? I think a 3 second timeout would be sufficient.

I don't think we can influence that - at least not while using these high-level APIs.

OK, no need to make this more complicated than necessary. Download failures will cause delays.

[johnhtodd]

Do you still think it makes sense to also have the play sound command that would just play sounds without being connected to a specific event (like implemented in #45)?

I do think is useful to have a sound without an event, though perhaps this would be creation of a "null" event with which a sound could be associated? I don't know what method would be more consistent with the method you've built.

Yeah, I think the null event or something like that would be better, so that we don't have duplicate concepts for playing sounds. Although, that breaks some other things then - we don't really need lat/lon for sounds, and we want them to not be included in the counter by default, so it would either require users to define lat/lon even though it is not needed and explicitly disable counter, or we would treat these events differently, but I am afraid that would introduce many issues that we didn't think about before, because we made assumptions about these events.

Maybe better to keep these commands separate for when sounds without events are required.

Well, we also have a pending feature with text fields now without lat/lon as well, correct? They have similar issues with not incrementing counters or having geographic coordinates.