How to improve audio performance #3809
-
|
Hi, I'm sure this is not even a Leptos question but probably about some browser interaction. I thought you may have some ideas. I'm trying to build live media streaming platform with Leptos. I thought getting microphone audio from browser may be a good starting point and here I'm. Problem 2025-04-01.01-35-33.mp4my code: use leptos::{
IntoView, ev,
html::{ElementChild, button},
logging::log,
prelude::{OnAttribute, Read, Show, ShowProps, ToChildren},
server::LocalResource,
task::spawn_local,
};
use wasm_bindgen_futures::JsFuture;
use web_sys::{
HtmlAudioElement, MediaStream, MediaStreamConstraints,
wasm_bindgen::{JsCast, JsValue},
window,
};
pub fn app() -> impl IntoView {
let audio_stream = LocalResource::new(|| media());
let props = ShowProps::builder()
.when(move || audio_stream.read().is_some())
.children(ToChildren::to_children(move || {
let audio_element = HtmlAudioElement::new().unwrap();
let audio_stream = audio_stream.read();
let audio_stream = audio_stream.as_deref();
audio_element.set_src_object(audio_stream);
button()
.on(ev::click, move |_| match audio_element.play() {
Ok(audio_element_play_promise) => {
log!("{}", "Play will");
spawn_local(async move {
JsFuture::from(audio_element_play_promise).await.ok();
});
log!("{}", "Play must");
}
Err(err_val) => log!("{:#?}", err_val),
})
.child("Happy Button")
.into_view()
}))
.fallback(|| button().child("Sad Button"))
.build();
Show(props)
}
async fn media() -> MediaStream {
let media_devices = window().unwrap().navigator().media_devices().unwrap();
let constraints = MediaStreamConstraints::new();
constraints.set_audio(&JsValue::TRUE);
let media_stream_promise = media_devices
.get_user_media_with_constraints(&constraints)
.unwrap();
let media_stream = JsFuture::from(media_stream_promise)
.await
.unwrap()
.dyn_into::<MediaStream>()
.unwrap();
let audio_tracks = media_stream.get_audio_tracks();
MediaStream::new_with_tracks(&audio_tracks).unwrap()
} |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 5 replies
-
|
This also happens in pure JavaScript too. I got code from : https://stackoverflow.com/questions/27846392/access-microphone-from-a-browser-javascript <!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title></title>
</head>
<body>
<audio id="audioCapture"></audio>
<button onclick=audioCapture.Init()>
start microphone
</button>
<p style="font-size: 42pt;">
This is a blank page for live captions like CNN video.
</p>
<script type="text/javascript">
var audioCapture = (function ()
{
var init = function ()
{
initCapture();
};
function initCapture()
{
var constraints = { audio: true, video: false };
navigator.mediaDevices.getUserMedia(constraints).then(function (stream)
{
var audio = document.getElementById('audioCapture');
audio.srcObject = stream;
audio.play();
}).catch(function (err)
{
console.log(err);
});
}
return {
Init: init
}
})();
</script>
</body>
</html> |
Beta Was this translation helpful? Give feedback.
-
|
This sounds like echo cancellation to me. If you're playing the music through your speakers, and picking it with the microphone, most users would want to still be able to talk into the microphone and have the sound of the speakers silenced to keep themselves clear. This can be implemented either in hardware directly (sometimes done in laptops), or in software (I see Pipewire, it has a module for echo cancellation, you can search on how to disable it). Alternatively, it might be noise cancellation, which has a similar effect when trying to play music through a microphone with it enabled. Pipewire may have another module for automatic noise cancellation, you can try to disable that too. With Pipewire, you may also probably be able to switch your devices to "Pro Audio" to remove all processing and directly get raw audio stream from the sound card. |
Beta Was this translation helpful? Give feedback.
-
|
I'm not giving audio through microphone manually. I give via patchbay (helvum). |
Beta Was this translation helpful? Give feedback.
-
|
It might be something that Firefox does, in which case you might be able to fiddle with either the Firefox settings or the way you request audio input, I guess. |
Beta Was this translation helpful? Give feedback.
-
|
I tried with zen (firefox), chromium, brave and result is same. Also even if I didn't do any modification and just speak through microphone. The quality of my voice is also feels clunky. Understandable and not that bad like the video above but not like a native application recording my voice (obs, audacity). I also feels like some kind of voice supression but I couldn't fix it. |
Beta Was this translation helpful? Give feedback.
-
|
Look at this article from the MDN docs: https://developer.mozilla.org/en-US/docs/Web/API/Media_Capture_and_Streams_API/Constraints More specifically, go check their "Constraint Exerciser" which is a demo page for testing various options for getting audio (and video) input. There are indeed options for echo cancellation and noise suppression, and it seems the latter is on by default. |
Beta Was this translation helpful? Give feedback.
-
|
You are awesome. Thanks a lot. In case if someone needs for future here is mine: use leptos::{
IntoView, ev,
html::{ElementChild, button},
logging::log,
prelude::{OnAttribute, Read, Show, ShowProps, ToChildren},
server::LocalResource,
task::spawn_local,
};
use wasm_bindgen_futures::JsFuture;
use web_sys::{
HtmlAudioElement, MediaStream, MediaStreamConstraints, MediaStreamTrack, MediaTrackConstraints,
wasm_bindgen::{JsCast, JsValue},
window,
};
pub fn app() -> impl IntoView {
let audio_stream = LocalResource::new(|| media());
let props = ShowProps::builder()
.when(move || audio_stream.read().is_some())
.children(ToChildren::to_children(move || {
let audio_element = HtmlAudioElement::new().unwrap();
let audio_stream = audio_stream.read();
let audio_stream = audio_stream.as_deref();
audio_element.set_src_object(audio_stream);
button()
.on(ev::click, move |_| match audio_element.play() {
Ok(audio_element_play_promise) => {
log!("{}", "Play will");
spawn_local(async move {
JsFuture::from(audio_element_play_promise).await.ok();
});
log!("{}", "Play must");
}
Err(err_val) => log!("{:#?}", err_val),
})
.child("Happy Button")
.into_view()
}))
.fallback(|| button().child("Sad Button"))
.build();
Show(props)
}
async fn media() -> MediaStream {
let media_devices = window().unwrap().navigator().media_devices().unwrap();
let media_stream_constraints = MediaStreamConstraints::new();
let media_track_constraints = MediaTrackConstraints::new();
media_stream_constraints.set_audio(&JsValue::TRUE);
media_track_constraints.set_echo_cancellation(&JsValue::FALSE);
media_track_constraints.set_noise_suppression(&JsValue::FALSE);
media_track_constraints.set_auto_gain_control(&JsValue::FALSE);
let media_stream_promise = media_devices
.get_user_media_with_constraints(&media_stream_constraints)
.unwrap();
let media_stream = JsFuture::from(media_stream_promise)
.await
.unwrap()
.dyn_into::<MediaStream>()
.unwrap();
let audio_tracks = media_stream.get_audio_tracks();
let audio_tracks = audio_tracks
.iter()
.map(|audio_track| audio_track.dyn_into::<MediaStreamTrack>().unwrap())
.collect::<Vec<_>>();
log!("{:#?}, len = {}", audio_tracks, audio_tracks.len());
let audio_track = audio_tracks.first().unwrap();
let media_track_apply_constraints_promise = audio_track
.apply_constraints_with_constraints(&media_track_constraints)
.unwrap();
JsFuture::from(media_track_apply_constraints_promise)
.await
.unwrap();
let media_stream = MediaStream::new().unwrap();
log!("{:#?}", audio_track.get_constraints());
media_stream.add_track(audio_track);
media_stream
} |
Beta Was this translation helpful? Give feedback.
Look at this article from the MDN docs: https://developer.mozilla.org/en-US/docs/Web/API/Media_Capture_and_Streams_API/Constraints
More specifically, go check their "Constraint Exerciser" which is a demo page for testing various options for getting audio (and video) input. There are indeed options for echo cancellation and noise suppression, and it seems the latter is on by default.