How to Capture Rendered Bevy Frames and send to Webrtc

[cs50victor]

Hey everyone,
New Bevy Engine user here : ) !

How do i capture the Bevy's rendered frames? I know there's a bevy_renderer plugin but I can I access frames like this ?

https://github.com/cs50victor/kitt2/blob/main/lkgpt/src/main.rs#L2892

I'm currently capturing image frames from a custom vulkan renderer ^ .

[ShadowMitia]

Depends on what you want to do, but you can take screenshots inside bevy: https://docs.rs/bevy/latest/bevy/render/view/window/screenshot/struct.ScreenshotManager.html.

[cs50victor]

I want to capture each frame quickly. I also want to run it headless.

[cs50victor]

would using screenhsots need me to use a window? (I'm planing on running bevy on a server)

[ShadowMitia]

Mmmh it might need a window looking quickly at the docs (but I haven't tried), but the implementation of it could help you out.

[cs50victor]

Thanks, will definitely look into it.

[paul-hansen]

You can set the target field on the camera to be an image and access that image like you do any other bevy image/assets.

There's an example that shows how to use this to render the camera to a cube's texture: https://github.com/bevyengine/bevy/blob/main/examples/3d/render_to_texture.rs#L105

Another useful thing to know is that you can use Image::try_into_dynamic to convert Bevy images into the rust image crate's dynamic image. Useful if whatever you are using for webrtc already has interop with the image crate. It also has methods for saving to files etc.

[cs50victor]

can this be done in a without a window ( headless ) ?

[paul-hansen]

I'm not sure, there is an open issue for that but it hasn't been updated in awhile: #22

There's also this issue that seems like it maybe addressed it? The last comment might be helpful: #3155 (comment) it shows how to disable the winit plugin and set a manual update rate (Might need updated to work with 0.12 though).

We might just need someone to try it and report back whatever issues they run into, it seems like Bevy is open to supporting it at least.

[cs50victor]

@ShadowMitia @paul-hansen . It seems that the bevy_renderer is tied to the window being available.

Hey @cart , is bevy_renderer tightly coupled with bevy_renderer ? I'm trying to run bevy headlessly on a server and stream the
rendered frames to the client ( some context : https://github.com/cs50victor/kitt2/blob/main/README.md ).

Is there a way to walk around this or would using a custom render loop like be a better solution? i.e https://github.com/hakolao/bevy_vulkano

[ShadowMitia]

I didn't dive deeply, but I think bevy_renderer and bevy_window are tightly coupled. No way around this.
It might not be too difficult to decouple them?

bevy_vulkano looks like it has a windowless mode, it might be easier to setup for you right now.

[ShadowMitia]

Oh, I found this PR, you might be able to adapt it: https://github.com/bevyengine/bevy/pull/5550/files

[cs50victor]

dang. bummer. I'll play around with the PR you provided. Thanks.

[cs50victor]

Figured it out. Thanks @ShadowMitia @paul-hansen
Hopefully this helps someone else in future.
Here's a link to a working example : https://github.com/mosure/bevy_gaussian_splatting/blob/main/examples/headless.rs