Dynamic Texture

[massimiliano-mantione]

I am trying to put CPU-generated data inside a texture.
What I ultimately need is to grab frames from a camera, process them, and show the result as images, in real time.
I wanted to show those images both in a 3D scene and inside an egui UI.

The problem is that I can update the texture, I see it changing in the UI, but it disappears from the 3D scene.
If I never change it it works in both egui and the 3D renderer.
However, as soon as I write inside the texture data vector, egui shows the modified texture but the 3D scene shows no texture at all (it apparently becomes transparent for the 3D renderer).

The texture uses the TextureFormat::Rgba8UnormSrgb format, I simply write RGBA data in the texture data vector (which I manually created of the correct size).

I have seen issues related to tracking texture modifications and reflecting them into materials (like #1161) but they seem solved now.

Is there anything specific I should do?

From here on I am explaining how my code works:

I figured out that I can create a Texture use it as a resource, then I could change it by using ResMut.

However, in order to use the texture (both in egui and in a StandardMaterial) I see I need a Handle<Texture>.
So I put the texture inside Assets<Texture>, get the handle, and use it everywhere.

I have a DynamicTexture type that I can use as a resource, it just contains texture handle as some other texture-related info.
My setup looks like this:

/// set up a simple 3D scene
fn setup(
    mut commands: Commands,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<StandardMaterial>>,
    mut textures: ResMut<Assets<Texture>>,
    mut axes: ResMut<DynamicTexture>,
    mut egui_context: ResMut<EguiContext>,
) {
    let texture = axes.build_texture();
    let texture_handle = textures.add(texture);
    axes.tex = Some(texture_handle.clone());
    egui_context.set_egui_texture(AXES_TEXTURE_ID, texture_handle.clone());

    let axes_material = StandardMaterial {
        base_color_texture: Some(texture_handle),
        unlit: true,
        ..Default::default()
    };

    // plane
    commands.spawn_bundle(PbrBundle {
        mesh: meshes.add(Mesh::from(shape::Plane { size: 5.0 })),
        //material: materials.add(Color::rgb(0.3, 0.5, 0.3).into()),
        material: materials.add(axes_material),
        ..Default::default()
    });

(the texture is called axes because of what it shows now, this is irrelevant).

Then I have this system:

fn move_axes(
    time: Res<Time>,
    mut ui_state: ResMut<UiState>,
    mut axes: ResMut<DynamicTexture>,
    mut textures: ResMut<Assets<Texture>>,
) {
    if ui_state.timer.tick(time.delta()).just_finished() {
        ui_state.tick += 1;
        axes.draw(&mut textures, |tex, size_x, size_y| {
            let mut ctx = DynamicTextureCtx::new(tex, size_x, size_y);
            // This line draws into the texture
            ctx.put_axes(ui_state.tick, ui_state.tick);
            println!("axes tick: {}", ui_state.tick);
        });
    }
}

So... what's wrong?
Do I need to notify the 3D renderer that the texture changed?
If yes, how?

[kosticmarin]

Hi @massimiliano-mantione ,

Think I can help, I've implemented something similar to what You described.
I have a camera stream which I can access via a channel from another thread.
The channel is inserted into the bevy as a resource.

let input = Arc::new(Mutex::new(camera_stream));
App::build()
  .insert_resource(input.clone())

Next I have a system that grabs each new frame that it receives from the stream and
then spawns a new SpriteBundle with the Texture that is created from the video frame,
each frame is just a Vec<u8>. The old frame is tracked in the game state
and removed on each new frame.

The system looks like this.

pub fn video_stream(
  mut commands: Commands,
  input: ResMut<Arc<Mutex<BufInput<VideoFrame>>>>,
  mut textures: ResMut<Assets<Texture>>,
  mut materials: ResMut<Assets<ColorMaterial>>,
  mut game: ResMut<GameState>,
) {
  let some_frame = input.lock().unwrap().read();
  if frame.is_none() {
    return;
  }
  let frame = some_frame.unwrap();
  let width = frame.width();
  let height = frame.height();
  // create new texture
  let texture_handle = textures.add(Texture {
        data: frame.data().to_vec(),
        dimension: bevy::render::texture::TextureDimension::D2,
        format: TextureFormat::Rgba8Unorm,
        size: Extent3d {
            width,
            height,
            depth: 1,
        },
        ..Default::default()
    });
   let frame_sprite = SpriteBundle {
        material: materials.add(texture_handle.into()),
        transform: Transform {
            translation: Vec3::new(0.0, 0.0, 1.0),
            ..Default::default()
        },
        ..Default::default()
    };
  // despawn old frame if there is some
  if let Some(current_frame) = game.current_frame {
    commands.entity(current_frame).despawn_recursive();
    game.current_frame = None;
  }
  // spawn new frame texture and track the frame entity in the game state
  game.current_frame = Some(commands.spawn_bundle(frame_sprite).id());
}

This approach worked for me, the FPS worked fine but I'm not sure if there is a better way
to handle this use case.