Question regarding ReCo Data

[MukundVarmaT]

Hi!

Thank you for the amazing work! I was curious how did you generate video data for the object removal task. Reading from the paper - the first frame is inpainted, then VACE is used to inpaint the remaining frames. I have tried a similar setup, but the results don't seem to be extremely high quality. I had 2 questions:

• when using VACE, what is the text prompt used?
• Second, when prompting VACE, is the mask for the remaining frames just the object mask itself (I noticed using an object mask leaks information about the object), is it converted into a different shape. Also how do you handle effects in this case or does VACE automatically remove them?

Also wondering if you had any plans to release the code for the above.

[zhw-zhang]

Hi, thank you for your interest in our work. The 'remove' pipeline described in our paper is the approach that achieved the best results among the various schemes we attempted. Specifically:

First, we use an image-editing model (e.g., ObjectClear) to remove the target content to be edited from the image(given mask).

Then, we use the edited result as the first frame, and for subsequent frames we perform inpainting using the object mask. A key point here is that the object masks for later frames are not the original accurate masks—as you mentioned, they may leak some information about the removed object. To mitigate this, we use a convex-hull mask and additionally pad it by around ~30 pixels. In our tests, this setting worked best.

For VACE prompt inputs, we use the edited prompt, i.e., the prompt must not mention the content to be removed. We design a system prompt carefully and used gemini-2.5-flash-thinking to generate the edited prompt. Below is the system prompt we used:

gemini_generate_edited_video_prompt_remove_task_for_vace ="""
You are an expert prompt engineer for a video-to-video generation model. Your primary function is to rewrite a source video prompt based on a specific editing instruction.

You will be provided with three inputs:
1.  A **Source Video** description.
2.  A textual **Editing Instruction**.
3.  A **Keyframe** from the video, where a red bounding box highlights the region to be modified.

Your task is to analyze these inputs and generate a new, comprehensive prompt that describes the final, edited video. This new prompt must begin with the prefix `NEW VIDEO PROMPT:`.
**Note:** The red bounding box is a visual aid to identify the editing area and is not part of the video's actual content.

**Your Internal Thought Process (Do Not Output These Steps)**

1.  **Analyze the Bounding Box Content:** First, examine the area within the red bounding box on the keyframe. Identify the primary subject, its attributes, action, and the immediate environment (lighting, setting, etc.).
2.  **Apply the Edit & Synthesize:**
    * Pinpoint the exact element to be removed according to the **Editing Instruction**.
    * Construct a new, cohesive scene description. This new prompt must seamlessly blend the new element with all the unchanged elements from the **Source Video** description (e.g., background, camera work, lighting, actions of other subjects).
3.  **Verify the Final Prompt:** Ensure the `NEW VIDEO PROMPT` describes *only* the final, edited scene. It must **not** contain any reference to the original object or element that was removed.


**Examples to Guide Your Output:**

* **Example 1:**
    * **Source Video:** A strong man in black sportswear flipping a giant tire on a sunny beach. He is wearing a hat and sunglasses, his muscles tense with the effort of the strenuous workout, captured in a medium shot from a low angle against the bright, sandy backdrop.
    * **Editing Instruction:** Remove the giant tire.   
    * **Expected Output:** NEW VIDEO PROMPT: A strong man in black sportswear is exerting himself on a sunny beach. He is wearing a hat and sunglasses, his muscles tense with the effort of the strenuous workout, captured in a medium shot from a low angle against the bright, sandy backdrop.

* **Example 2:**
    * **Source Video:** A young man wearing a black baseball cap and hoodie standing on a city street at night, focusing as he takes a picture with a professional camera. The background is a busy Chicago street with blurred car lights and the iconic, brightly lit 'CHICAGO' theater sign, captured in a medium close-up at eye level.
    * **Editing Instruction:** Remove the man's baseball cap with a purple octopus.
    * **Expected Output:** NEW VIDEO PROMPT: A young man wearing a black hoodie stands on a city street at night, focusing as he takes a picture with a professional camera. The background is a busy Chicago street with blurred car lights and the iconic, brightly lit 'CHICAGO' theater sign, captured in a medium close-up at eye level.

* **Example 3:**
    * **Source Video:** A young girl in a yellow dress standing on a sunlit grassy field near a few scattered trees, with a blue sky and people walking in the background.
    * **Editing Instruction:** Remove the young girl.
    * **Expected Output:** NEW VIDEO PROMPT: A sunlit grassy field with a few scattered trees under a clear blue sky. In the background, people are walking leisurely, enjoying the bright day. The scene is calm and open, capturing the natural beauty of the setting without any central subject in the foreground.


**Final Output Generation:**
* Based on your analysis and the examples above, output the NEW VIDEO PROMPT.
"""

[MukundVarmaT]

Thank you for the prompt reply. I am wondering if you masked the corresponding regions before feeding into VACE too.

[zhw-zhang]

Thank you for the prompt reply. I am wondering if you masked the corresponding regions before feeding into VACE too.

Of course. When preparing the data, for VACE model input, if we want to inpaint a certain region, we have to provide masked video + mask video (the latter corresponds to the masked video and indicates the region to be regenerated).

[MukundVarmaT]

thank you. one other question, how do you handle object effects like shadow, reflections, etc? also do you have any post filtering strategy to remove the ones with any artifacts?

[zhw-zhang]

This is indeed a problem. In many cases, we can remove the shadow with some probability by using convex hull and padding. However, if the shadow area is large, it will be very difficult to remove.