Skip to content

[wwb] Add video generation to wwb #3134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Wovchena merged 13 commits into from
Jan 22, 2026
Merged

[wwb] Add video generation to wwb #3134

Wovchena merged 13 commits into from
Jan 22, 2026

Conversation

@sbalandi
Copy link
Contributor

@sbalandi sbalandi commented Dec 20, 2025 •
edited
Loading

Description

How to run:

cd ./openvino.genai/tools/who_what_benchmark
pip install .
python ./whowhatbench/wwb.py --base-model Lightricks/LTX-Video --gt-data ltx_gt.csv --model-type text-to-video --num-inference-steps 40 --video-frames-num 25

optimum-cli export openvino -m Lightricks/LTX-Video ./models/ltx_video
python ./whowhatbench/wwb.py --target-model ./models/ltx_video --gt-data ltx_gt.csv --model-type text-to-video  -v --output ./output/  --num-inference-steps 40 --video-frames-num 25

CVS-176896

Fixes #(issue)

Checklist:

  • Tests have been updated or added to cover the new code.
  • This patch fully addresses the ticket.
  • I have made corresponding changes to the documentation.

@github-actions github-actions bot added the category: WWB PR changes WWB label Dec 20, 2025
@sbalandi sbalandi force-pushed the wwb_ltx branch 10 times, most recently from 8edff7c to e5004d3 Compare December 24, 2025 18:44
@sgonorov sgonorov force-pushed the wwb_ltx branch 2 times, most recently from d358772 to cdc17ad Compare January 6, 2026 09:50
@likholat likholat requested review from Wovchena and likholat January 7, 2026 12:18
@sgonorov sgonorov force-pushed the wwb_ltx branch 2 times, most recently from 5ebed4d to ad74cce Compare January 14, 2026 09:48
@github-actions github-actions bot added category: GHA CI based on Github actions category: Python API Python API for GenAI category: LLM samples GenAI LLM samples category: CPP API Changes in GenAI C++ public headers no-match-files category: GGUF GGUF file reader category: VLM samples GenAI VLM samples labels Jan 14, 2026
@sgonorov sgonorov force-pushed the wwb_ltx branch 2 times, most recently from 971aecf to 610e542 Compare January 16, 2026 08:11
@github-actions github-actions bot removed category: GHA CI based on Github actions category: Python API Python API for GenAI category: LLM samples GenAI LLM samples category: CPP API Changes in GenAI C++ public headers labels Jan 16, 2026
@sgonorov sgonorov requested a review from Wovchena January 21, 2026 19:56
Copilot AI review requested due to automatic review settings January 21, 2026 22:55
Copilot AI reviewed Jan 21, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tools/who_what_benchmark/whowhatbench/text2video_evaluator.py
self.metrics = metrics
self.crop_prompt = crop_prompts
self.num_samples = num_samples
self.num_inference_steps = num_inference_steps or self.DEF_NUM_INF_STEPS
Copy link

Copilot AI Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The constant DEF_NUM_INF_STEPS is set to 25 on line 20, but DEF_NUM_INFERENCE_STEP in text2image_evaluator.py is set to 4. This inconsistency may confuse users about appropriate defaults for different generation types. Consider documenting why video generation requires more inference steps than image generation.

Copilot uses AI. Check for mistakes.
Copilot AI review requested due to automatic review settings January 22, 2026 00:08
Copilot AI reviewed Jan 22, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tools/who_what_benchmark/whowhatbench/whowhat_metrics.py
# according to post processing of outputs in get_video_features of LlavaNextVideoModel
# https://github.com/huggingface/transformers/blob/v4.53.2/src/transformers/models/llava_next_video/modular_llava_next_video.py#L387
outputs = outputs.hidden_states[layer_idx][:, 1:]
return outputs.mean(dim=2)
Copy link

Copilot AI Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dimension parameter in mean(dim=2) should use a named constant or include a comment explaining which dimension is being averaged (e.g., hidden_dim). This improves code readability and maintainability.

Copilot uses AI. Check for mistakes.
Wovchena
Wovchena requested changes Jan 22, 2026
View reviewed changes
Copy link
Collaborator

@Wovchena Wovchena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#3134 (review) is still applicable

@sgonorov
Copy link
Contributor

sgonorov commented Jan 22, 2026

#3134 (review) is still applicable
Created 2 bugs for the CI/tests:
https://jira.devtools.intel.com/browse/CVS-179749
https://jira.devtools.intel.com/browse/CVS-179750

@sgonorov sgonorov requested a review from Wovchena January 22, 2026 10:59
Wovchena
Wovchena approved these changes Jan 22, 2026
View reviewed changes
Copilot AI review requested due to automatic review settings January 22, 2026 11:11
Copilot AI reviewed Jan 22, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@Wovchena Wovchena enabled auto-merge January 22, 2026 11:12
@Wovchena Wovchena added this pull request to the merge queue Jan 22, 2026
Merged via the queue into openvinotoolkit:master with commit af7838e Jan 22, 2026
135 of 137 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@sgonorov sgonorov sgonorov approved these changes

@Wovchena Wovchena Wovchena approved these changes

@likholat likholat Awaiting requested review from likholat

@as-suvorov as-suvorov Awaiting requested review from as-suvorov

Assignees

@sgonorov sgonorov

Labels

category: WWB PR changes WWB Code Freeze

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sbalandi @sgonorov @Wovchena @MaximProshin