[CI DEBUGGING] #3125

olpipi · 2025-12-17T09:51:15Z

No description provided.

Copilot

Pull request overview

This PR adds comprehensive logging infrastructure to test files for CI debugging purposes. The changes introduce timestamped logging utilities (_ts(), _log(), _stage()) and detailed timing measurements to help diagnose test execution issues in the CI environment.

Key changes:

Added standardized logging helper functions across all test files with timestamps in UTC
Instrumented test execution with timing measurements for model downloads, conversions, and generation steps
Added checks for offline mode environment variables to skip tests when network access is unavailable

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tools/who_what_benchmark/tests/test_cli_vlm.py	Added logging utilities, timing measurements for optimum-cli exports, and executable checks
tools/who_what_benchmark/tests/test_cli_text.py	Added logging utilities with timing for model downloads and conversions during test setup
tools/who_what_benchmark/tests/test_cli_reranking.py	Added logging utilities, executable checks, and timing for model exports
tools/who_what_benchmark/tests/test_cli_image.py	Added logging utilities, context manager for timing stages, and executable validation
tools/who_what_benchmark/tests/test_cli_embeddings.py	Added logging utilities with timing measurements for optimum-cli exports
tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py	Added logging utilities, offline mode checks, and timing for dataset loading and generation
tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py	Added logging utilities, offline mode checks, and timing for all major test stages

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-09T13:48:11Z

tools/who_what_benchmark/tests/test_cli_image.py

+def _truncate(s: str, limit: int = 4000) -> str:
+    if s is None:
+        return ""


The function signature expects a str type, but the implementation checks for None. Consider accepting Optional[str] in the type annotation or remove the None check if it's not needed.

Copilot · 2026-01-09T13:48:11Z

tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py


-    data = datasets.load_dataset('THUDM/LongBench', subset, split='test[:32]')
+    with _stage("load_dataset"):
+        data = datasets.load_dataset('THUDM/LongBench', subset, split='test[:32]', trust_remote_code=True)


The trust_remote_code=True parameter allows execution of arbitrary code from the dataset. This is a security risk. Consider validating the dataset source or using an alternative approach if possible.

Suggested change

data = datasets.load_dataset('THUDM/LongBench', subset, split='test[:32]', trust_remote_code=True)

trust_remote_code = os.getenv("HF_DATASETS_TRUST_REMOTE_CODE", "").lower() in ("1", "true", "yes")

data = datasets.load_dataset('THUDM/LongBench', subset, split='test[:32]', trust_remote_code=trust_remote_code)

Copilot · 2026-01-09T13:48:12Z

tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py


-    data = datasets.load_dataset('THUDM/LongBench', subset, split='test[:32]', trust_remote_code=True)
+    with _stage("load_dataset"):
+        data = datasets.load_dataset('THUDM/LongBench', subset, split='test[:32]', trust_remote_code=True)


The trust_remote_code=True parameter allows execution of arbitrary code from the dataset. This is a security risk. Consider validating the dataset source or using an alternative approach if possible.

Suggested change

data = datasets.load_dataset('THUDM/LongBench', subset, split='test[:32]', trust_remote_code=True)

data = datasets.load_dataset('THUDM/LongBench', subset, split='test[:32]')

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 10 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-19T12:07:18Z

.github/workflows/windows.yml

            timeout: 120
          - name: 'Cacheopt E2E (Part 1)'
-            cmd: 'python -m pytest -s -v tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py'
+            cmd: 'python -m pytest -s -v tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py -s'


The -s flag is duplicated in the pytest command. Remove the duplicate to avoid confusion.

Copilot · 2026-01-19T12:07:18Z

.github/workflows/windows.yml

            timeout: 180
          - name: 'Cacheopt E2E (Part 2)'
-            cmd: 'python -m pytest -s -v tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py'
+            cmd: 'python -m pytest -s -v tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py -s'


The -s flag is duplicated in the pytest command. Remove the duplicate to avoid confusion.

Copilot · 2026-01-19T12:07:19Z

.github/workflows/windows.yml

            timeout: 30
          - name: 'WWB tests'
-            cmd: 'python -m pytest -s -v tools/who_what_benchmark/tests -m "not nanollava"'
+            cmd: 'python -m pytest -s -v tools/who_what_benchmark/tests -m "not nanollava" -s'


The -s flag is duplicated in the pytest command. Remove the duplicate to avoid confusion.

Copilot · 2026-01-19T12:07:19Z

.github/workflows/manylinux_2_28.yml

            timeout: 120
          - name: 'Cacheopt E2E (Part 1)'
-            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py'
+            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py -s'


The -s flag is duplicated in the pytest command. Remove the duplicate to avoid confusion.

Copilot · 2026-01-19T12:07:19Z

.github/workflows/manylinux_2_28.yml

            timeout: 180
          - name: 'Cacheopt E2E (Part 2)'
-            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py'
+            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py -s'


The -s flag is duplicated in the pytest command. Remove the duplicate to avoid confusion.

Copilot · 2026-01-19T12:07:19Z

.github/workflows/manylinux_2_28.yml

            timeout: 30
          - name: 'WWB tests'
-            cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava"'
+            cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava" -s'


The -s flag is duplicated in the pytest command. Remove the duplicate to avoid confusion.

Suggested change

cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava" -s'

cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava"'

Copilot · 2026-01-19T12:07:20Z

.github/workflows/mac.yml

            timeout: 30
          - name: 'WWB tests'
-            cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava"'
+            cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava" -s'


The -s flag is duplicated in the pytest command. Remove the duplicate to avoid confusion.

Suggested change

cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava" -s'

cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava"'

Copilot · 2026-01-19T12:07:20Z

.github/workflows/linux.yml

            timeout: 45
          - name: 'Cacheopt E2E (Part 1)'
-            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py'
+            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py -s'


The -s flag is duplicated in the pytest command. Remove the duplicate to avoid confusion.

Copilot · 2026-01-19T12:07:20Z

.github/workflows/linux.yml

            timeout: 180
          - name: 'Cacheopt E2E (Part 2)'
-            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py'
+            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py -s'


The -s flag is duplicated in the pytest command. Remove the duplicate to avoid confusion.

Suggested change

cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py -s'

cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py'

Copilot · 2026-01-19T12:07:21Z

.github/workflows/linux.yml

            timeout: 30
          - name: 'WWB tests'
-            cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava"'
+            cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava" -s'


The -s flag is duplicated in the pytest command. Remove the duplicate to avoid confusion.

Suggested change

cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava" -s'

cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava"'

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-27T13:59:33Z

.github/workflows/windows.yml

            timeout: 120
          - name: 'Cacheopt E2E (Part 1)'
-            cmd: 'python -m pytest -s -v tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py'
+            cmd: 'python -m pytest -s -v tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py -s'


The -s flag appears twice in the pytest command. Remove the duplicate flag to keep the command clean and avoid potential confusion.

Copilot · 2026-01-27T13:59:34Z

.github/workflows/windows.yml

            timeout: 180
          - name: 'Cacheopt E2E (Part 2)'
-            cmd: 'python -m pytest -s -v tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py'
+            cmd: 'python -m pytest -s -v tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py -s'


The -s flag appears twice in the pytest command. Remove the duplicate flag to keep the command clean and avoid potential confusion.

Copilot · 2026-01-27T13:59:34Z

.github/workflows/windows.yml

            timeout: 30
          - name: 'WWB tests'
-            cmd: 'python -m pytest -s -v tools/who_what_benchmark/tests -m "not nanollava"'
+            cmd: 'python -m pytest -s -v tools/who_what_benchmark/tests -m "not nanollava" -s'


The -s flag appears twice in the pytest command. Remove the duplicate flag to keep the command clean and avoid potential confusion.

Copilot · 2026-01-27T13:59:34Z

.github/workflows/manylinux_2_28.yml

            timeout: 120
          - name: 'Cacheopt E2E (Part 1)'
-            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py'
+            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py -s'


The -s flag appears twice in the pytest command. Remove the duplicate flag to keep the command clean and avoid potential confusion.

Copilot · 2026-01-27T13:59:35Z

.github/workflows/manylinux_2_28.yml

            timeout: 180
          - name: 'Cacheopt E2E (Part 2)'
-            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py'
+            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py -s'


The -s flag appears twice in the pytest command. Remove the duplicate flag to keep the command clean and avoid potential confusion.

Suggested change

cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py -s'

cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py'

Copilot · 2026-01-27T13:59:35Z

.github/workflows/manylinux_2_28.yml

            timeout: 30
          - name: 'WWB tests'
-            cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava"'
+            cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava" -s'


The -s flag appears twice in the pytest command. Remove the duplicate flag to keep the command clean and avoid potential confusion.

Copilot · 2026-01-27T13:59:35Z

.github/workflows/mac.yml

            timeout: 30
          - name: 'WWB tests'
-            cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava"'
+            cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava" -s'


The -s flag appears twice in the pytest command. Remove the duplicate flag to keep the command clean and avoid potential confusion.

Copilot · 2026-01-27T13:59:35Z

.github/workflows/linux.yml

            timeout: 45
          - name: 'Cacheopt E2E (Part 1)'
-            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py'
+            cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_1.py -s'


The -s flag appears twice in the pytest command. Remove the duplicate flag to keep the command clean and avoid potential confusion.

Copilot · 2026-01-27T13:59:36Z

.github/workflows/linux.yml

            timeout: 30
          - name: 'WWB tests'
-            cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava"'
+            cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava" -s'


The -s flag appears twice in the pytest command. Remove the duplicate flag to keep the command clean and avoid potential confusion.

Copilot AI review requested due to automatic review settings December 17, 2025 09:51

olpipi added do_not_merge do_not_review labels Dec 17, 2025

github-actions bot added category: WWB PR changes WWB category: continuous batching Continuous batching category: GGUF GGUF file reader labels Dec 17, 2025

Copilot AI reviewed Dec 17, 2025

View reviewed changes

olpipi force-pushed the tests_profiling branch from c043c12 to 0dbaa6b Compare January 9, 2026 13:05

olpipi requested review from sbalandi and sgonorov as code owners January 9, 2026 13:05

olpipi removed request for sbalandi and sgonorov January 9, 2026 13:11

olpipi force-pushed the tests_profiling branch from 0dbaa6b to 3827700 Compare January 9, 2026 13:47

Copilot AI review requested due to automatic review settings January 9, 2026 13:47

Copilot AI reviewed Jan 9, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings January 19, 2026 12:04

olpipi force-pushed the tests_profiling branch from 8e89c2b to 6825c88 Compare January 19, 2026 12:04

olpipi requested a review from akashchi as a code owner January 19, 2026 12:04

github-actions bot added the category: GHA CI based on Github actions label Jan 19, 2026

Copilot AI reviewed Jan 19, 2026

View reviewed changes

olpipi force-pushed the tests_profiling branch from 1f489ce to 57ad4dd Compare January 27, 2026 13:57

Copilot AI review requested due to automatic review settings January 27, 2026 13:57

Copilot AI reviewed Jan 27, 2026

View reviewed changes

olpipi added 3 commits January 27, 2026 16:32

Add debug info to tests

84caad2

fix

e19c523

Update cache evuction tests to use new model

6da9258

olpipi force-pushed the tests_profiling branch from 57ad4dd to 6da9258 Compare January 27, 2026 15:33

	data = datasets.load_dataset('THUDM/LongBench', subset, split='test[:32]', trust_remote_code=True)
	trust_remote_code = os.getenv("HF_DATASETS_TRUST_REMOTE_CODE", "").lower() in ("1", "true", "yes")
	data = datasets.load_dataset('THUDM/LongBench', subset, split='test[:32]', trust_remote_code=trust_remote_code)

	cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava" -s'
	cmd: 'python -m pytest -v ./tools/who_what_benchmark/tests -m "not nanollava"'

	cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py -s'
	cmd: 'python -m pytest -v ./tests/python_tests/test_kv_cache_eviction/test_kv_cache_eviction_2.py'

[CI DEBUGGING] #3125

Are you sure you want to change the base?

[CI DEBUGGING] #3125

Conversation

olpipi commented Dec 17, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!