Timeout issues on clamav scan in Konflux pr builds

[jiridanek]

Timeout issues on clamav scan

• Red Hat Konflux/odh-workbench-jupyter-pytorch-rocm-py311-ubi9-on-pull-request has failed.
• Red Hat Konflux/odh-workbench-jupyter-pytorch-rocm-py312-ubi9-on-pull-request has failed.

observed in

• RHOAIENG-21690: chore(Dockerfiles): augument micropipenv with uv for faster package installs #968

from jobs

• https://github.com/opendatahub-io/notebooks/pull/968/checks?check_run_id=47536308777
• https://github.com/opendatahub-io/notebooks/pull/968/checks?check_run_id=47536308895

@coderabbitai do we already have some issue about tuning konflux timeouts and/or cpu and memory limits and requests for resources, where we talk about tuning these, so that all jobs pass?

[jiridanek]

@coderabbitai do we already have some issue about tuning konflux timeouts and/or cpu and memory limits and requests for resources, where we talk about tuning these, so that all jobs pass?

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Search for existing GitHub issues related to Konflux, timeouts, and resource limits
echo "=== Searching for Konflux-related issues ==="
gh issue list --state=all --search="konflux" --limit=50

echo -e "\n=== Searching for timeout-related issues ==="
gh issue list --state=all --search="timeout" --limit=50

echo -e "\n=== Searching for clamav-related issues ==="
gh issue list --state=all --search="clamav" --limit=50

echo -e "\n=== Searching for resource limits/requests issues ==="
gh issue list --state=all --search="resource limits" --limit=30
gh issue list --state=all --search="resource requests" --limit=30

echo -e "\n=== Searching for CPU and memory related issues ==="
gh issue list --state=all --search="cpu memory" --limit=30

Length of output: 8888

---

@jiridanek Yes, there are several existing issues that address Konflux timeouts and resource optimization:

Main Resource Tuning Issues:

Issue #977 - "tune the konflux pipelines for better resource utilization given a certain desired level of reliability" - This appears to be the most comprehensive issue for general Konflux resource tuning.

Issue #952 - "tune openshift/release jobs resource requests for all the various pods" - Covers resource requests optimization for OpenShift jobs.

ClamAV-Specific Issues:

Issue #1695 - "Timeout issues on clamav scan in Konflux pr builds" - This is the current issue you referenced, which matches the PR description exactly.

Issue #975 - "run clamav konflux build step in parallel" - Addresses clamav performance optimization.

Issue #998 - "konflux issue, got oomkill from clamav-scan in rocm-runtime-pytorch-ubi9-python-3-11-on-pull-request" - Specific OOM issue with clamav scan.

Memory-Related Issues:

Issue #979 - "step-app-check in task ecosystem-cert-preflight-checks for rocm-jupyter-minimal-ubi9 got oomkilled with 8Gi memory limit" - Another memory limit issue in Konflux.

Issue #1380 - "Missing global timeout configuration in PipelineRun specs causes resource monopolization risk" - Addresses timeout configuration systematically.

So yes, there are comprehensive issues already tracking these concerns, with #977 being the main umbrella issue for Konflux resource optimization and #1695 being the specific clamav timeout issue you're dealing with.

🐰💭 Thoughtfully cataloging the existing issue landscape...

---

🧠 Learnings used

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1379
File: .tekton/odh-workbench-codeserver-datascience-cpu-py312-ubi9-push.yaml:38-142
Timestamp: 2025-07-11T11:14:26.823Z
Learning: jiridanek requested GitHub issue creation for missing PipelineRun timeout configuration during PR #1379 review, affecting 18 out of 32 total PipelineRun files across both Python 3.11 and 3.12 pipelines. A comprehensive issue was created covering resource monopolization risks, complete affected files analysis, standard 8-hour timeout solution, detailed acceptance criteria, implementation guidance, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1379
File: .tekton/odh-workbench-codeserver-datascience-cpu-py312-ubi9-push.yaml:38-142
Timestamp: 2025-07-11T11:14:26.823Z
Learning: jiridanek requested GitHub issue creation for missing PipelineRun timeout configuration during PR #1379 review, affecting 18 out of 32 total PipelineRun files across both Python 3.11 and 3.12 pipelines. Issue #1380 was successfully created with comprehensive problem description covering resource monopolization risks, complete affected files analysis (18 missing, 14 correct), standard 8-hour timeout solution with code examples, detailed acceptance criteria, implementation guidance, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1269
File: codeserver/ubi9-python-3.12/nginx/api/kernels/access.cgi:6-6
Timestamp: 2025-07-03T16:17:05.475Z
Learning: jiridanek requested GitHub issue creation for CGI script health-check URL configurability and timeout improvement in codeserver/ubi9-python-3.12/nginx/api/kernels/access.cgi during PR #1269 review. Issue #1312 was successfully created with comprehensive problem description covering hard-coded URL limitations, timeout protection, error handling, acceptance criteria, implementation guidance with code examples, and proper context linking, continuing the established pattern of systematic code quality improvements.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1269
File: codeserver/ubi9-python-3.12/nginx/serverconf/proxy.conf.template:52-52
Timestamp: 2025-07-03T12:40:40.885Z
Learning: jiridanek requested GitHub issue creation for nginx proxy read timeout improvement identified during PR #1269 review, specifically for reducing the 20-day timeout in codeserver/ubi9-python-3.12/nginx/serverconf/proxy.conf.template to prevent resource exhaustion and security risks. Issue #1288 was created with comprehensive problem description, risk assessment, acceptance criteria, implementation guidance, and proper context linking.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1269
File: codeserver/ubi9-python-3.12/utils/process.sh:17-19
Timestamp: 2025-07-03T14:00:00.909Z
Learning: jiridanek efficiently identifies when CodeRabbit review suggestions are already covered by existing comprehensive issues, demonstrating excellent issue management and avoiding duplicate tracking of the same improvements across multiple locations.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1269
File: codeserver/ubi9-python-3.12/nginx/api/kernels/access.cgi:6-6
Timestamp: 2025-07-03T16:17:05.475Z
Learning: jiridanek requested GitHub issue creation for CGI script health-check URL configurability and timeout improvement in codeserver/ubi9-python-3.12/nginx/api/kernels/access.cgi during PR #1269 review. The request follows the established pattern of systematic code quality improvements with comprehensive issue creation covering problem description, solution details, acceptance criteria, implementation guidance, and proper context linking.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1259
File: jupyter/rocm/tensorflow/ubi9-python-3.12/kustomize/base/service.yaml:5-15
Timestamp: 2025-07-02T18:59:15.788Z
Learning: jiridanek creates targeted GitHub issues for specific test quality improvements identified during PR reviews in opendatahub-io/notebooks. Issue #1268 demonstrates this by converting a review comment about insufficient tf2onnx conversion test validation into a comprehensive improvement plan with clear acceptance criteria, code examples, and ROCm-specific context.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1333
File: runtimes/tensorflow/ubi9-python-3.12/Pipfile:13-15
Timestamp: 2025-07-08T19:29:32.006Z
Learning: jiridanek requested GitHub issue creation for investigating TensorFlow "and-cuda" extras usage patterns during PR #1333 review. Issue #1340 was created with comprehensive investigation framework covering platform-specific analysis, deployment scenarios, TensorFlow version compatibility, clear acceptance criteria, testing approach, and implementation timeline, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1306
File: jupyter/trustyai/ubi9-python-3.12/test/test_notebook.ipynb:102-102
Timestamp: 2025-07-03T16:26:18.718Z
Learning: jiridanek requested GitHub issue creation for improving hardcoded threshold constants in TrustyAI test notebook during PR #1306 review. Issue #1317 was created with comprehensive problem description covering both bias detection threshold (-0.15670061634672994) and fairness threshold (0.0036255104824703954), multiple solution options with code examples, clear acceptance criteria, implementation guidance, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-11T11:16:05.131Z
Learning: jiridanek requested GitHub issue creation for RStudio py311 Tekton push pipelines during PR #1379 review. Issue #1384 was successfully created covering two RStudio variants (CPU and CUDA) found in manifests/base/params-latest.env, with comprehensive problem description, implementation requirements following the same pattern as other workbench pipelines, clear acceptance criteria, and proper context linking, continuing the established pattern of systematic code quality improvements through detailed issue tracking.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-09T18:14:03.719Z
Learning: When analyzing OpenShift CI/Prow job failures in opendatahub-io/notebooks, pod readiness timeouts during runtime validation tests (like "kubectl wait --for=condition=ready pod runtime-pod --timeout=300s" timing out) are typically infrastructure-related issues due to cluster resource constraints, scheduling problems, or image pull failures, rather than code issues in the PR.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-01T07:11:44.903Z
Learning: Resource limits in StatefulSet manifests in opendatahub-io/notebooks are configured for testing purposes, not production deployments. This affects risk assessment when reviewing resource configurations like memory and CPU limits.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1269
File: codeserver/ubi9-python-3.12/nginx/root/usr/share/container-scripts/nginx/common.sh:1-3
Timestamp: 2025-07-03T12:07:19.365Z
Learning: jiridanek consistently requests GitHub issue creation for technical improvements identified during code reviews in opendatahub-io/notebooks, ensuring systematic tracking of code quality enhancements like shell script portability issues with comprehensive descriptions, solution options, and acceptance criteria.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#968
File: codeserver/ubi9-python-3.11/Dockerfile.cpu:138-140
Timestamp: 2025-08-01T16:07:58.701Z
Learning: jiridanek prefers architectural solutions that eliminate problems entirely rather than just fixing immediate technical issues. When presented with a pipeline safety concern about micropipenv requirements generation, he suggested removing micropipenv from the build process altogether by using pre-committed requirements.txt files, demonstrating preference for simplification and deterministic builds over complex workarounds.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1600
File: Makefile:0-0
Timestamp: 2025-08-02T08:49:03.735Z
Learning: jiridanek decided to eliminate '+' characters entirely from Makefile target names during PR #1600 review instead of implementing complex substitution workarounds, demonstrating his consistent preference for architectural solutions that eliminate problems at the source rather than adding complexity to handle edge cases.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1151
File: jupyter/tensorflow/ubi9-python-3.12/test/test_notebook.ipynb:31-34
Timestamp: 2025-07-01T07:03:05.385Z
Learning: jiridanek demonstrates excellent pattern recognition for identifying duplicated code issues across the opendatahub-io/notebooks repository. When spotting a potential problem in test notebooks, he correctly assesses that such patterns are likely replicated across multiple similar files rather than being isolated incidents, leading to more effective systematic solutions.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1259
File: jupyter/rocm/tensorflow/ubi9-python-3.12/Dockerfile.rocm:56-66
Timestamp: 2025-07-02T18:19:49.397Z
Learning: jiridanek consistently creates comprehensive follow-up GitHub issues for security concerns raised during PR reviews in opendatahub-io/notebooks, ensuring systematic tracking and resolution of supply-chain security improvements like GPG signature verification for package repositories.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#1230
File: jupyter/pytorch/ubi9-python-3.12/kustomize/base/statefulset.yaml:54-60
Timestamp: 2025-06-30T14:43:08.138Z
Learning: Issue #1212 in opendatahub-io/notebooks demonstrates that missing securityContext configuration (allowPrivilegeEscalation, runAsNonRoot, seccompProfile) causes runtime pods to fail reaching ready state and timeout after 300s on OpenShift due to PodSecurity policy violations.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-06-16T11:32:09.203Z
Learning: In the opendatahub-io/notebooks repository, there is a known issue with missing `runtimes/rocm/pytorch/ubi9-python-3.11/kustomize/base/kustomization.yaml` file that causes rocm runtime tests to fail with "no such file or directory" error. This is tracked in JIRA RHOAIENG-22044 and was intended to be fixed in PR #1015.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-09T11:22:11.653Z
Learning: When analyzing OpenShift CI/Prow job failures in opendatahub-io/notebooks, jobs that appear to "complete successfully" may actually have failed at the image build stage and never executed any e2e tests. Proper analysis requires checking if image builds succeeded before concluding that tests passed or the job was successful.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-09T11:22:11.653Z
Learning: When analyzing OpenShift CI/Prow job failures in opendatahub-io/notebooks, jobs that appear to "complete successfully" may actually have failed at the image build stage and never executed any e2e tests. Proper analysis requires checking if image builds succeeded before concluding that tests passed or the job was successful.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-06-16T11:32:09.203Z
Learning: OpenShift CI infrastructure issues in opendatahub-io/notebooks can manifest as "ContainerFailed one or more containers exited" errors in release steps, or as "Entrypoint received interrupt: terminated" messages when pods are killed during CI runs. These are typically infrastructure-level issues rather than code problems.

---

Learnt from: jiridanek
PR: opendatahub-io/notebooks#0
File: :0-0
Timestamp: 2025-07-09T18:14:03.719Z
Learning: When analyzing OpenShift CI/Prow job failures in opendatahub-io/notebooks, namespace initialization failures with "the object has been modified; please apply your changes to the latest version and try again" errors are typically transient infrastructure issues caused by concurrent modifications to the namespace object during CI setup, not code problems.

[jiridanek]

• https://github.com/konflux-ci/build-definitions/blob/main/task/clamav-scan/0.3/MIGRATION.md

now runs 8 threads