re-write, separate steps/jobs analysis

Author: skudasov

.github/workflows/framework.yml

@@ -50,6 +50,10 @@ jobs:
             go-modules-${{ hashFiles('**/go.sum') }}-${{ runner.os }}-framework
       - name: Install dependencies
         run: go mod download
+      - name: Run CLI Tests
+        if: steps.changes.outputs.src == 'true'
+        run: |
+          go test -v -count 1 -run TestCLI
       - name: Run Docker Component Tests
         if: steps.changes.outputs.src == 'true'
         run: |

.nancy-ignore

@@ -15,4 +15,5 @@ CVE-2024-10086 # Improper Neutralization of Input During Web Page Generation ('C
 CVE-2024-51744 # CWE-755: Improper Handling of Exceptional Conditions
 CVE-2024-45338 # CWE-770: Allocation of Resources Without Limits or Throttling
 CVE-2024-45337 # CWE-863: Incorrect Authorization in golang.org/x/[email protected]
-CVE-2025-24883 # CWE-248: Uncaught Exception
+CVE-2025-24883 # CWE-248: Uncaught Exception
+CVE-2025-22869 # CWE-770: Allocation of Resources Without Limits or Throttling

book/src/framework/components/analyze_ci.md

@@ -2,14 +2,36 @@
 
 We offer a straightforward CLI tool designed to analyze CI runs, focusing on Jobs and Steps, to provide deeper insights into system-level tests.
 
+![img_1.png](img_1.png)
+
+## What to look for
+
+Check execution times, results are sorted by `median`.
+
+We exclude any runs without `conclusion` or if they have `conclusion: "skipped"`.
+
+🔄 - if you see high numbers here it means your pipeline is flaky, and it does not pass from the first attempt on some commits. It may indicate test flakiness!
+
+🚫 - if numbers are high there it means your pipeline is constantly `cancelled` so it wastes time without bringing any results/value!
+
 ## Examples
 ```
 # GITHUB_TOKEN must have access to "actions" API
 export GITHUB_TOKEN=...
 
-# E2E tests from core, the last day
-ctf ci -r "smartcontractkit/chainlink" -w "Integration Tests"
+# E2E test jobs from core for the last day
+ctf ci -r "smartcontractkit/chainlink" -w "CI Core" -t jobs
+
+# E2E test steps from core for the last day
+ctf ci -r "smartcontractkit/chainlink" -w "CI Core" -t steps
+
+# E2E tests from core for the last week (1k jobs limit!)
+ctf ci -r "smartcontractkit/chainlink" -w "CI Core" -t jobs -s 7
+
+# E2E tests from core for some specific days from 5 days ago to 3 days ago
+ctf ci -r "smartcontractkit/chainlink" -w "CI Core" -t jobs -s 5 -e 3
 
 # Last 3 days runs for e2e framework tests
-ctf ci -r "smartcontractkit/chainlink-testing-framework" -w "Framework Golden Tests Examples" -d 3
+ctf ci -r "smartcontractkit/chainlink-testing-framework" -w "Framework Golden Tests Examples" -t jobs -s 3
 ```
+You can also use `-debug` flag to dump all the workflow runs/jobs to `ctf-ci-debug` folder.