Add scheduled nightly E2E test runs for continuous integration

[vdemeester]

Problem Statement

Currently, the tektoncd/pipeline repository has comprehensive E2E test workflows (e2e-matrix.yml and e2e-matrix-extras.yaml) but they only run:

• On pull requests
• On repository dispatch events
• Via manual workflow triggers

Unlike the nightly-builds.yaml workflow which runs on a schedule (03:00 UTC daily), there is no scheduled E2E test execution against the main branch. This means:

• We don't proactively catch integration issues on main
• We can't track E2E test stability trends over time
• We lack a consistent baseline for E2E test health

Proposed Solution

Add a new workflow nightly-e2e.yaml that runs the full E2E matrix on a schedule against the main branch.

Implementation

name: Nightly E2E Tests
on:
  schedule:
    - cron: '0 3 * * *'  # 03:00 UTC daily
  workflow_dispatch:

jobs:
  e2e-matrix:
    uses: ./.github/workflows/e2e-matrix.yml
    with:
      # Run full matrix
      k8s-versions: '["v1.28.x", "v1.29.x", "v1.30.x", "v1.31.x"]'
      feature-flags: '["stable", "beta", "alpha"]'

Benefits

1. Proactive Issue Detection: Catch breaking changes and flaky tests before they impact development
2. Trend Analysis: Track E2E test stability and performance over time
3. Artifact Availability: Consistent artifacts for debugging and analysis
4. Main Branch Health: Ensure main branch is always in a releasable state
5. Developer Workflow: Easier to download latest E2E results for local debugging

Artifact Naming

The nightly run would produce artifacts following the existing pattern:

• v1.28.x-stable-ubuntu-latest
• v1.28.x-beta-ubuntu-latest
• v1.28.x-alpha-ubuntu-latest
• etc.

Developers could download artifacts using:

# Find latest scheduled run
runid=$(gh run list --workflow=nightly-e2e.yaml | head -1 | awk '{print $7}')

# Download specific artifact
gh run download "$runid" --pattern "v1.28.x-stable-ubuntu-latest"

Related Workflows

This follows the pattern established by nightly-builds.yaml but focuses on E2E testing rather than artifact building.

Alternatives Considered

1. Use nightly-builds.yaml: This workflow is focused on building and releasing, not comprehensive E2E testing
2. Increase PR E2E frequency: This doesn't help with main branch health monitoring
3. Manual triggers only: Requires human intervention and isn't consistent

Additional Context

This would align tektoncd/pipeline with similar projects (e.g., openshift-pipelines/pipelines-as-code) that run scheduled E2E tests for continuous integration monitoring.

[7h3-3mp7y-m4n]

Can I please work on this @vdemeester , I would really love to do it :)

[vdemeester]

/assign @7h3-3mp7y-m4n