Skip to content

E2E tests killed by OOM on self-hosted runners #5202

@paolino

Description

@paolino

Summary

Linux E2E tests are consistently killed by the kernel OOM killer on self-hosted runners (builder-new). The e2e process is terminated by the memory cgroup limit on the gha-runner service.

Evidence

From dmesg on builder-new:

oom-kill: constraint=CONSTRAINT_MEMCG, cpuset=system-gha\x2drunner.slice
  task_memcg=/system.slice/system-gha\x2drunner.slice/gha-runner@9.service
  task=e2e, pid=766866
Memory cgroup out of memory: Killed process 766866 (e2e)

Options

  1. Increase MemoryMax on gha-runner systemd service
  2. Split E2E into smaller jobs with lower peak memory
  3. Add swap as overflow buffer

Status

Metadata

Metadata

Assignees

Labels

BugTestsAdding additional tests

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions