[Performance] High latency (~1 minute) when hundreds of processes initialize libvgpu.so concurrently

[shizheng163]

[Performance] High latency (~1 minute) when hundreds of processes initialize libvgpu.so concurrently

Summary

In a high-density vGPU deployment scenario, when hundreds of processes start simultaneously and initialize CUDA, each process experiences significant startup delay (up to 1 minute) due to lock contention in libvgpu.so. This severely impacts resource utilization and task throughput.

Environment

Component	Version/Config
HAMi version	v2.8.0
GPU	8x NVIDIA GPU (48GB each)
Kubernetes	1.32.4
Node density	40-50 Pods per node
Processes per Pod	4-5 child processes using vGPU
Total concurrent processes	~200-300 per node

Use Case Description

We run an inference service cluster with the following characteristics:

• Each Pod requests a few GB of GPU memory (e.g., 2-4GB)
• With 8 GPUs × 48GB each = 384GB total, we can run 40-50 Pods per node
• Each Pod consumes tasks from a queue serially
• When processing a task, the Pod spawns multiple child processes that all require GPU memory (vGPU)
• This results in hundreds of processes initializing CUDA simultaneously on a single node

Problem

When a batch of tasks arrives, all Pods start processing and spawn child processes at roughly the same time. We observe:

1. Startup command issued → process should start
2. Actual process start → delayed by ~1 minute
3. Root cause: All processes compete for the global unified_lock in /tmp/vgpulock/lock

Observed Behavior

# In container logs, we see repeated messages:
[HAMI-core Msg]: unified_lock locked, waiting 1 second...
[HAMI-core Msg]: unified_lock locked, waiting 1 second...
[HAMI-core Msg]: unified_lock locked, waiting 1 second...
# ... continues for dozens of iterations

# On host, watching the lock file:
watch -n 0.5 'ls -la /tmp/vgpulock/'
# Lock file appears/disappears rapidly, showing high contention

Using LD_DEBUG to diagnose

time LD_DEBUG=libs,statistics ./our_binary

• Dynamic library loading phase: fast (a few seconds)
• initialize program phase: very slow (~1 minute)

The delay occurs during CUDA initialization inside libvgpu.so, not during library loading.

Impact

Metric	Impact
Process startup latency	Increased from <1s to ~60s
GPU utilization	Decreased due to initialization queuing
Task throughput	Significantly reduced
Resource efficiency	Pods idle while waiting for lock

In a queue-based workload, this means:

• Tasks pile up in the queue
• GPU resources sit idle while processes wait for the lock
• Overall cluster throughput drops dramatically

Root Cause Analysis

After analyzing the source code, we identified the following bottlenecks:

1. unified_lock implementation (libvgpu/src/utils.c)

const char* unified_lock = "/tmp/vgpulock/lock";
const int retry_count = 20;

int try_lock_unified_lock() {
    int fd = open(unified_lock, O_CREAT | O_EXCL, S_IRWXU);
    int cnt = 0;
    while (fd == -1 && cnt <= retry_count) {
        LOG_MSG("unified_lock locked, waiting 1 second...");
        sleep(rand() % 5 + 1);   // Random wait 1-5 seconds!
        cnt++;
        fd = open(unified_lock, O_CREAT | O_EXCL, S_IRWXU);
    }
    // ...
}

Issues:

• Uses file existence (O_CREAT | O_EXCL) as lock mechanism
• Sleep time is 1-5 seconds per retry (too long)
• Maximum 20 retries = potential 100 seconds worst case
• All containers on the node share this single lock via hostPath

2. Lock is acquired unconditionally (libvgpu/src/libvgpu.c)

void postInit() {
    allocator_init();
    map_cuda_visible_devices();
    
    try_lock_unified_lock();      // Called UNCONDITIONALLY
    nvmlReturn_t res = set_task_pid();
    try_unlock_unified_lock();
    
    // env_utilization_switch is set AFTER lock is released
    env_utilization_switch = set_env_utilization_switch();
}

Key finding: The lock acquisition happens before any environment variable checks. This means:

• GPU_CORE_UTILIZATION_POLICY=disable does NOT skip the lock
• disablecorelimit=true does NOT reduce startup lock contention
• Only CUDA_DISABLE_CONTROL=true bypasses this, but it disables all vGPU features

3. Slow operations while holding the lock

set_task_pid() executes expensive NVML/CUDA operations while holding the lock:

• nvmlInit()
• nvmlDeviceGetComputeRunningProcesses() (iterates all GPUs)
• cuDevicePrimaryCtxRetain() (CUDA context creation)

Suggested Improvements

Short-term fixes

1. Reduce sleep time: Change sleep(rand() % 5 + 1) to usleep((rand() % 100 + 10) * 1000) (10-110ms)
2. Use proper locking: Replace O_CREAT | O_EXCL with flock() which supports blocking wait
3. Reduce retry count: 20 retries is excessive

Medium-term improvements

4. Move slow operations outside critical section: Execute NVML/CUDA calls before acquiring lock, only update shared memory while holding lock
5. Add configuration option: Allow users to disable the lock for memory-only isolation scenarios

Example of improved locking

// Use flock instead of file existence
int fd = open(unified_lock, O_CREAT | O_RDWR, 0666);
if (flock(fd, LOCK_EX) == 0) {  // Blocks efficiently without polling
    // Critical section
    flock(fd, LOCK_UN);
}
close(fd);

Related Issues

• unified_lock locked, waiting 1 second #976 - unified_lock locked, waiting 1 second (PyTorch distributed training)
• 部署HAMi-core后，执行测试案例，一直提示unified_lock locked, waiting 1 second... #696 - Repeated lock waiting messages
• gunicorn多worker情况下，使用vgpu出现死锁 #588 - gunicorn multi-worker deadlock

Questions

1. Is there any planned improvement for the lock mechanism in upcoming releases?
2. Is there a configuration option we might have missed that could help our scenario?
3. Would a PR with the suggested improvements be welcome?

Thank you for your time and for the great work on HAMi!

中文内容：

[性能问题] 高并发场景下数百进程同时初始化 libvgpu.so 导致启动延迟约1分钟

问题概述

在高密度 vGPU 部署场景下，当数百个进程同时启动并初始化 CUDA 时，每个进程都会经历严重的启动延迟（最长可达 1 分钟），原因是 libvgpu.so 中的锁竞争。这严重影响了资源利用率和任务吞吐量。

环境信息

组件	版本/配置
HAMi 版本	v2.8.0
GPU	8 × NVIDIA GPU（每卡 48GB）
Kubernetes	1.32.4
单节点 Pod 密度	40-50 个 Pod
每 Pod 进程数	4-5 个子进程使用 vGPU
总并发进程数	每节点约 200-300 个

使用场景描述

我们运行推理服务集群，具有以下特点：

• 每个 Pod 申请几 GB 的 GPU 显存（如 2-4GB）
• 8 张 GPU × 每卡 48GB = 总共 384GB，可以运行 40-50 个 Pod
• 每个 Pod 串行消费队列中的任务
• 处理任务时，Pod 会启动多个子进程，这些子进程都需要 GPU 显存（vGPU）
• 这导致单节点上同时有数百个进程在初始化 CUDA

问题表现

当一批任务到达时，所有 Pod 几乎同时开始处理并启动子进程。我们观察到：

1. 启动命令下发 → 进程应该启动
2. 实际进程启动 → 延迟约 1 分钟
3. 根本原因：所有进程竞争 /tmp/vgpulock/lock 中的全局 unified_lock

观察到的现象

# 容器日志中看到重复的消息：
[HAMI-core Msg]: unified_lock locked, waiting 1 second...
[HAMI-core Msg]: unified_lock locked, waiting 1 second...
[HAMI-core Msg]: unified_lock locked, waiting 1 second...
# ... 持续数十次迭代

# 在宿主机上监控锁文件：
watch -n 0.5 'ls -la /tmp/vgpulock/'
# 锁文件快速出现/消失，显示竞争激烈

使用 LD_DEBUG 诊断

time LD_DEBUG=libs,statistics ./our_binary

• 动态库加载阶段：很快（几秒）
• initialize program 阶段：非常慢（约 1 分钟以上）

延迟发生在 libvgpu.so 内部的 CUDA 初始化过程中，而不是库加载阶段。

影响

指标	影响
进程启动延迟	从 <1 秒增加到 约 60 秒
GPU 利用率	因初始化排队而下降
任务吞吐量	显著降低
资源效率	Pod 在等待锁时处于空闲状态

在基于队列的工作负载中，这意味着：

• 任务在队列中堆积
• GPU 资源在进程等待锁时处于空闲状态
• 整体集群吞吐量大幅下降

根因分析

分析源码后，我们定位到以下瓶颈：

1. unified_lock 实现（libvgpu/src/utils.c）

const char* unified_lock = "/tmp/vgpulock/lock";
const int retry_count = 20;

int try_lock_unified_lock() {
    int fd = open(unified_lock, O_CREAT | O_EXCL, S_IRWXU);
    int cnt = 0;
    while (fd == -1 && cnt <= retry_count) {
        LOG_MSG("unified_lock locked, waiting 1 second...");
        sleep(rand() % 5 + 1);   // 随机等待 1-5 秒！
        cnt++;
        fd = open(unified_lock, O_CREAT | O_EXCL, S_IRWXU);
    }
    // ...
}

问题点：

• 使用文件存在性（O_CREAT | O_EXCL）作为锁机制
• 每次重试的睡眠时间是 1-5 秒（太长）
• 最多 20 次重试 = 最坏情况下 100 秒
• 节点上所有容器通过 hostPath 共享这一把锁

2. 锁获取是无条件的（libvgpu/src/libvgpu.c）

void postInit() {
    allocator_init();
    map_cuda_visible_devices();
    
    try_lock_unified_lock();      // 无条件调用
    nvmlReturn_t res = set_task_pid();
    try_unlock_unified_lock();
    
    // env_utilization_switch 在锁释放后才设置
    env_utilization_switch = set_env_utilization_switch();
}

关键发现： 锁获取发生在任何环境变量检查之前。这意味着：

• GPU_CORE_UTILIZATION_POLICY=disable 不会跳过锁获取
• disablecorelimit=true 不会减少启动时的锁竞争
• 只有 CUDA_DISABLE_CONTROL=true 可以绕过，但它会禁用所有 vGPU 功能

3. 持锁期间执行耗时操作

set_task_pid() 在持有锁的情况下执行昂贵的 NVML/CUDA 操作：

• nvmlInit()
• nvmlDeviceGetComputeRunningProcesses()（遍历所有 GPU）
• cuDevicePrimaryCtxRetain()（CUDA Context 创建）

改进建议

短期修复

1. 减少睡眠时间：将 sleep(rand() % 5 + 1) 改为 usleep((rand() % 100 + 10) * 1000)（10-110 毫秒）
2. 使用合适的锁机制：用 flock() 替代 O_CREAT | O_EXCL，支持阻塞等待
3. 减少重试次数：20 次重试过多

中期改进

4. 将耗时操作移出临界区：在获取锁之前执行 NVML/CUDA 调用，只在更新共享内存时持有锁
5. 添加配置选项：允许用户在不需要某些功能时跳过锁

改进后的锁机制示例

// 使用 flock 替代文件存在性检查
int fd = open(unified_lock, O_CREAT | O_RDWR, 0666);
if (flock(fd, LOCK_EX) == 0) {  // 高效阻塞，无需轮询
    // 临界区
    flock(fd, LOCK_UN);
}
close(fd);

相关 Issue

• unified_lock locked, waiting 1 second #976 - unified_lock locked, waiting 1 second（PyTorch 分布式训练）
• 部署HAMi-core后，执行测试案例，一直提示unified_lock locked, waiting 1 second... #696 - 重复的锁等待消息
• gunicorn多worker情况下，使用vgpu出现死锁 #588 - gunicorn 多 worker 死锁

问题

1. 后续版本是否有计划改进锁机制？
2. 是否有我们可能遗漏的配置选项可以帮助解决这个场景？
3. 如果我们提交包含上述改进的 PR，是否会被接受？

感谢您的时间，也感谢 HAMi 团队的出色工作！

[shizheng163]

HAMi 高并发进程启动延迟优化方案

作者：Zelos
日期：2026 年 2 月
状态：已在业务集群验证通过

前言

本文分享一套针对 HAMi 高并发场景下进程启动延迟问题的优化方案。该方案已在我们的业务集群中测试通过，效果显著。

由于提交 PR 涉及多个仓库（hami-core / HAMi 主仓库）且流程较为繁琐，加之之前无相关经验，这里仅做思路分享，供社区参考讨论。

---

问题背景

场景描述

在 AI 训练/推理场景中，经常需要在单个容器内启动大量进程并发使用 GPU。例如：

• 分布式训练框架启动多个 worker 进程
• 推理服务预热时并发初始化多个模型实例
• 数据预处理 pipeline 的多进程并行

问题现象

当容器内同时启动 200+ 进程 调用 CUDA 时，整体启动时间从预期的几秒延长到 30 秒以上，严重影响业务启动速度和弹性伸缩效率。

根因分析

HAMi 的 libvgpu.so 在 cuInit() 时需要获取当前进程的 宿主机 PID (hostpid)，用于后续的资源监控和利用率统计。原有实现存在三个瓶颈：

┌─────────────────────────────────────────────────────────────────┐
│  瓶颈 1: 文件锁实现低效                                          │
│  ─────────────────────                                          │
│  使用 O_CREAT|O_EXCL 创建文件作为锁，获取失败后 sleep 100ms 重试    │
│  问题：非公平、CPU 空转、200 进程竞争时退化严重                     │
├─────────────────────────────────────────────────────────────────┤
│  瓶颈 2: 全局锁竞争                                              │
│  ─────────────────                                              │
│  所有 GPU 共用一把锁 /tmp/vgpulock/lock                           │
│  问题：8 卡机器上 8 个容器互相阻塞，并行度为 1                      │
├─────────────────────────────────────────────────────────────────┤
│  瓶颈 3: hostpid 获取耗时                                        │
│  ──────────────────────                                         │
│  每个进程需要：加锁 → NVML 快照 → cuInit → NVML 快照 → 对比 → 解锁  │
│  单次耗时 60-220ms，200 进程串行 = 30s+                           │
└─────────────────────────────────────────────────────────────────┘

---

优化方案

方案一：文件锁机制优化 (flock 替代轮询)

原方案

int try_lock() {
    for (int i = 0; i < retry_count; i++) {
        // 使用 O_CREAT|O_EXCL 原子创建文件作为锁
        int fd = open(lock_path, O_CREAT | O_EXCL | O_RDWR, 0666);
        if (fd >= 0) {
            return 0;  // 获取锁成功
        }
        // 获取失败，随机 sleep 后重试
        usleep(100000);  // 100ms
    }
    return -1;
}

问题：

• 非公平调度：后来的进程可能先获得锁
• CPU 空转：sleep 期间浪费 CPU 时间片
• 无法充分利用锁释放的时机

优化方案

int try_lock() {
    // 打开（或创建）锁文件
    int fd = open(lock_path, O_CREAT | O_RDWR, 0666);
    if (fd == -1) return -1;
    
    // 使用 flock 阻塞等待 —— 内核级高效等待
    if (flock(fd, LOCK_EX) == -1) {
        close(fd);
        return -1;
    }
    
    unified_lock_fd = fd;
    return 0;
}

int try_unlock() {
    flock(unified_lock_fd, LOCK_UN);
    close(unified_lock_fd);
    unified_lock_fd = -1;
    return 0;
}

优势：

• 内核级阻塞：无 CPU 空转，内核在锁释放时唤醒等待进程
• 公平调度：按 FIFO 顺序获取锁
• 即时响应：锁释放后立即获取，无 sleep 延迟

---

方案二：锁粒度优化 (Per-GPU Lock)

原方案

所有进程竞争同一把全局锁：

/tmp/vgpulock/lock  ← 全局唯一

8 卡机器上，8 个容器的进程全部串行化。

优化方案

根据 NVIDIA_VISIBLE_DEVICES 环境变量确定当前进程使用的 GPU，为每个 GPU 创建独立的锁文件：

static const char* get_first_gpu_id() {
    static char first_gpu[128];
    char *env = getenv("NVIDIA_VISIBLE_DEVICES");
    if (env == NULL) return NULL;
    
    // 提取第一个 GPU 标识（支持 UUID 或索引格式）
    // GPU-xxxxxx,GPU-yyyyyy → GPU-xxxxxx
    // 0,1,2 → 0
    char *comma = strchr(env, ',');
    if (comma) {
        strncpy(first_gpu, env, comma - env);
        first_gpu[comma - env] = '\0';
    } else {
        strncpy(first_gpu, env, sizeof(first_gpu) - 1);
    }
    return first_gpu;
}

int try_lock() {
    char lock_path[256];
    const char *gpu_id = get_first_gpu_id();
    
    if (gpu_id != NULL && strcmp(gpu_id, "all") != 0) {
        // Per-GPU 锁
        snprintf(lock_path, sizeof(lock_path), "/tmp/vgpulock/lock_%s", gpu_id);
    } else {
        // 兜底：全局锁
        snprintf(lock_path, sizeof(lock_path), "/tmp/vgpulock/lock");
    }
    
    // ... flock 逻辑 ...
}

技术要点：

nvidia-container-runtime 会设置 NVIDIA_VISIBLE_DEVICES 环境变量，格式如下：

• UUID 格式：GPU-xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
• 索引格式：0 或 0,1,2
• 特殊值：all、none

容器内通过 NVML 只能看到被隔离后的设备（从 0 开始编号），但环境变量中保留了宿主机上的真实标识，可以用于区分不同 GPU。

效果：8 卡机器并行度从 1 提升到 8。

---

方案三：Unix Socket + SO_PEERCRED (无锁方案)

核心思路

既然获取 hostpid 是瓶颈，能否绕过 NVML 快照对比？

关键洞察：Linux 内核的 SO_PEERCRED 机制可以获取 Unix Socket 连接对端进程的凭证（PID、UID、GID），且这个 PID 是宿主机视角的 PID。

┌──────────────────────┐      Unix Socket      ┌──────────────────────┐
│   业务容器           │  ─────────────────▶   │   vgpu-monitor       │
│   (namespace PID=1)  │                       │   (hostPID: true)    │
│                      │  ◀─────────────────   │                      │
│   hostpid=12345      │      SO_PEERCRED      │   ucred.Pid=12345    │
└──────────────────────┘      返回 hostpid     └──────────────────────┘

架构设计

vgpu-monitor (DaemonSet, hostPID: true)
    │
    ├── 监听: /tmp/vgpulock/hami.sock
    │
    └── 处理连接:
        1. 通过 SO_PEERCRED 获取对端 hostpid
        2. 返回 hostpid 给客户端

libvgpu.so (业务容器内)
    │
    ├── cuInit() 时
    │   ├── 1. 尝试连接 /tmp/vgpulock/hami.sock
    │   ├── 2. 发送请求，接收 hostpid
    │   └── 3. 失败则降级到 flock + NVML 方案
    │
    └── 无需加锁，完全并行

可扩展协议设计

考虑到未来可能需要扩展更多信令（如 GPU 信息查询、心跳等），设计了可扩展的协议格式：

请求格式:
+------------+------------+-------------------+
| cmd (2B)   | length (2B)|  data (可选)      |
+------------+------------+-------------------+

响应格式:
+------------+------------+-------------------+
| status (2B)| length (2B)|  data             |
+------------+------------+-------------------+

当前命令:
  CMD_GET_HOSTPID = 0x0001  → 响应 4 字节 hostpid

预留扩展:
  CMD_GET_GPU_INFO = 0x0002
  CMD_HEARTBEAT    = 0x0003
  ...

扩展新信令时：

• 服务端：在 switch 中添加新 case
• 客户端：添加新的请求函数

无需修改协议解析框架。

降级策略

为保证向后兼容和高可用，设计了完善的降级逻辑：

void postInit() {
    // 1. 优先尝试 Socket 方式（无锁、高性能）
    int hostpid = get_hostpid_from_socket();
    if (hostpid > 0) {
        set_host_pid(hostpid);
        LOG_INFO("Initialized (socket method, hostpid=%d)", hostpid);
        return;
    }
    
    // 2. 降级到 flock + NVML 方案
    LOG_INFO("Socket unavailable, falling back to lock-based method");
    try_lock_unified_lock();
    set_task_pid();  // NVML 快照对比
    try_unlock_unified_lock();
}

降级触发条件：

• Socket 文件不存在（旧版本 vgpu-monitor）
• 连接超时/失败
• 协议错误

服务更新期间的可靠性

问题：vgpu-monitor 重启时，socket 文件可能暂时不可用，如何避免大量进程降级？

解决方案：更新标记文件 + 重试机制

// 服务端：vgpu-monitor 启动时
func (s *HostPIDServer) Start(ctx context.Context) error {
    // 1. 创建更新标记文件
    os.Create("/tmp/vgpulock/.updating")
    
    // 2. 创建 socket listener
    listener, _ := net.Listen("unix", "/tmp/vgpulock/hami.sock")
    
    // 3. 准备就绪，删除更新标记
    os.Remove("/tmp/vgpulock/.updating")
    
    // 4. 开始接受连接
    ...
}

// 客户端：libvgpu.so 重试逻辑
#define MAX_SOCKET_RETRY 3
#define SOCKET_RETRY_INTERVAL_US 100000  // 100ms
#define UPDATING_FLAG_MAX_AGE_SEC 30     // 标记文件最大有效期

int get_hostpid_from_socket() {
    for (int retry = 0; retry < MAX_SOCKET_RETRY; retry++) {
        int sock_exists = (access(SOCKET_PATH, F_OK) == 0);
        int updating = is_updating_flag_valid();  // 检查标记文件是否存在且未过期
        
        // 快速失败：socket 不存在且不在更新
        if (!sock_exists && !updating) {
            return -1;  // 触发降级
        }
        
        // 正在更新，等待重试
        if (!sock_exists && updating) {
            LOG_INFO("Service is updating, waiting...");
            usleep(SOCKET_RETRY_INTERVAL_US);
            continue;
        }
        
        // 尝试连接
        int hostpid = try_get_hostpid_once();
        if (hostpid > 0) return hostpid;
        
        // 连接失败，重试
        usleep(SOCKET_RETRY_INTERVAL_US);
    }
    return -1;  // 触发降级
}

// 检查更新标记文件是否有效（防止服务崩溃导致标记遗留）
static int is_updating_flag_valid() {
    struct stat st;
    if (stat(UPDATING_FLAG_PATH, &st) != 0) return 0;
    
    time_t file_age = time(NULL) - st.st_mtime;
    if (file_age > UPDATING_FLAG_MAX_AGE_SEC) {
        LOG_WARN("Stale updating flag (age=%lds), ignoring", file_age);
        return 0;  // 标记文件过期，视为无效
    }
    return 1;
}

设计要点：

• 标记文件 30 秒有效期：防止服务崩溃导致标记文件遗留
• 最多重试 3 次，间隔 100ms：覆盖正常重启时间窗口
• 优雅降级：重试失败后回退到 lock-based 方案

---

性能对比

单次 hostpid 获取耗时

方案	耗时	说明
原方案 (NVML 快照)	60-220ms	含 CUDA Context 创建开销
方案三 (Unix Socket)	~10 μs	纯内核内存操作

200 进程并发启动总耗时

方案	耗时	说明
原方案	30s+	全局锁串行
方案一 + 方案二	~4s	flock + Per-GPU 锁
方案三	< 1s	完全并行，无锁

各操作开销实测（参考值）

操作	耗时
access() 检查文件	0.5-0.6 μs
stat() 读取文件属性	0.5 μs
Unix Socket 完整通信	8 μs
Socket 方案总开销	~10 μs

---

兼容性考虑

向后兼容

场景	行为
旧 vgpu-monitor（无 socket 服务）	自动降级到 lock-based
新 libvgpu + 旧 vgpu-monitor	正常工作（降级）
旧 libvgpu + 新 vgpu-monitor	正常工作（不使用 socket）

升级路径

1. 先升级 vgpu-monitor（提供 socket 服务）
2. 再升级 libvgpu（使用 socket 服务）
3. 任一组件可独立回滚

---

涉及文件

文件	改动
cmd/vGPUmonitor/hostpid_server.go	新增 Socket 服务
cmd/vGPUmonitor/main.go	启动 Socket 服务
libvgpu/src/utils.c	flock、Per-GPU 锁、Socket 客户端
libvgpu/src/libvgpu.c	postInit 降级逻辑

---

总结

本方案通过三个层次的优化，将 200 进程并发启动耗时从 30 秒以上 降低到 1 秒以内：

1. flock 替代轮询：消除 CPU 空转，实现公平高效的锁等待
2. Per-GPU 锁：提升 8 卡机器的并行度从 1 到 8
3. Unix Socket + SO_PEERCRED：彻底消除锁竞争，实现完全并行的 hostpid 获取

方案设计充分考虑了向后兼容、优雅降级、服务更新等生产环境因素。

---

反馈与讨论

欢迎社区开发者讨论交流：

• 是否有更优雅的实现方式？
• 是否有其他场景需要考虑？
• 如果社区认可这个方向，我可以尝试学习提交 PR 的流程

感谢阅读！

[wawa0210]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

HAMi high-concurrency process startup delay optimization solution

Author: Zelos
Date: February 2026
Status: Passed verification in business cluster

Preface

This article shares a set of optimization solutions for the process startup delay problem in HAMi high concurrency scenarios. This solution has been tested in our business cluster and has achieved remarkable results.

Since submitting a PR involves multiple warehouses (hami-core / HAMi main warehouse) and the process is relatively cumbersome, and there is no relevant experience before, here is only ideas sharing for community reference and discussion.

---

Problem background

Scene description

In AI training/inference scenarios, it is often necessary to launch a large number of processes within a single container to use the GPU concurrently. For example:

• Distributed training framework starts multiple worker processes
• Concurrently initialize multiple model instances when the inference service is warmed up
• Multi-process parallelization of data preprocessing pipeline

Problem phenomenon

When 200+ processes are started simultaneously in the container and CUDA is called, the overall startup time is extended from the expected few seconds to more than 30 seconds, seriously affecting the business startup speed and elastic scaling efficiency.

Root cause analysis

HAMi's libvgpu.so needs to obtain the host PID (hostpid) of the current process during cuInit() for subsequent resource monitoring and utilization statistics. There are three bottlenecks in the original implementation:

┌────────────────────────────────────────────────────────────┐
│ Bottleneck 1: Inefficient file lock implementation │
│ ───────────────────── │
│ Use O_CREAT|O_EXCL to create a file as a lock. After the acquisition fails, sleep 100ms and try again │
│ Problems: unfairness, CPU idling, serious degradation when 200 processes compete │
├────────────────────────────────────────────────────────────┤
│ Bottleneck 2: Global lock competition │
│ ───────────────── │
│ All GPUs share a lock /tmp/vgpulock/lock │
│ Problem: 8 containers on an 8-card machine block each other, and the parallelism is 1 │
├────────────────────────────────────────────────────────────┤
│Bottleneck 3: Time consuming to obtain hostpid │
│ ────────────────────── │
│ Each process requires: lock → NVML snapshot → cuInit → NVML snapshot → compare → unlock │
│ Single time consumption 60-220ms, 200 processes serial = 30s+ │
└────────────────────────────────────────────────────────────┘

---

Optimization plan

Solution 1: File lock mechanism optimization (flock replaces polling)

Original plan

int try_lock() {
    for (int i = 0; i < retry_count; i++) {
        // Use O_CREAT|O_EXCL to create the file atomically as a lock
        int fd = open(lock_path, O_CREAT | O_EXCL | O_RDWR, 0666);
        if (fd >= 0) {
            return 0; // Obtain lock successfully
        }
        // Failed to obtain, try again after random sleep
        usleep(100000); // 100ms
    }
    return -1;
}

Question:

• Unfair scheduling: later processes may acquire the lock first
• CPU idling: CPU time slices are wasted during sleep
• Unable to take full advantage of lock release opportunities

Optimization plan

int try_lock() {
    //Open (or create) lock file
    int fd = open(lock_path, O_CREAT | O_RDWR, 0666);
    if (fd == -1) return -1;
    
    // Use flock to block waiting - kernel-level efficient waiting
    if (flock(fd, LOCK_EX) == -1) {
        close(fd);
        return -1;
    }
    
    unified_lock_fd = fd;
    return 0;
}

int try_unlock() {
    flock(unified_lock_fd, LOCK_UN);
    close(unified_lock_fd);
    unified_lock_fd = -1;
    return 0;
}

Advantages:

• Kernel-level blocking: No CPU idling, the kernel wakes up waiting processes when the lock is released
• Fair Scheduling: acquire locks in FIFO order
• Instant response: Acquire immediately after the lock is released, no sleep delay

---

Solution 2: Lock granularity optimization (Per-GPU Lock)

Original plan

All processes compete for the same global lock:

/tmp/vgpulock/lock ← Globally unique

On an 8-card machine, the processes of all 8 containers are serialized.

Optimization plan

Determine the GPU used by the current process according to the NVIDIA_VISIBLE_DEVICES environment variable and create a separate lock file for each GPU:

static const char* get_first_gpu_id() {
    static char first_gpu[128];
    char *env = getenv("NVIDIA_VISIBLE_DEVICES");
    if (env == NULL) return NULL;
    
    // Extract the first GPU identifier (supports UUID or index format)
    // GPU-xxxxxx,GPU-yyyyyy → GPU-xxxxxx
    // 0,1,2 → 0
    char *comma = strchr(env, ',');
    if (comma) {
        strncpy(first_gpu, env, comma - env);
        first_gpu[comma - env] = '\0';
    } else {
        strncpy(first_gpu, env, sizeof(first_gpu) - 1);
    }
    return first_gpu;
}

int try_lock() {
    char lock_path[256];
    const char *gpu_id = get_first_gpu_id();
    
    if (gpu_id != NULL && strcmp(gpu_id, "all") != 0) {
        // Per-GPU lock
        snprintf(lock_path, sizeof(lock_path), "/tmp/vgpulock/lock_%s", gpu_id);
    } else {
        // Bottom line: global lock
        snprintf(lock_path, sizeof(lock_path), "/tmp/vgpulock/lock");
    }
    
    // ... flock logic ...
}

Technical Points:

nvidia-container-runtime will set the NVIDIA_VISIBLE_DEVICES environment variable with the following format:

• UUID format: GPU-xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
• Index format: 0 or 0,1,2
• Special values: all, none

Only the isolated devices (numbered starting from 0) can be seen in the container through NVML, but the real identification on the host is retained in the environment variables, which can be used to distinguish different GPUs.

Effect: The parallelism of 8-card machines is increased from 1 to 8.

---

Solution three: Unix Socket + SO_PEERCRED (lock-free solution)

Core idea

Since getting hostpid is the bottleneck, can NVML snapshot comparison be bypassed?

Key Insight: The SO_PEERCRED mechanism of the Linux kernel can obtain the credentials (PID, UID, GID) of the Unix Socket connection peer process, and this PID is the PID from the host's perspective.

┌─────────────────────┐ Unix Socket ┌────────────────────────┐
│ Business container │ ─────────────────▶ │ vgpu-monitor │
│ (namespace PID=1) │ │ (hostPID: true) │
│ │ ◀───────────────── │ │
│ hostpid=12345 │ SO_PEERCRED │ ucred.Pid=12345 │
└──────────────────────┘ Return to hostpid └────────────────────────┘

Architecture design

vgpu-monitor (DaemonSet, hostPID: true)
    │
    ├── Monitor: /tmp/vgpulock/hami.sock
    │
    └── Handling connections:
        1. Obtain the peer hostpid through SO_PEERCRED
        2. Return hostpid to the client

libvgpu.so (in the business container)
    │
    ├── when cuInit()
    │ ├── 1. Try to connect to /tmp/vgpulock/hami.sock
    │ ├── 2. Send request and receive hostpid
    │ └── 3. If failed, downgrade to flock + NVML solution
    │
    └── No locking required, completely parallel

Extensible protocol design

Considering that more signaling may need to be expanded in the future (such as GPU information query, heartbeat, etc.), an extensible protocol format is designed:

Request format:
+------------+------------+-----------+
| cmd (2B) | length (2B) | data (optional) |
+------------+------------+-----------+

Response format:
+------------+------------+-----------+
| status (2B)| length (2B)| data |
+------------+------------+-----------+

Current command:
  CMD_GET_HOSTPID = 0x0001 → response 4 bytes hostpid

Reserved extensions:
  CMD_GET_GPU_INFO = 0x0002
  CMD_HEARTBEAT = 0x0003
  ...

When extending new signaling:

• Server side: add new case in switch
• Client: Add new request function

No need to modify the protocol parsing framework.

Downgrade strategy

To ensure backward compatibility and high availability, complete downgrade logic is designed:

void postInit() {
    // 1. Try the Socket method first (lock-free, high performance)
    int hostpid = get_hostpid_from_socket();
    if (hostpid > 0) {
        set_host_pid(hostpid);
        LOG_INFO("Initialized (socket method, hostpid=%d)", hostpid);
        return;
    }
    
    // 2. Downgrade to flock + NVML solution
    LOG_INFO("Socket unavailable, falling back to lock-based method");
    try_lock_unified_lock();
    set_task_pid(); // NVML snapshot comparison
    try_unlock_unified_lock();
}

Downgrade trigger conditions:

• Socket file does not exist (old version vgpu-monitor)
• Connection timeout/failure
• protocol error

Reliability during service updates

Question: When vgpu-monitor restarts, the socket file may be temporarily unavailable. How to avoid the downgrade of a large number of processes?

Solution: Update tag file + retry mechanism

// Server side: vgpu-monitor when starting
func (s *HostPIDServer) Start(ctx context.Context) error {
    // 1. Create update mark file
    os.Create("/tmp/vgpulock/.updating")
    
    // 2. Create socket listener
    listener, _ := net.Listen("unix", "/tmp/vgpulock/hami.sock")
    
    // 3. When ready, delete the update mark
    os.Remove("/tmp/vgpulock/.updating")
    
    // 4. Start accepting connections
    ...
}

// Client: libvgpu.so retry logic
#define MAX_SOCKET_RETRY 3
#define SOCKET_RETRY_INTERVAL_US 100000 // 100ms
#define UPDATING_FLAG_MAX_AGE_SEC 30 // Maximum validity period of marked files

int get_hostpid_from_socket() {
    for (int retry = 0; retry < MAX_SOCKET_RETRY; retry++) {
        int sock_exists = (access(SOCKET_PATH, F_OK) == 0);
        int updating = is_updating_flag_valid(); // Check whether the flag file exists and has not expired
        
        // Fast failure: socket does not exist and is not being updated
        if (!sock_exists && !updating) {
            return -1; // trigger downgrade
        }
        
        //Updating, waiting to retry
        if (!sock_exists && updating) {
            LOG_INFO("Service is updating, waiting...");
            usleep(SOCKET_RETRY_INTERVAL_US);
            continue;
        }
        
        //try to connect
        int hostpid = try_get_hostpid_once();
        if (hostpid > 0) return hostpid;
        
        // Connection failed, try again
        usleep(SOCKET_RETRY_INTERVAL_US);
    }
    return -1; // trigger downgrade
}

// Check whether the updated mark file is valid (to prevent the service from crashing and causing the mark to be left behind)
static int is_updating_flag_valid() {
    struct stat st;
    if (stat(UPDATING_FLAG_PATH, &st) != 0) return 0;
    
    time_t file_age = time(NULL) - st.st_mtime;
    if (file_age > UPDATING_FLAG_MAX_AGE_SEC) {
        LOG_WARN("Stale updating flag (age=%lds), ignoring", file_age);
        return 0; // Mark the file as expired and considered invalid
    }
    return 1;
}

Design Points:

• Marked files have a 30-second validity period: to prevent service crashes from leaving marked files behind
• Retry up to 3 times with 100ms interval: covering the normal restart time window
• Graceful degradation: fall back to lock-based solution after retry failure

---

Performance comparison

Time taken to obtain a single hostpid

Solution	Time consuming	Description
Original solution (NVML snapshot)	60-220ms	Including CUDA Context creation overhead
Solution 3 (Unix Socket)	~10 μs	Pure kernel memory operation

Total time taken for concurrent startup of 200 processes

Solution	Time consuming	Description
Original plan	30s+	Global lock serial
Option 1 + Option 2	~4s	flock + Per-GPU lock
Option 3	< 1s	Fully parallel, no locks

Actual measurement of each operation overhead (reference value)

Operation	Time consuming
access() Check file	0.5-0.6 μs
stat() reads file attributes	0.5 μs
Unix Socket complete communication	8 μs
Total overhead of Socket solution	~10 μs

---

Compatibility considerations

Backwards Compatibility

Scene	Behavior
Old vgpu-monitor (no socket service)	Automatic downgrade to lock-based
new libvgpu + old vgpu-monitor	works fine (downgraded)
old libvgpu + new vgpu-monitor	works fine (not using socket)

Upgrade path

1. Upgrade vgpu-monitor first (providing socket service)
2. Then upgrade libvgpu (use socket service)
3. Any component can be rolled back independently

---

Involved files

Documentation	Changes
cmd/vGPUmonitor/hostpid_server.go	Added Socket service
cmd/vGPUmonitor/main.go	Start Socket service
libvgpu/src/utils.c	flock, Per-GPU lock, Socket client
libvgpu/src/libvgpu.c	postInit downgrade logic

---

Summary

Through three levels of optimization, this solution reduces the concurrent startup time of 200 processes from more than 30 seconds to less than 1 second:

1. flock replaces polling: eliminates CPU idling and achieves fair and efficient lock waiting
2. Per-GPU lock: Increase the parallelism of 8-card machines from 1 to 8
3. Unix Socket + SO_PEERCRED: Completely eliminate lock competition and achieve fully parallel hostpid acquisition

The solution design fully takes into account production environment factors such as Backward Compatibility, Graceful Downgrade, and Service Update.

---

Feedback and Discussion

Community developers are welcome to discuss and exchange:

• Is there a more elegant way to do this?
• Are there other scenarios to consider?
• If the community recognizes this direction, I can try to learn the process of submitting a PR

Thanks for reading!