CUDA OOM during evaluation after Epoch 0 on A100 80GB (RFDETRSegSmall Seg) — OOM at lwdetr.py:774 res_i['masks'] = masks_i > 0.0 (alloc 2.32 GiB) even with run_test=False / eval=False

[Aasim-ComputerVision]

Search before asking

• I have searched the RF-DETR issues and found no similar bug report.

Bug

Training of RFDETRSegSmall runs through Epoch 0 (2512 iters) successfully, then an evaluation phase labeled Test: starts and crashes with:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.32 GiB

The stack trace points to mask postprocessing:

rfdetr/models/lwdetr.py, line 774: res_i['masks'] = masks_i > 0.0

Despite passing run_test=False and (from printed args) eval=False, the evaluation still runs at the end of the epoch and triggers the OOM.

Environment

Environment
GPU / Driver

From nvidia-smi:

GPU: NVIDIA A100-SXM4-80GB

Driver: 565.57.01

CUDA: 12.7

nvidia-smi snapshot (from the run) showed:

Memory usage: 72415 MiB / 81920 MiB

GPU-Util: 0%

Processes table: empty (this looked odd because memory was still reported as used)

Timestamp in that output:

Fri Feb 6 21:09:49 2026
Python

From stack paths:

Python 3.11 (/usr/local/lib/python3.11/dist-packages/...)

CPU / RAM

From top snapshot:

load average: 33.87, 37.98, 49.11

CPU: 13.6 us, 1.4 sy, 85.0 id (lots of idle CPU overall)

RAM: 1031853 MiB total, 67125 MiB free, 860969 MiB buff/cache

Swap: 0

Multiple pt_main_thread processes were active; one showed very high CPU% (e.g. 2523%), others ~19–22%.

Minimal Reproducible Example

model.train(
dataset_dir=data_dir,
output_dir=out_dir,
epochs=20,
batch_size=16,
grad_accum_steps=2,
amp=True,
run_test=False,
fp16_eval=True,
eval_max_dets=100,
do_benchmark=False,
num_workers=12,
early_stopping=False,
tensorboard=True
)

Additional

Also printed during training:

Grad accum steps: 2

Total batch size: 32

LENGTH OF DATA LOADER: 2512

Are you willing to submit a PR?

• Yes, I'd like to help by submitting a PR!

[isaacrob]

What's your image resolution?

[Aasim-ComputerVision]

My dataset images are high‑res and mixed: roughly 2K → 4K on disk (I’m not doing any offline resizing/rewriting of the dataset).

For training/eval/inference I’m using the built‑in RF-DETR resizing path (my understanding from the RF-DETR segmentation workflow is that it handles resizing internally), and the run’s printed config shows the model is operating at:

resolution=384 (this is also the resolution I intend to use at inference time)

square_resize_div_64=True

multi_scale=True / expanded_scales=True

Why i did this?
I’ve previously trained the segmentation preview using the same approach (dataset kept at original 2K–4K on disk, training at 384 via the pipeline), and it worked fine trained at 384 and handled the upsampling behavior correctly.

Why I think this is a bug in this release!

Even after setting what should make this OOM-safe:

run_test=False

early_stopping=False

fp16_eval=True

eval_max_dets=100

…the run still enters evaluation (Test: [0/513] ...) right after finishing Epoch 2, and then OOMs immediately at:

rfdetr/models/lwdetr.py:774
res_i['masks'] = masks_i > 0.0
(tries to allocate ~2.32 GiB, fails because GPU is already ~78–79 GiB used)

So it looks like eval-time mask postprocess is still upsampling to original image size (2K–4K) and then the boolean threshold forces an additional allocation and it happens even when run_test=False / eval=False are shown in the printed args.

Question: Is evaluation expected to always upsample masks back to original resolution for COCO eval? If yes, is there a way to:

keep eval masks at the fixed 384 resolution, or

chunk masks / process per-image/per-mask to avoid the large single allocation, or

move the boolean threshold / mask materialization to CPU?

Also: can you confirm which flag is supposed to fully disable evaluation during training for segmentation? Because from the logs, evaluation still runs and is the direct trigger for the OOM.

There are Two additional issues I’m seeing in this release (in case related)

Distributed / multi-GPU training
I’ve tried multiple times but only one GPU gets used. Logs show: “Not using distributed mode” with world_size=1, distributed=False.
Is DDP for segmentation supported in this branch? If yes, what is the exact recommended launcher/flags?

GPU utilization looks bursty / unstable
VRAM climbs and stays very high, but GPU utilization often drops to ~0% between bursts, which makes the run feel “stalled” at times (data time spikes are also visible in logs).

My current machine snapshot:

A100‑SXM4‑80GB

VRAM ~98% (reported as 84 GB / 86 GB)

Utilization 0% at snapshot (then bursts during steps)

Driver 570.195.03

OS Ubuntu 22.04.5 LTS