GPUSitter

Watch for idle GPUs and run your jobs: launches jobs in tmux, keeps logs/status and sends start/finish emails..

---

UPDATE: If the num of free gpus is larger than the required, we can use all the free gpu. In addition, it can find the CUDA ID automatically, we do not need to specify the CUDA.

Features

• Real-time GPU usage monitoring
• Command-line interface, easy to integrate into workflows
• Email notifications
• Scheduled automatic job running

Dependencies

• tmux

Installation

pip install gpusitter

Usage

Important

Make sure the job environment (especially the Python environment) is correctly set up before running gpust. There are two common ways to do this:

1. Activate your environment before running gpust. e.g. conda activate xxx, source .venv/bin/activate. (If you are using uv to manage environments, you can also run: uv run gpust)
2. Specify the Python path directly in the job command, e.g. gpust --job="~/myproject/.venv/bin/python train.py"

# One job with 1 gpu
gpust --job="python train.py"

# One job with 4 gpus
gpust --job="python train.py:4"

# Two jobs with 1 gpu and 4 gpus respectively
gpust --job="python train.py" --job="python train.py --epoch=12 --lr=-.001:4"

# With CUDA_VISIBLE_DEVICES env
CUDA_VISIBLE_DEVICES=2 gpust --job="python train.py"

# With different python envs
gpust --job="~/job1/.venv/bin/python train1.py" --job="~/job2/.venv/bin/python train2.py"

After starting your job, you can monitor its progress using tmux.

# List all running tmux sessions
tmux ls

# Attach to your job session (replace GPUSitter_xxx_xx with your session name)
tmux a -t GPUSitter_xxx_xx

Parameter description:

class ConfigData:
    """Configuration data for GPU Snatcher."""

    gpu_free_memory_ratio_threshold: float
    friendly_min: float
    email_host: str
    email_user: str
    email_pwd: str
    email_sender: str
    email_receivers: list[str]

• gpu_free_memory_ratio_threshold: The minimum free GPU memory ratio required to consider a GPU available. Only GPUs with free memory above this threshold will be used.
• friendly_min: Waiting time (in seconds) before allocating GPUs. Helps prevent OOM from previous jobs.
• email_host: Email server, e.g., smtp.qq.com
• email_user: Email address
• email_pwd: SMTP authorization code
• email_sender: Sender
• email_receivers: Recipients

Contribution

Issues and pull requests are welcome. Please follow the project's code style guidelines.

License

This project is licensed under the MIT License.