[algo, cfg, worker, doc] feat: add SOAP optimizer support

[CedricHwong]

What does this PR do?

Add SOAP optimizer support and a paper-default config. Includes the optimizer implementation,
config template, docs, and a unit test.
Paper: https://arxiv.org/abs/2409.11321
Closes #4680

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here:
• Format the PR title as [{modules}] {type}: {description}

Test

• pre-commit run --all-files
• pytest -s -x tests/special_sanity
• python -m pytest tests/workers/config/test_build_optimizer_soap.py -q

API and Usage Example

# Paper defaults
optim:
  optimizer: SOAP
  optimizer_impl: verl.optimizers.soap
  lr: 3e-3
  betas: [0.95, 0.95]
  override_optimizer_config:
    precondition_frequency: 10
    max_precond_dim: 10000
    merge_dims: false
    precondition_1d: false

You can also set optim: soap to use trainer/config/optim/soap.yaml.

Design & Code Changes

• Add verl/optimizers/soap.py and export it in verl/optimizers/init.py (MIT license
  preserved).
• Ensure build_optimizer passes betas for SOAP in verl/workers/config/optimizer.py.
• Add paper-default config verl/trainer/config/optim/soap.yaml.
• Update docs and add unit test tests/workers/config/test_build_optimizer_soap.py.

Checklist Before Submitting

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit run --all-files
• Add / Update documentation.
• Add unit or end-to-end test(s) to CI workflow: tests/workers/config/
  test_build_optimizer_soap.py (picked up by gpu_unit_tests)
• Once ready for CI, send a message in the ci-request channel.

[gemini-code-assist]

Code Review

This pull request introduces the SOAP optimizer, including its implementation, configuration, documentation, and a unit test. The implementation is adapted from an external source and seems robust. My main feedback is to improve exception handling by replacing a bare except clause with a specific exception type to avoid catching unintended errors and improve code clarity and safety.

[CedricHwong]

@vermouth1992 The commit 44682a3 updates verl/optimizers/soap.py to replace the bare except in the torch.linalg.eigh fallback so ruff’s E722 check passes.

PYTHONPATH=. pytest tests/workers/config/test_build_optimizer_soap.py
└ /usr/local/lib/python3.12/dist-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources
… +12 lines

============================== 1 passed in 8.66s ===============================