Fix Backward Compatibility for Convergence Test

[Mecoli1219]

Summary

This PR restores backward compatibility for convergence tests with transformers v4 (v4.49.0 ~ v4.57.6). During the initial development phase for transformers v5 support, backward compatibility was intentionally deprioritized, leading to significant test regressions. This PR fixes those regressions while maintaining a stable foundation for the ongoing v5 integration.

Related Issues & PRs

• Transformers v5 compatibility #978
• Support transfomers v5 #994

Details

The current codebase assumes transformers v5 conventions, which broke compatibility with the v4.x series in two major areas:

1. RoPE Parameters: Some model miss some rope parameters (rope_scaling) since they are unified to rope_parameters in transformer v5.
2. Tokenizer Consistency: v5 and v4 handle the Tokenizer interfaces differently. V5's Tokenizer will select the appropriate backend, while v4's Tokenizer is the python-based implementation using SentencePiece as backend.

Key Fixes:

• Added conditional logic to provide different rope parameters for different transformers versions.
• Enforced TokenizerFast usage for transformers < v5 to resolve interface mismatches.

Testing Done

I ran python -m pytest test/convergence/* on different versions of transformers on the original branch and after making changes. The result is shown below:

Branches	v4.49.0	v4.57.6	v5.0.0
transformer-5.0.0rc1	8 failed, 37 passed, 98 skipped, 1 warning	42 failed, 92 passed, 9 skipped, 3 warnings	19 failed, 115 passed, 9 skipped, 29 warnings
This PR	0 failed, 45 passed, 98 skipped, 1 warning	0 failed, 134 passed, 9 skipped, 19 warnings	19 failed, 115 passed, 9 skipped, 29 warnings

All of the failed tests in v5 are inspected carefully that all of them are identical to the previously thrown error.

• Hardware Type: H100
• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence

[vaibhavjindal]

Amazing work @Mecoli1219! 🚢