[Refactor] Simplify BOS/EOS token handling

[DarkLight1337]

Purpose

• Move Request.eos_token_id -> SamplingParams.eos_token_id, which is only populated if SamplingParams.ignore_eos is False.
• Move get_bos_token_id and get_eos_token_id from InputPreprocessor to Renderer.
• Remove unnecessary failbacks for accessing tokenizer.eos_token_id as it's a required field now.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request refactors the handling of BOS/EOS tokens by moving eos_token_id from Request to SamplingParams and centralizing the logic to get BOS/EOS token IDs into the Renderer. The changes are consistently applied throughout the codebase, including extensive updates to tests. This refactoring simplifies the code and removes unnecessary fallbacks, assuming tokenizer.eos_token_id is a required field. The implementation appears correct and improves code clarity. I have not found any issues with the changes.

[DarkLight1337]

/gemini review

[gemini-code-assist]

Code Review

This pull request is a solid refactoring that simplifies token handling by moving eos_token_id from Request to SamplingParams and relocating BOS/EOS token retrieval methods to the Renderer. The changes are applied consistently across the codebase. However, I've identified a couple of issues related to the ignore_eos handling. The production scheduler logic in check_stop and the MockEngineCore in the test suite no longer respect the ignore_eos flag, which could lead to incorrect behavior and broken tests. I've provided suggestions to restore the previous, correct logic.

[DarkLight1337]

/gemini review

[gemini-code-assist]

Code Review

This pull request is a well-executed refactoring that simplifies the handling of BOS/EOS tokens. Moving eos_token_id from Request to SamplingParams and centralizing token-related methods in the Renderer class are positive changes that improve code clarity and maintainability. The removal of unnecessary fallbacks for tokenizer.eos_token_id is also a good simplification. The changes have been applied consistently across the codebase, including extensive updates to the test suite, which demonstrates thoroughness. I have not found any issues of high or critical severity.