[Deprecation] Remove old locations of get_tokenizer and resolve_hf_chat_template

[DarkLight1337]

Purpose

• We can remove these deprecations now that lm-eval has been updated in Bump lm-eval version for Transformers v5 compatibility #33994.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[dosubot]

Related Documentation

Checked 0 published document(s) in 1 knowledge base(s). No updates required.

^{How did I do? Any feedback?}  

[gemini-code-assist]

Code Review

This pull request removes deprecated code related to get_tokenizer and resolve_hf_chat_template. The changes are straightforward and correct, consisting of the removal of the vllm/transformers_utils/tokenizer.py file and the __getattr__ function in vllm/entrypoints/chat_utils.py. This cleanup is well-justified by the update to the lm-eval dependency as mentioned in the description. The changes improve code maintainability by removing obsolete code paths. I approve these changes.

[DarkLight1337]

Ok turns out lm-eval hasn't been updated with the new import yet. I have opened a PR on their end: EleutherAI/lm-evaluation-harness#3595