Tokenization of the data

[gzagarwal]

Pull Request Template

Description

Please include a summary of the changes and the related issue. Highlight any key points that reviewers should focus on.

Checklist

• I have added tests that prove my fix is effective or that my feature works.
• I have added necessary documentation (if applicable).
• My code follows the style guidelines, gitflow branching strategy, and naming conventions of this project [Contribution Guidelines](https://github.com/The-School-of-AI/LLM/tree/main/experiments/

Reviewers

• Reviewer 1: A member from your own team.
• Reviewer 2: A member from the repo owners team (@The-School-of-AI/llm-repo-owners).

Note: Every pull request requires atleast 2 reviewers/approvers before it can be merged.

[pankaj1311]

Closing it as confirmed by Asang, P06 has another PR #461