Mini CodeLLM

Write-up:

• https://dudeperf3ct.github.io/projects/train_llm_part0/ (data)
• https://dudeperf3ct.github.io/projects/train_llm_part1/ (tokenizer)
• https://dudeperf3ct.github.io/projects/train_llm_part2/ (pretraining)

Getting Started

• codellm_data: Parses and downloads datasets.
• codellm_tokenizer: Train a custom byte-level BPE tokenizer using subset of tokyotech-llm/swallow-code-v2 dataset
• codellm_pretrain: Training a Llama 3.2 model using custom tokenizer and data. Two implementations
  • torch_titan: It uses torchtitan library for training
  • NeMo (WIP)

License

This project is licensed under the MIT License - see the LICENSE file for details.