Skip to content

P3/feat/stage wise coreset selection v2#556

Draft
pankaj1311 wants to merge 132 commits intostagingfrom
p3/feat/stage-wise-coreset-selection_v2
Draft

P3/feat/stage wise coreset selection v2#556
pankaj1311 wants to merge 132 commits intostagingfrom
p3/feat/stage-wise-coreset-selection_v2

Conversation

@pankaj1311
Copy link
Collaborator

@pankaj1311 pankaj1311 commented Mar 3, 2026

Pull Request Template

Description

Please include a summary of the changes and the related issue. Highlight any key points that reviewers should focus on.

Checklist

  • I have added tests that prove my fix is effective or that my feature works.
  • I have added necessary documentation (if applicable).
  • My code follows the style guidelines, gitflow branching strategy, and naming conventions of this project [Contribution Guidelines](https://github.com/The-School-of-AI/LLM/tree/main/experiments/

Reviewers

  • Reviewer 1: A member from your own team.
  • Reviewer 2: A member from the repo owners team (@The-School-of-AI/llm-repo-owners).

Note: Every pull request requires atleast 2 reviewers/approvers before it can be merged.

aviban15 and others added 30 commits February 4, 2026 15:08
@aviban15
Add readme for coreset selection engine
55577ce
@sidrocks
Add files via upload
390c5c4
@sidrocks
initial commit
f2fabb8
@sidrocks
initial commit
be875cc
@sidrocks
Add files via upload
08f086a
@pankaj1311
feat: adapting to curriculum_v6.yaml
b0f5042
@pankaj1311
feat: added sharding logic and global dedup hash within shard and aft…
924d99f 
…er sharding
@sidrocks
Add coreset engine v5 (stage-wise selection)
fc5d5eb
@BalajiAJ
Initial Data processing/Exact dedup on curriculum data
9951988
@BalajiAJ
Initial Data processing/Exact dedup on curriculum data
7611254
@BalajiAJ
Exact dedup on curriculum data using pyspark
86fb9a9
@BalajiAJ
Exact dedup on curriculum data using pyspark
a936491
@sidrocks
Disable total_token as it will be read from curriculum.yaml stage-wise
2e44956
@sidrocks
Merge branch 'p3/feat/stage-wise-coreset-selection_v2' of github.com:…
05669cd 
…The-School-of-AI/LLM into p3/feat/stage-wise-coreset-selection_v2

:wq

commit.
@sidrocks
Added optional Band-Infer feature to infer a band on band or difficul…
41bde5e 
…t score and fix of ablation report due to multishard process overwrites. Curriculum version updated to 0.7
@sidrocks
remove total-tokens
7492ba1
@abhi1021
Added glue_job_single.py
ec91a2a
@abhi1021
Merge branch 'p3/feat/stage-wise-coreset-selection_v2' of https://git…
907e316 
…hub.com/The-School-of-AI/LLM into p3/feat/stage-wise-coreset-selection_v2
@abhi1021
Update S3 bucket name in Glue job configuration
891825b
@pankaj1311
feat: add the emr scripts for dedup and stats
d2f3e58
@abhi1021
added new allowed_domains
4f90cd4
@pankaj1311
feat: fixed shard.sh
ca3099a
@pankaj1311
feat: fixed batch_processor
5970629
@pankaj1311
feat: fixed batch_processor
e0b2d67
@pankaj1311
feat: fixed batch_processor
0201c26
@pankaj1311
feat: includes changes to folder segregation (Author - Balaji)
6c0c0fc
@pankaj1311
feat: included commands for running on EC2 and notebook for distribut…
f8568f1 
…ion analysis
@pankaj1311
feat: included commands for running on EC2 and fixes to resume operat…
7cc4141 
…ion in shard.sh
@sidrocks
added missing fields in the coreset selection indices
7ea7d33
@sidrocks
Merge branch 'p3/feat/stage-wise-coreset-selection_v2' of
21a4f92 
github.com:The-School-of-AI/LLM into
p3/feat/stage-wise-coreset-selection_v2
@pankaj1311 pankaj1311 self-assigned this Mar 3, 2026
@pankaj1311 pankaj1311 requested a review from BalajiAJ March 3, 2026 16:47
@sidrocks sidrocks force-pushed the p3/feat/stage-wise-coreset-selection_v2 branch from c6f6554 to 3eee423 Compare March 4, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BalajiAJ BalajiAJ Awaiting requested review from BalajiAJ

@firekind firekind Awaiting requested review from firekind firekind will be requested when the pull request is marked ready for review firekind is a code owner

@aiplaybookin aiplaybookin Awaiting requested review from aiplaybookin aiplaybookin will be requested when the pull request is marked ready for review aiplaybookin is a code owner

@NSR9 NSR9 Awaiting requested review from NSR9 NSR9 will be requested when the pull request is marked ready for review NSR9 is a code owner

@sidrocks sidrocks Awaiting requested review from sidrocks sidrocks will be requested when the pull request is marked ready for review sidrocks is a code owner

@smitasasindran smitasasindran Awaiting requested review from smitasasindran smitasasindran will be requested when the pull request is marked ready for review smitasasindran is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

@pankaj1311 pankaj1311

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@pankaj1311 @AnkitaMungalpara @aviban15 @sidrocks @BalajiAJ @abhi1021 @vj1117