Improve `.llm` code coverage #10516

xnuohz · 2025-10-30T14:23:12Z

Issue

Close #10514, #10529

CodeCov

Before

After

codecov · 2025-10-31T04:49:26Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.53%. Comparing base (c211214) to head (b143673).
⚠️ Report is 138 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10516      +/-   ##
==========================================
+ Coverage   86.11%   87.53%   +1.41%     
==========================================
  Files         496      510      +14     
  Lines       33655    35960    +2305     
==========================================
+ Hits        28981    31476    +2495     
+ Misses       4674     4484     -190

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

xnuohz · 2025-11-08T11:35:02Z

@puririshi98 @akihironitta .llm test coverage has been successfully uploaded to codecov. ready for review and merge.

puririshi98

lgtm, just plz address my one comment

puririshi98 · 2025-11-10T19:49:38Z

torch_geometric/llm/models/g_retriever.py

            batch_unique = batch.unique()
            batch_size = len(question)
-            if len(batch_unique) < batch_size:
+            if len(batch_unique) <= batch_size:


why less than equal?

to test coverage when they are equal, forgot to remove, will update.

puririshi98

1 more

puririshi98 · 2025-11-10T19:50:44Z

torch_geometric/datasets/molecule_gpt_dataset.py

+            model_name='Qwen/Qwen3-0.6B',
            num_params=1,
            dtype=torch.bfloat16,
+            sys_prompt='You are an agent, answer my questions.',


can you actually run the molGPT example w this change and share a full log of it? just want to make sure everything is still smooth

root@7df2f109d384:/workspace/pytorch_geometric# python examples/llm/molecule_gpt.py Setting up 'Qwen/Qwen3-0.6B' with configuration: {'revision': 'main', 'max_memory': {0: '23GiB'}, 'low_cpu_mem_usage': True, 'device_map': 'auto', 'torch_dtype': torch.bfloat16} Some weights of RobertaModel were not initialized from the model checkpoint at DeepChem/ChemBERTa-77M-MTR and are newly initialized: ['pooler.dense.bias', 'pooler.dense.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Total Preparation Time: 9.311407s Training beginning... Epoch: 1|3: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 4.71it/s] Epoch: 1|3, Train loss: 2.322746, Val loss: 2.416511 Epoch: 2|3: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 8.08it/s] Epoch: 2|3, Train loss: 1.376676, Val loss: 2.370785 Epoch: 3|3: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 7.97it/s] Epoch: 3|3, Train loss: 1.102427, Val loss: 2.344596 Total Training Time: 4.702314s Test loss: 0.022353 Total Time: 14.047626s

can you run master brnach as well for comparison?

root@7df2f109d384:/workspace/pytorch_geometric# python examples/llm/molecule_gpt.py Setting up 'TinyLlama/TinyLlama-1.1B-Chat-v0.1' with configuration: {'revision': 'main', 'max_memory': {0: '23GiB'}, 'low_cpu_mem_usage': True, 'device_map': 'auto', 'torch_dtype': torch.bfloat16} Some weights of RobertaModel were not initialized from the model checkpoint at DeepChem/ChemBERTa-77M-MTR and are newly initialized: ['pooler.dense.bias', 'pooler.dense.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Total Preparation Time: 8.718750s Training beginning... Epoch: 1|3: 0%| | 0/4 [00:00<?, ?it/s]/workspace/pytorch_geometric/torch_geometric/llm/models/molecule_gpt.py:158: UserWarning: HuggingFace model TinyLlama/TinyLlama-1.1B-Chat-v0.1 is not using a chat template, using Llama 2 style prompting. Please consider using a more recent model and initialize the LLM with `sys_prompt`. ) = self.llm._get_embeds(instructions, additional_text_context, xs, Epoch: 1|3: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00, 3.85it/s] Epoch: 1|3, Train loss: 1.763808, Val loss: 2.043718 Epoch: 2|3: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 6.02it/s] Epoch: 2|3, Train loss: 1.431526, Val loss: 1.987978 Epoch: 3|3: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 6.16it/s] Epoch: 3|3, Train loss: 1.353049, Val loss: 1.963581 Total Training Time: 6.796006s Test loss: 0.242709 Total Time: 15.544912s

i notice the test los is 10x smaller now while the val and training loss havent changed much between branches. could you rationalize why this is the case? it indicates to me there may be a bug at test time in the new branch unless you can explain why this might be to me

xnuohz · 2025-11-18T12:29:15Z

@puririshi98 I forgot to force reload MoleculeDataset when llm switched to Qwen, so the text in the dataset was generated by TinyLlama but trained it with Qwen.
See the training log from scratch below.

master branch from scratch

Setting up 'TinyLlama/TinyLlama-1.1B-Chat-v0.1' with configuration: {'revision': 'main', 'max_memory': {0: '21GiB'}, 'low_cpu_mem_usage': True, 'device_map': 'auto', 'torch_dtype': torch.bfloat16}
Some weights of RobertaModel were not initialized from the model checkpoint at DeepChem/ChemBERTa-77M-MTR and are newly initialized: ['pooler.dense.bias', 'pooler.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Total Preparation Time: 3359.429658s
Training beginning...
Epoch: 1|3:   0%|                                                                                      | 0/1682 [00:00<?, ?it/s]/workspace/pytorch_geometric/torch_geometric/llm/models/molecule_gpt.py:158: UserWarning: HuggingFace model TinyLlama/TinyLlama-1.1B-Chat-v0.1 is not using a chat template, using Llama 2 style prompting. Please consider using a more recent model and initialize the LLM with `sys_prompt`.
  ) = self.llm._get_embeds(instructions, additional_text_context, xs,
Epoch: 1|3: 100%|███████████████████████████████████████████████████████████████████████████| 1682/1682 [04:12<00:00,  6.67it/s]
Epoch: 1|3, Train loss: 1.020544, Val loss: 0.994869
Epoch: 2|3: 100%|███████████████████████████████████████████████████████████████████████████| 1682/1682 [04:11<00:00,  6.69it/s]
Epoch: 2|3, Train loss: 0.816425, Val loss: 0.960044
Epoch: 3|3: 100%|███████████████████████████████████████████████████████████████████████████| 1682/1682 [04:10<00:00,  6.70it/s]
Epoch: 3|3, Train loss: 0.795707, Val loss: 0.943275
Total Training Time: 778.760076s
Test loss: 0.957731
Total Time: 4144.875072s

this pr from scratch

Setting up 'Qwen/Qwen3-0.6B' with configuration: {'revision': 'main', 'max_memory': {0: '22GiB'}, 'low_cpu_mem_usage': True, 'device_map': 'auto', 'torch_dtype': torch.bfloat16}
Some weights of RobertaModel were not initialized from the model checkpoint at DeepChem/ChemBERTa-77M-MTR and are newly initialized: ['pooler.dense.bias', 'pooler.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Total Preparation Time: 5493.621828s
Training beginning...
Epoch: 1|3: 100%|███████████████████████████████████████████████████████████████████████████| 1682/1682 [03:21<00:00,  8.37it/s]
Epoch: 1|3, Train loss: 0.581263, Val loss: 0.575342
Epoch: 2|3: 100%|███████████████████████████████████████████████████████████████████████████| 1682/1682 [03:20<00:00,  8.38it/s]
Epoch: 2|3, Train loss: 0.435040, Val loss: 0.553491
Epoch: 3|3: 100%|███████████████████████████████████████████████████████████████████████████| 1682/1682 [03:20<00:00,  8.38it/s]
Epoch: 3|3, Train loss: 0.405096, Val loss: 0.549858
Total Training Time: 622.549687s
Test loss: 0.585572
Total Time: 6122.272548s

puririshi98

okay since the loss has dropped 2x on the test branch i think this is safe to merge just need CI to be green

xnuohz · 2025-11-26T04:49:57Z

@puririshi98 ready to merge. the same nightly PyTorch CI error as the master branch.

xnuohz added 2 commits October 30, 2025 22:19

update

b9e664b

update

4cfa0a3

xnuohz requested review from akihironitta, puririshi98, rusty1s and wsad1 as code owners October 30, 2025 14:23

xnuohz mentioned this pull request Oct 30, 2025

Improving torch_geometric.llm Code Coverage #10514

Open

xnuohz changed the title ~~Improve .llm code coverage~~ [Code Coverage] llm/models/sentence_transformer.py and llm/models/vision_transformer.py Oct 31, 2025

add changelog

3080221

xnuohz added 7 commits November 1, 2025 17:35

Merge branch 'master' into cov/llm/vit

fef36f4

update

ec9c462

update

beafdc4

update

eca7f10

update

8881879

update

57c9b55

update

7c9efbe

xnuohz changed the title ~~[Code Coverage] llm/models/sentence_transformer.py and llm/models/vision_transformer.py~~ Improve .llm code coverage Nov 1, 2025

xnuohz added 6 commits November 3, 2025 22:30

improve models/llm.py

2c97909

Merge branch 'master' into cov/llm/vit

a1a39d7

improve models/llm_judge.py

93d0693

update changelog

94ae196

improve models/molecule_gpt.py

3bb82b2

improve models/glem.py

7dd529a

akihironitta assigned puririshi98 Nov 3, 2025

xnuohz added 5 commits November 4, 2025 01:04

update

5680e00

improve models/txt2kg.py

17c2210

update

2a6ab6f

update

117fe75

improve utils/backend_utils.py

9be3f11

xnuohz added 12 commits November 8, 2025 00:34

add flag when upload coverage

c121caa

add flag in coverage not useful

64c960c

fix molecule_gpt_dataset test

c0cef4a

Merge branch 'master' into cov/llm/vit

3cb3efc

remove onlyrag in llm/utils

52b9d87

ignore llm test in minimal/nightly/prev test

554bc38

remove onlyrag in llm/models

d5caa53

update timeout minutes

99aff66

remove onlyrag in txt2kg

4e5ac17

set onlyLinux in txt2kg test

c978391

cleanup

8ae6f28

fuse rag and latest test

58d0484

puririshi98 requested changes Nov 10, 2025

View reviewed changes

xnuohz added 10 commits November 11, 2025 21:47

update

1f9a56e

update

f382f38

remove onlyRAG

a1809fc

remove onlyRAG

33e9ebb

update

17cd41c

fix ci

42ffa92

fix ci

44b780b

fix ci

7b31783

update

00ca6b9

Merge branch 'master' into cov/llm/vit

ac0796c

Merge branch 'master' into cov/llm/vit

da82264

puririshi98 approved these changes Nov 25, 2025

View reviewed changes

trigger ci

b143673

Improve .llm code coverage #10516

Are you sure you want to change the base?

Improve .llm code coverage #10516

Uh oh!

Conversation

xnuohz commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue

CodeCov

Before

After

Uh oh!

codecov bot commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

xnuohz commented Nov 8, 2025

Uh oh!

puririshi98 left a comment

Choose a reason for hiding this comment

Uh oh!

puririshi98 Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

xnuohz Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

puririshi98 left a comment

Choose a reason for hiding this comment

Uh oh!

puririshi98 Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

xnuohz Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

puririshi98 Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

xnuohz Nov 16, 2025

Choose a reason for hiding this comment

Uh oh!

puririshi98 Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

xnuohz commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

master branch from scratch

this pr from scratch

Uh oh!

puririshi98 left a comment

Choose a reason for hiding this comment

Uh oh!

xnuohz commented Nov 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Improve `.llm` code coverage #10516

Improve `.llm` code coverage #10516

xnuohz commented Oct 30, 2025 •

edited

Loading

codecov bot commented Oct 31, 2025 •

edited

Loading

xnuohz commented Nov 18, 2025 •

edited

Loading