Add KL/Entorpy Fn #64

pan-x-c · 2025-06-04T09:15:22Z

Description

As the title says

Checklist

Please check the following items before code is ready to be reviewed.

Code has passed all tests
Docstrings have been added/updated in Google Style
Documentation has been updated
Code is ready for review

pan-x-c · 2025-06-04T09:23:45Z

/run-unittest

pan-x-c · 2025-06-04T09:46:32Z

/run-unittest

pan-x-c · 2025-06-04T12:07:06Z

/run-unittest

github-actions · 2025-06-04T12:21:33Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
24	24	0	0	0	0	0	829ms

Failed Tests

No failed tests ✨

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name	Status	Duration
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	✅	4ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer	✅	1ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid	✅	1ms
tests/common/config_test.py::TestConfig::test_load_default_config	✅	5ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion	✅	1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion	✅	1ms
tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate	✅	44ms
tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate	✅	43ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate	✅	53ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate	✅	52ms
tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate	✅	39ms
tests/common/vllm_test.py::TestAPIServer::test_api	✅	22ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask	✅	1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer	✅	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	✅	120ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	✅	123ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	✅	19ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool_with_auxiliary_models	✅	3ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow	✅	1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer	✅	1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	✅	288ms

Github Test Reporter by CTRF 💚

pan-x-c · 2025-06-04T13:10:25Z

/run-unittest

github-actions · 2025-06-04T13:11:46Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
27	13	14	0	0	0	0	46ms

Failed Tests

Failed Tests ❌	Fail Message
❌ tests/common/config_test.py::TestConfig::test_all_examples_are_valid	The test failed in the call phase due to an exception
❌ tests/common/config_test.py::TestConfig::test_load_default_config	The test failed in the call phase
❌ tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate	The test failed in the call phase
❌ tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate	The test failed in the call phase
❌ tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate	The test failed in the call phase
❌ tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate	The test failed in the call phase
❌ tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate	The test failed in the call phase
❌ tests/common/vllm_test.py::TestAPIServer::test_api	The test failed in the call phase
❌ tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	The test failed in the call phase
❌ tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	The test failed in the call phase
❌ tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	The test failed in the call phase
❌ tests/trainer/trainer_test.py::TestTrainerGSM8K::test_trainer	The test failed in the call phase
❌ tests/trainer/trainer_test.py::TestTrainerGSM8KWithSFT::test_trainer	The test failed in the call phase
❌ tests/trainer/trainer_test.py::TestTrainerDPO::test_trainer	The test failed in the call phase

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name	Status	Duration
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	✅	4ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer	✅	1ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid	❌	1ms
tests/common/config_test.py::TestConfig::test_load_default_config	❌	1ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion	✅	1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion	✅	1ms
tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate	❌	1ms
tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate	❌	1ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate	❌	1ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate	❌	1ms
tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate	❌	1ms
tests/common/vllm_test.py::TestAPIServer::test_api	❌	1ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask	✅	1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer	✅	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	❌	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	❌	1ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	✅	19ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool_with_auxiliary_models	✅	3ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow	✅	1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer	✅	1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	❌	1ms
tests/trainer/trainer_test.py::TestTrainerGSM8K::test_trainer	❌	1ms
tests/trainer/trainer_test.py::TestTrainerGSM8KWithSFT::test_trainer	❌	1ms
tests/trainer/trainer_test.py::TestTrainerDPO::test_trainer	❌	1ms

Github Test Reporter by CTRF 💚

pan-x-c · 2025-06-04T13:14:40Z

/run-unittest

pan-x-c · 2025-06-04T14:05:40Z

/run-unittest

github-actions · 2025-06-04T14:25:14Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
27	25	2	0	0	0	0	1.1s

Failed Tests

Failed Tests ❌	Fail Message
❌ tests/trainer/trainer_test.py::TestTrainerGSM8K::test_trainer	The test failed in the call phase due to an exception
❌ tests/trainer/trainer_test.py::TestTrainerGSM8KWithSFT::test_trainer	The test failed in the call phase due to an exception

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name	Status	Duration
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	✅	4ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer	✅	1ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid	✅	1ms
tests/common/config_test.py::TestConfig::test_load_default_config	✅	4ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion	✅	1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion	✅	1ms
tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate	✅	46ms
tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate	✅	43ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate	✅	51ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate	✅	53ms
tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate	✅	38ms
tests/common/vllm_test.py::TestAPIServer::test_api	✅	21ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask	✅	1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer	✅	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	✅	124ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	✅	131ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	✅	19ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool_with_auxiliary_models	✅	3ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow	✅	1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer	✅	1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	✅	410ms
tests/trainer/trainer_test.py::TestTrainerGSM8K::test_trainer	❌	32ms
tests/trainer/trainer_test.py::TestTrainerGSM8KWithSFT::test_trainer	❌	28ms
tests/trainer/trainer_test.py::TestTrainerDPO::test_trainer	✅	115ms

Github Test Reporter by CTRF 💚

pan-x-c · 2025-06-05T02:27:00Z

/run-unittest

github-actions · 2025-06-05T02:52:00Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
27	27	0	0	0	0	0	1.5s

Failed Tests

No failed tests ✨

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name	Status	Duration
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	✅	4ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer	✅	1ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid	✅	1ms
tests/common/config_test.py::TestConfig::test_load_default_config	✅	15ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion	✅	1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion	✅	1ms
tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate	✅	45ms
tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate	✅	43ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate	✅	52ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate	✅	53ms
tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate	✅	38ms
tests/common/vllm_test.py::TestAPIServer::test_api	✅	21ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask	✅	1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer	✅	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	✅	117ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	✅	145ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	✅	19ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool_with_auxiliary_models	✅	3ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow	✅	1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer	✅	1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	✅	454ms
tests/trainer/trainer_test.py::TestTrainerGSM8K::test_trainer	✅	140ms
tests/trainer/trainer_test.py::TestTrainerGSM8KWithSFT::test_trainer	✅	164ms
tests/trainer/trainer_test.py::TestTrainerDPO::test_trainer	✅	135ms

Github Test Reporter by CTRF 💚

pan-x-c · 2025-06-05T03:30:19Z

/run-unittest

github-actions · 2025-06-05T03:53:25Z

Summary

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
27	27	0	0	0	0	0	1.3s

Failed Tests

No failed tests ✨

Flaky Tests

No flaky tests ✨

Skipped

No skipped tests ✨

Tests

Test Name	Status	Duration
tests/buffer/queue_test.py::TestQueueBuffer::test_queue_buffer	✅	4ms
tests/buffer/sql_test.py::TestSQLBuffer::test_create_sql_buffer	✅	1ms
tests/common/config_test.py::TestConfig::test_all_examples_are_valid	✅	1ms
tests/common/config_test.py::TestConfig::test_load_default_config	✅	4ms
tests/common/experience_test.py::TestExperienceConversion::test_batch_conversion	✅	1ms
tests/common/experience_test.py::TestExperienceConversion::test_experience_model_experience_conversion	✅	1ms
tests/common/vllm_test.py::TestModelWrapperSyncV0::test_generate	✅	45ms
tests/common/vllm_test.py::TestModelWrapperAsyncV0::test_generate	✅	43ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV0::test_generate	✅	52ms
tests/common/vllm_test.py::TestModelWrapperAsyncTPV1::test_generate	✅	52ms
tests/common/vllm_test.py::TestModelWrapperAsyncV1::test_generate	✅	39ms
tests/common/vllm_test.py::TestAPIServer::test_api	✅	21ms
tests/common/vllm_test.py::TestTokenizer::test_assistant_token_mask	✅	1ms
tests/explorer/explorer_test.py::BaseExplorerCase::test_explorer	✅	1ms
tests/explorer/explorer_test.py::TestExplorerCountdownEval::test_explorer	✅	123ms
tests/explorer/explorer_test.py::TestExplorerCountdownNoEval::test_explorer	✅	147ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool	✅	19ms
tests/explorer/runner_pool_test.py::RunnerPoolTest::test_runner_pool_with_auxiliary_models	✅	3ms
tests/explorer/workflow_test.py::WorkflowTest::test_gsm8k_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_complex_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_fraction_workflow	✅	1ms
tests/explorer/workflow_test.py::WorkflowTest::test_math_workflow	✅	1ms
tests/trainer/trainer_test.py::BaseTrainerCase::test_trainer	✅	1ms
tests/trainer/trainer_test.py::TestTrainerCountdown::test_trainer	✅	415ms
tests/trainer/trainer_test.py::TestTrainerGSM8K::test_trainer	✅	124ms
tests/trainer/trainer_test.py::TestTrainerGSM8KWithSFT::test_trainer	✅	124ms
tests/trainer/trainer_test.py::TestTrainerDPO::test_trainer	✅	117ms

Github Test Reporter by CTRF 💚

yanxi-chen

lgtm

pan-x-c added 4 commits June 4, 2025 14:11

add kl fn

f1fa9c8

add kl fn

3f61044

fix tests

6fb8cdf

fix dependencies

4c773bb

pan-x-c added 2 commits June 4, 2025 18:14

fix trainer test

526d0aa

fix metrics

d618dca

pan-x-c added 2 commits June 4, 2025 20:56

add entropy_loss

3f58e0e

merge dev

3ed49c8

pan-x-c changed the title ~~[WIP] Add KL/Entorpy Fn~~ Add KL/Entorpy Fn Jun 4, 2025

add entropy loss fn

dff12cb

fix tests

0e29c19

pan-x-c added 4 commits June 5, 2025 10:56

fix entropy_loss_coef

a51480a

simplify name of advantage

fd61f54

update loss

2513b95

update default entropy_coef

135c0d1

update default value

bf1d535

yanxi-chen approved these changes Jun 5, 2025

View reviewed changes

yanxi-chen merged commit 732d801 into modelscope:algorithm_dev Jun 5, 2025
2 checks passed

Add KL/Entorpy Fn #64

Add KL/Entorpy Fn #64

Uh oh!

Conversation

pan-x-c commented Jun 4, 2025

Description

Checklist

Uh oh!

pan-x-c commented Jun 4, 2025

Uh oh!

pan-x-c commented Jun 4, 2025

Uh oh!

pan-x-c commented Jun 4, 2025

Uh oh!

github-actions bot commented Jun 4, 2025

Summary

Failed Tests

Flaky Tests

Skipped

Tests

Uh oh!

pan-x-c commented Jun 4, 2025

Uh oh!

github-actions bot commented Jun 4, 2025

Summary

Failed Tests

Flaky Tests

Skipped

Tests

Uh oh!

pan-x-c commented Jun 4, 2025

Uh oh!

pan-x-c commented Jun 4, 2025

Uh oh!

github-actions bot commented Jun 4, 2025

Summary

Failed Tests

Flaky Tests

Skipped

Tests

Uh oh!

pan-x-c commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Summary

Failed Tests

Flaky Tests

Skipped

Tests

Uh oh!

pan-x-c commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Summary

Failed Tests

Flaky Tests

Skipped

Tests

Uh oh!

yanxi-chen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants