Questions about test results

[Stefanie927]

When I use CausCell to train the MERFISH_Brain dataset , I found that the concept celltype classification results were very poor. Are there any tricks for training? Or could you provide relevant hyperparameters such as learning rate, optimizer, etc.

[GaoYiChengTJ]

Hi, thanks for your interest. Can you show some of your training codes?

[Stefanie927]

Thanks for your reply!
I trained the model using the tutorial's real_demo.ipynb file and set training_num_steps to 100,000.
Then I tested the classification results using the concept_prediction function in CausCell.py.
The training code is as follows:

model = CausCell(save_and_sample_every=10000)

concept_list = ['Age','Domain','Celltype']
concept_counts = [3, 8, 10]
concept_cdag = [[0,0,0,0],[0,0,0,0],[1,1,0,0],[0,0,0,0]]

results_folder = "./Output/Merfish_Brain_model"

transformed_train_data = model.data_transformation(data_pwd="./Data/Merfish_Brain_training_data.h5ad", 
                                                   save_pwd="./Data", 
                                                   concept_list=concept_list,
                                                   log_norm=True)

model.train(training_data_pwd="./Data/transformed_Merfish_Brain_training_data.h5ad", 
            model_save_pwd=results_folder, 
            concept_list=concept_list, concept_counts=concept_counts, concept_cdag=concept_cdag, 
            training_num_steps=100000, 
            train_log=True)

After I finished training, I saved a total of 10 models and tested them.
The testing code is as follows:

transformed_test_data = model.data_transformation(data_pwd="./Data/Merfish_Brain_testing_data.h5ad", 
                                                save_pwd="./Data", 
                                                concept_list=concept_list)

testing_data_pwd = "./Data/transformed_Merfish_Brain_testing_data.h5ad"

for id in range(1,11):
    model.load_trained(concept_list=concept_list, concept_counts=concept_counts, concept_cdag=concept_cdag, 
                    results_folder=results_folder, 
                    trained_profile_size=374, 
                    milestone=id)

    concept_embs = model.disentanglement(testing_data_pwd=testing_data_pwd, 
                                        saved_pwd="./Output", 
                                        concept_list=concept_list, concept_counts=concept_counts, concept_cdag=concept_cdag)

    model.concept_prediction(testing_data_pwd, concept_embs, concept_list, concept_counts, concept_cdag)

[Stefanie927]

Thanks for your reply! I set save_and_sample_every=10000 and training_num_steps=100000. From line 819 of Modules.py, I think that the saved model-1.pt, model-2.pt, ..., model-10.pt actually correspond to epoch-10000, epoch-20000, ..., epoch-100000.

…

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: ***@***.***&gt;; 发送时间: 2025年10月28日(星期二) 下午4:33 收件人: ***@***.***&gt;; 抄送: ***@***.***&gt;; ***@***.***&gt;; 主题: Re: [bm2-lab/CausCell] Questions about test results (Issue #3) GaoYiChengTJ留下评论（bm2-实验室/CausCell#3） 看来您正在使用训练里程碑的前 10 个步骤进行测试。也许您可以更改此模型的里程碑数量。 for id in range(1,11): model.load_trained(concept_list=concept_list, concept_counts=concept_counts, concept_cdag=concept_cdag, results_folder=results_folder, trained_profile_size=374, milestone=id — 直接回复此邮件、在 GitHub 上查看或取消订阅。 您收到此邮件是因为您是该主题的作者。消息 ID：<bm2-lab/CausCell/issues/3/3455179748 @ github . com&gt;

[GaoYiChengTJ]

The dataset you used actually has already been preprocessed. The log_norm is not needed anymore. I think this point in tutorial has some misunderstanding. Sorry for that!

[Stefanie927]

Thank you again for your patient explanation!
I modified the log_norm settings, and the code is as follows:

model = CausCell(save_and_sample_every=10000)

concept_list = ['Age','Domain','Celltype']
concept_counts = [3, 8, 10]
concept_cdag = [[0,0,0,0],[0,0,0,0],[1,1,0,0],[0,0,0,0]]

results_folder = "./Output/Merfish_Brain_model"

transformed_train_data = model.data_transformation(data_pwd="./Data/Merfish_Brain_training_data.h5ad", 
                                                   save_pwd="./Data", 
                                                   concept_list=concept_list,
                                                   log_norm=False)

# training
model.train(training_data_pwd="./Data/transformed_Merfish_Brain_training_data.h5ad", 
            model_save_pwd=results_folder, 
            concept_list=concept_list, concept_counts=concept_counts, concept_cdag=concept_cdag, 
            training_num_steps=100000, 
            train_log=True)



transformed_test_data = model.data_transformation(data_pwd="./Data/Merfish_Brain_testing_data.h5ad", 
                                                save_pwd="./Data", 
                                                concept_list=concept_list,
                                                log_norm=False)

testing_data_pwd = "./Data/transformed_Merfish_Brain_testing_data.h5ad"

# testing
for id in range(1,11):
    model.load_trained(concept_list=concept_list, concept_counts=concept_counts, concept_cdag=concept_cdag, 
                    results_folder=results_folder, 
                    trained_profile_size=374, 
                    milestone=id)

    concept_embs = model.disentanglement(testing_data_pwd=testing_data_pwd, 
                                        saved_pwd="./Output", 
                                        concept_list=concept_list, concept_counts=concept_counts, concept_cdag=concept_cdag)

    model.concept_prediction(testing_data_pwd, concept_embs, concept_list, concept_counts, concept_cdag)

The classification results are output as follows:

Factor_Age: ACC:0.6129699622040978      Precision:0.5898873441673342    Recall:0.6129699622040978       F1:0.5844543754162642
Factor_Domain: ACC:0.4724885617664611   Precision:0.4294108704261934    Recall:0.4724885617664611       F1:0.44788338883061013
Factor_Celltype: ACC:0.3092102645713149 Precision:0.30584319878414185   Recall:0.3092102645713149       F1:0.30722267699419814

The results for cell_type are different from those in the paper.
I would like to ask what factors might be causing this. Could it be the learning rate, loss weights, etc.?

[zengyuanpeng2022-hub]

I think the tutorial need be update...

[GaoYiChengTJ]

Thanks for interests! Sure, we will update the tutorial.

[zengyuanpeng2022-hub]

Thank you again for your patient explanation! I modified the log_norm settings, and the code is as follows:

model = CausCell(save_and_sample_every=10000)

concept_list = ['Age','Domain','Celltype']
concept_counts = [3, 8, 10]
concept_cdag = [[0,0,0,0],[0,0,0,0],[1,1,0,0],[0,0,0,0]]

results_folder = "./Output/Merfish_Brain_model"

transformed_train_data = model.data_transformation(data_pwd="./Data/Merfish_Brain_training_data.h5ad",
save_pwd="./Data",
concept_list=concept_list,
log_norm=False)

training

model.train(training_data_pwd="./Data/transformed_Merfish_Brain_training_data.h5ad",
model_save_pwd=results_folder,
concept_list=concept_list, concept_counts=concept_counts, concept_cdag=concept_cdag,
training_num_steps=100000,
train_log=True)

transformed_test_data = model.data_transformation(data_pwd="./Data/Merfish_Brain_testing_data.h5ad",
save_pwd="./Data",
concept_list=concept_list,
log_norm=False)

testing_data_pwd = "./Data/transformed_Merfish_Brain_testing_data.h5ad"

testing

for id in range(1,11):
model.load_trained(concept_list=concept_list, concept_counts=concept_counts, concept_cdag=concept_cdag,
results_folder=results_folder,
trained_profile_size=374,
milestone=id)

concept_embs = model.disentanglement(testing_data_pwd=testing_data_pwd, 
                                    saved_pwd="./Output", 
                                    concept_list=concept_list, concept_counts=concept_counts, concept_cdag=concept_cdag)

model.concept_prediction(testing_data_pwd, concept_embs, concept_list, concept_counts, concept_cdag)

The classification results are output as follows:

Factor_Age: ACC:0.6129699622040978 Precision:0.5898873441673342 Recall:0.6129699622040978 F1:0.5844543754162642
Factor_Domain: ACC:0.4724885617664611 Precision:0.4294108704261934 Recall:0.4724885617664611 F1:0.44788338883061013
Factor_Celltype: ACC:0.3092102645713149 Precision:0.30584319878414185 Recall:0.3092102645713149 F1:0.30722267699419814
The results for cell_type are different from those in the paper. I would like to ask what factors might be causing this. Could it be the learning rate, loss weights, etc.?
应该是做data_transformation数据标签没对应上，参考一下其他issue。