Inquiring about reproducing LatentMAS results with Qwen3-4B (Sequential setting)

[wonjun-chung]

Hi LatentMAS team,

First of all, thank you for sharing your amazing work, LatentMAS. I am deeply impressed by the core ideas of your project and have a keen interest in your research.

I am currently working on reproducing your experiments, but I am facing some difficulties and would like to request your guidance. I have been conducting experiments using the Qwen3-4B model in a sequential setting, but I have not been able to reproduce the reported performance across all benchmarks.

Below are the settings, results, and commands I used for MBPP+ and HumanEval+:

Settings

Model	max_new_tokens	temperature	top_p	prompt	latent_steps	split	generate_bs
Qwen3-4B	4096	0.6	0.95	sequential	20	test	1

Results
 

Method	MBPP+	HumanEval+
Baseline	0.727	0.762
TextMAS	0.761	0.878
LatentMAS	0.613	0.682
LatentMAS (think)	0.571	0.640
LatentMAS (think + realign)	0.558	0.670

Command

# basline
python run.py \
            --method baseline \
            --model_name "Qwen/Qwen3-4B" \
            --task "$task" \
            --max_samples -1 \
            --use_vllm \
            --generate_bs 10
# TextMAS
python run.py \
            --method text_mas \
            --model_name "Qwen/Qwen3-4B" \
            --task "$task" \
            --max_samples -1 \
            --prompt "sequential" \
            --use_vllm \
            --generate_bs 10
# LatentMAS
python run.py \
            --method latent_mas \
            --model_name "Qwen/Qwen3-4B" \
            --task "$task" \
            --max_samples -1 \
            --prompt "sequential" \
            --latent_steps $LATENT_STEPS \
            --generate_bs 1

As shown in the table, while the performance of the Baseline and TextMAS is measured higher than the reported results, the performance of LatentMAS is measured even lower than the Baseline.

Additionally, when I conducted experiments on the latent_steps for MBPP+, I observed a trend where performance decreases as the latent_step increases.

Results

latent_steps	MBPP+
0	0.714
10	0.621
20	0.613

Furthermore, I set generate_bs to 1 for LatentMAS because a significant drop in performance was observed during batch inference. To address this, I attempted left-padding since Qwen is a decoder-only model, but the same issue persisted.

Thank you again for sharing such an interesting piece of work. I would deeply appreciate it if you could look into these discrepancies I’ve observed.

[xiyuanyang45]

Hi @wonjun-chung ,

Thanks for your questions! We will try our best to help you address them one by one.

Regarding the LatentMAS on MBPP+ and HumanEval+, we noticed that you have a different set of hyperparameters from those we provided. To fully reproduce the results of LatentMAS, please kindly follow our commands below:

git pull

# mbpp+ sequential
python run.py --method latent_mas --model_name Qwen/Qwen3-4B --task mbppplus --max_samples -1 --prompt sequential --max_new_tokens 4096 --generate_bs 30 

# humaneval+ sequential
python run.py --method latent_mas --model_name Qwen/Qwen3-4B --task humanevalplus --max_samples -1 --prompt sequential --max_new_tokens 4096 --generate_bs 30 

In our experiments, we perform hyperparameter tuning to identify the most suitable number of latent steps for each model. As a result, the optimal setting may vary across model sizes and tasks. For example, on ARC-C with Qwen3-14B, we observe consistent performance improvements as the number of latent steps increases.

Regarding the question on generate_bs, in batched generation, within the run_batch() function in methods/latent_mas.py (line 82), we enable padding=True for batched inputs of different lengths. The corresponding padding tokens are properly masked and are not involved in self-attention computation. This implementation ensures that batched inference produces identical outputs to single-sample inference, and the generation results should not be affected by the batch size.

Also, in our real experiments, we use a context length of 4096 and set generate_bs to 30 for MBPP+ and HumanEval+. Varying generate_bs might only affect GPU memory usage and does not influence the generated outputs. If you want to share the modified parts of your source code, we would be happy to help identify what might be causing the discrepancy.

[wonjun-chung]

Thank you for your kind guidance. Based on the commands you provided, I was able to achieve LatentMAS performance similar to the reported results.

However, I am still observing that the performance for Baseline and TextMAS is measured higher than what was reported.
Below is the comparison of my experimental settings and results:

	baseline (vLLM)	TextMAS (vLLM)	LatentMAS (HF)
MBPP+	0.706	0.777	0.716
HumanEval+	0.768	0.884	0.792

# Baseline
python run.py --method baseline --model_name Qwen/Qwen3-4B --task mbppplus --max_samples -1 --use_vllm --max_new_tokens 4096
python run.py --method baseline --model_name Qwen/Qwen3-4B --task humanevalplus --max_samples -1 --use_vllm --max_new_tokens 4096

# TextMAS
python run.py --method text_mas --model_name Qwen/Qwen3-4B --task mbppplus --prompt sequential --max_samples -1 --use_vllm --max_new_tokens 4096
python run.py --method text_mas --model_name Qwen/Qwen3-4B --task humanevalplus --prompt sequential --max_samples -1 --use_vllm --max_new_tokens 4096

# LatentMAS
python run.py --method latent_mas --model_name Qwen/Qwen3-4B --task mbppplus --max_samples -1 --prompt sequential --max_new_tokens 4096 --generate_bs 20
python run.py --method latent_mas --model_name Qwen/Qwen3-4B --task humanevalplus --max_samples -1 --prompt sequential --max_new_tokens 4096 --generate_bs 20

I used the codebase as is from a fresh clone without any modifications. Could you please share if there are any differences in the hyperparameters used for the Baseline and TextMAS? Also, I will open a separate issue to address the batch-related concerns.

Thank you for your guidance again.