Question about A.2 and B.1

[FahdSeddik]

Hello,

Great work on the paper! I had a couple of questions about the proofs in Sections A.2 and B.1.

Regarding A.2

My understanding is that you assume the final hidden state is effectively equivalent to selecting a row from the output embedding matrix, as stated in “We assume … $h = W_{\text{out}, x}$ ....” I’m having trouble seeing the justification for this assumption and would appreciate clarification on the basis for it.

Regarding B.1

Could you explain why the argument cannot be inverted? Specifically, if we require the latent sequence to losslessly encode the original tokens generated by the model, wouldn’t we need a surjective map from $H^m$ to $V^{m'}$ ? This would imply that the latent reasoning sequence must be longer than the token sequence, essentially giving a counter-argument to the same proof.

[jiaruzouu]

Hi @FahdSeddik ,

Response to Your Question on A.2

Thank you very much for this question! We would like to clarify that this assumption is introduced primarily to facilitate theoretical analysis by formalizing a well-known intuition in neural language models.

Concretely, in standard autoregressive decoding, the next-token probability is computed as $p(x \mid h) = \text{softmax}(h W_{\text{out}}),$ which implies that when a token $x$ is assigned high probability, its corresponding output embedding vector $W_{\text{out},x}$ must have a large inner product with the hidden state $h$. In this sense, decoding a high-probability token naturally corresponds to a strong geometric alignment between $h$ and $W_{\text{out},x}$. Based on this widely accepted intuition, we further formalize it as an assumption to enable a clean theoretical analysis in our paper. Additionally, the conclusions derived under this assumption are also supported by our empirical results, which consistently lead to favorable downstream empirical performance.

Therefore, while the assumption is introduced for analytical tractability, both the modeling intuition and the experimental evidence jointly suggest that it is a reasonable assumption.

Response to Your Question on B.1

Thank you for asking this question! The key point is that inverting the bound in Theorem B.1 does not imply that the latent sequence must be longer than the token sequence.

In Theorem B.1, we show that if a token sequence of length $m'$ can losslessly encode a latent sequence of length $m$, then
$m' \ge \Omega\left(\frac{d_h}{\log |V|} m\right)$, which reflects the fact that encoding high-dimensional continuous representations with discrete tokens is costly.

If this logic (as mentioned in your question) is inverted, we obtain $m \ge \Omega\left(\frac{\log |V|}{d_h} m'\right)$. Since in practice $d_h \gg \log |V|$, the factor $\frac{\log |V|}{d_h}$ is much smaller than 1. As a result, this bound does not imply that $m > m'$.

For example, for Qwen3-4B, this gives approximately $m \ge 0.0043 m'$, which still allows the latent sequence to be much shorter than the token sequence. Therefore, the “inverted” inequality does not form a counter-argument to our proof and is fully consistent with our original claim that latent reasoning is more length-efficient than text-based reasoning.

Thank you again for your careful reading of our paper and theoretical analyses! If you have any additional questions or want to discuss more about our paper, please feel free to leave additional comments!

[FahdSeddik]

Hello @jiaruzouu,

Thank you for the detailed clarification! I appreciate you replying to my questions. If you don't mind, I'd be happy to get an answer for a follow-up question:

Based on your explanation, there seems to be a contradiction in how the hidden state $h$ is treated between sections.

• In A.2: You justify the assumption by arguing that decoding a high-probability token naturally corresponds to a strong geometric alignment between $h$ and $W_{\text{out,x}}$. However, this implies that $h$ is effectively constrained to the "token manifold", meaning that $|H|\approx|V|$ or no. of columns in $W_{\text{out}}$. In the case of Qwen3-4B, this implies that $|H|\approx152k$ (vocab size).
• In B.1: You calculate the capacity of $|H|$ using the full combinatorial space of the basis ($3^{d_h}$). In the case of Qwen3-4B, this implies that $|H|=3^{4096}$ which is exponentially larger than the size implied by A.2.

If the alignment assumption in A.2 holds, shouldn't the effective $|H|$ be much lower (closer to $|V|$ than to $3^{d_h}$)? If we substitute this into your bound, the efficiency factor disappears ($m = m'$).

Lastly, regarding A.2, the empirical results you mentioned are compelling and clearly show an improvement with the projection matrix! However, my questions are mainly aimed at better reconciling this practical success with the theoretical framework.

Thanks again!

[opter2021]

I am also very confused about the proof of A.2.

Can you explain what exactly w_{out,x} and w_{in,x} are, i mean, the mathematical formulation?