Reproducibility Question

[TravisNP]

Hello. I downloaded your repo and am trying to recreate the results in Table 4. I downloaded the Bibtex dataset and when I run your code, I cannot get the nDCG or the PSnDCG numbers that are in Table 4 (I haven't tried the rest yet). I consistently get nDCG around 59.8 and PScDCG of 45.5 - 47, which are not the 61.7 and 48.6 reported in the paper.

I was curious if you set a wrong hyperparameter when uploading the code to github, or perhaps the dataset has been updating or something? I pulled the Bibtex dataset from the website in the readme.

Thank you for taking the time to help me reproduce your results!

[EdwardRaff]

@MahmudulAlam can you help on the code side?

[MahmudulAlam]

Hi, @TravisNP
I have re-run the code and reproduced the following results:
BIBTEX (nDCG & PSnDCG) = (61.258, 47.935)
DELICIOUS = (68.571, 33.711)
MEDIAMILL = (88.112, 67.665)
EURLEX-4K = (77.016, 31.192)
Have you tried other datasets yet? Not sure whether the dataset has been updated or not. I will try to look into them for any disparity.

[TravisNP]

Hi @MahmudulAlam
I have run the code for those four datasets, and am not getting those results. When running the code exactly as is, and also when running multiple times and taking average and std, I am not getting the clear dominance of HLB that the Table shows.

Yes, if you could let me know if the datasets have been updated or could find out what's going on, that would be much appreciated! Are you still using an old environment? If you redo your environment, do the results change? Maybe a package has been updated that is affecting something?

[EdwardRaff]

Travis what GPU are you using too. I have seen occasional GPU + framework result in surprisingly large dispenses Sent from my iPhoneOn Jan 20, 2026, at 11:52 PM, Travis Pence ***@***.***> wrote:TravisNP left a comment (FutureComputing4AI/Hadamard-derived-Linear-Binding#1) Hi @MahmudulAlam I have run the code for those four datasets, and am not getting those results. When running the code exactly as is, and also when running multiple times and taking average and std, I am not getting the clear dominance of HLB that the Table shows. Yes, if you could let me know if the datasets have been updated or could find out what's going on, that would be much appreciated! Are you still using an old environment? If you redo your environment, do the results change? Maybe a package has been updated that is affecting something? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[MahmudulAlam]

Hello,

I have just rerun the experiments and obtained similar results to those in Table 4. Here are the mean and std scores in those datasets.

	BIBTEX	DELICIOUS	MEDIAMILL	EURLEX-4K
nDCG	61.86 ± 0.47	65.90 ± 0.33	88.19 ± 0.38	77.55
PSnDCG	48.56 ± 0.39	32.40 ± 0.20	67.76 ± 0.41	30.92

Since all the labels are represented by some random vectors, randomness in the results is expected; however, the results should be close. I have not had the time to look into the dataset yet or compare the codebases. I am still using the original environment used in the development of the code. I will be busy until the end of this month. I will try to do a review and look into this issue.

[TravisNP]

@EdwardRaff I am using an A100 - 40GB. What GPU were these experiments run on?
@MahmudulAlam I tried multiple 5 trial runs and had small std each time, and HLB is first usually once (sometimes zero, sometimes twice) for each set of trials. If you could paste your exact environment, I could install the exact versions of those packages. If you have time before the ICML deadline, that would be much appreciated.

[MahmudulAlam]

@TravisNP
I have re-uploaded my local xml code directory to this github repo. Please download the code again and let me know if you still face the same issues. I included a fixed seed value to reproduce exact same results, so you should get the exact results listed below. So far, I have run these four datasets:

BIBTEX		DELICIOUS		MEDIAMILL		EURLEX-4K
nDCG	PSnDCG	nDCG	PSnDCG	nDCG	PSnDCG	nDCG	PSnDCG
61.799	48.154	67.503	31.698	87.972	67.438	77.191	30.049