Case dataset

[wzczc]

Thanks for your great work! I would like to know if the case dataset mentioned in the paper is available？

[ZhangqiJiang07]

Hi, thanks for your interest! I'm sorry, I no longer have this dataset saved. However, it can be generated using the code block under the ''VAR Distribution and Logit Contribution'' section in middle_layers_indicating_hallucination.ipynb.

[ReihanehZohrabi]

Hi, thanks a lot for releasing the code and data, really appreciate the effort, this is a very nice piece of work.

I noticed that the repository provides both the list of 500 samples used for training and the extracted VAR features. However, when loading the released VAR features, the number of real and hallucinated tokens differs quite significantly from what is reported in the paper. In particular, the counts differ by more than 1000 tokens for each class compared to the numbers suggested by the above figure. (Labels: 0 = 497, 1 = 1056)

This made me wonder:

What is the exact data used to train the classifier reported in the paper? Is it trained strictly on the provided 500 samples, or were additional samples / tokens used?

I also observe a noticeable gap when reproducing Table 1 using the released data and settings. For example:

Layers 5–18: reproduced Overall Accuracy = 77.17%, AUC-ROC = 0.8409 (paper: 84.46 / 90.19)

Layers 0–31: reproduced Overall Accuracy = 79.42%, AUC-ROC = 0.8616 (paper: 84.46 / 90.28)

I just wanted to check whether there are any additional preprocessing or data selection details not reflected in the released code. Since I’m reproducing the results for comparison in my own work, I wanted to make sure my setup is correct.

[ZhangqiJiang07]

Hello, thank you for your interest in our work! The dataset used to train the classifier consists of 2,000 samples randomly selected from MS COCO. Unfortunately, the experiments were conducted on rented GPU instances (AutoDL), and the data was not saved before the instances were released. We sincerely apologize, but we are unable to recover that specific dataset at this time. The 500 samples provided in the repo are intended as a demo to verify that the scripts work correctly. Apologies for the inconvenience! Reihaneh Zohrabi ***@***.***> 于2025年12月14日周日 13:59写道：

…

*ReihanehZohrabi* left a comment (ZhangqiJiang07/middle_layers_indicating_hallucinations#6) <#6 (comment)> Hi, thanks a lot for releasing the code and data, really appreciate the effort, this is a very nice piece of work. I noticed that the repository provides both the list of 500 samples used for training and the extracted VAR features. However, when loading the released VAR features, the number of real and hallucinated tokens differs quite significantly from what is reported in the paper. In particular, the counts differ by more than 1000 tokens for each class compared to the numbers suggested by the above figure. (Labels: 0 = 497, 1 = 1056) This made me wonder: What is the exact data used to train the classifier reported in the paper? Is it trained strictly on the provided 500 samples, or were additional samples / tokens used? I also observe a noticeable gap when reproducing Table 1 using the released data and settings. For example: Layers 5–18: reproduced Overall Accuracy = 77.17%, AUC-ROC = 0.8409 (paper: 84.46 / 90.19) Layers 0–31: reproduced Overall Accuracy = 79.42%, AUC-ROC = 0.8616 (paper: 84.46 / 90.28) I just wanted to check whether there are any additional preprocessing or data selection details not reflected in the released code. Since I’m reproducing the results for comparison in my own work, I wanted to make sure my setup is correct. — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2QVVC4WIYMI2OVCKUGZ2CL4BT4E7AVCNFSM6AAAAACJRDBN56VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNJQGI4TKMRYGQ> . You are receiving this because you commented.Message ID: <ZhangqiJiang07/middle_layers_indicating_hallucinations/issues/6/3650295284 @github.com>