Question about Training and Dataset Preparation

[Cerbera-odollam]

Hello, thank you very much for providing this PyTorch-based implementation of EEND.
It's been really helpful, and I’m planning to build on this code to explore some additional ideas.

While setting up the training pipeline, I had a couple of questions regarding the training setup:

In the https://arxiv.org/pdf/2204.00890 paper (titled "From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarization"), was the training conducted using the current code in this repository?

For training on the sim2spk dataset, did you generate the data using the run_prepare_shared.sh script from this link, and use it as part of pretraining?

Lastly, when reporting the DER results in the paper, may I ask which specific evaluation code was used? It would be helpful to know how the DER was computed, especially in the context of reproducing the results. (e.g., nryant/dscore, BUTSpeechFIT/DiaPer, etc.)

I’d greatly appreciate any clarification you can provide.
Thank you again for sharing this great resource.

[fnlandini]

Hi @Cerbera-odollam
Thank you for the kind words.

In the https://arxiv.org/pdf/2204.00890 paper (titled "From Simulated Mixtures to Simulated Conversations as Training Data for End-to-End Neural Diarization"), was the training conducted using the current code in this repository?

Yes.

For training on the sim2spk dataset, did you generate the data using the run_prepare_shared.sh script from this link, and use it as part of pretraining?

Yes, the simulated mixtures were from that repository. The simulated conversations from https://github.com/BUTSpeechFIT/EEND_dataprep

Lastly, when reporting the DER results in the paper, may I ask which specific evaluation code was used? It would be helpful to know how the DER was computed, especially in the context of reproducing the results. (e.g., nryant/dscore, BUTSpeechFIT/DiaPer, etc.)

The results for the paper were obtained with nryant/dscore

[Cerbera-odollam]

Thank you very much for your response.
I truly appreciate your time and support.

To further ensure proper reproduction of the experiments, I’d like to ask a few follow-up questions:

• When generating the simulated mixtures, could you let me know which random seed was used? In the code, it seems to be set to 777 by default — did you use this default setting?

• In this PyTorch version of EEND, I noticed that the training seed is set to 3. Would it be appropriate to use this value for reproducing the results?

• Lastly, if possible, could you share what DER (without collar) you obtained on the simulated dataset? This would be very helpful as a reference point.

Thank you again for your assistance and for making this work publicly available.

[fnlandini]

A colleague of mine generated the simulated mixtures a few years ago. I assume she used the default seed.

You can use that seed but other things still affect the results so you will not be able to obtain the exact same results. Something in the ballpark should be fine. In the papers I ran some experiments a few times to create error bars. That can give you an idea of how much difference to expect.

I am afraid I do not have those results anymore but I remember it was around 3 DER when evaluated with collar of 0.25s

[Cerbera-odollam]

Thank you again for your kind response.

If I have any further questions after running additional experiments, would it be alright to reach out again?

[Cerbera-odollam]

Hello @fnlandini ,

I have been conducting some experiments using the current code, and I had a few follow-up questions.

When training on the sim2spk dataset alone, I measured the DER on the sim2spk validation set using the dscore code with a 0.25s collar, and obtained a DER of 5.01. However, you mentioned achieving a DER of approximately 3, so I wanted to confirm whether this result was based on training not just on sim2spk, but also on the simNspk dataset.

Thank you in advance for your time and guidance.

[fnlandini]

Hi @Cerbera-odollam
It was simulated conversations with 2 speakers only. The results I had on the test set were: 2.8, 3.13, 3.27, 2.83, 2.99
These were 5 different training runs with different seeds on the same dataset.

[Cerbera-odollam]

@fnlandini
Hello again, and thank you for your patience.

Apologies for the delay in following up — I wanted to carefully recheck my setup and rerun the experiments from scratch to ensure everything was correct.
After repeating the training without any changes to the settings, I’m still observing a DER of around 5 on the sim2spk validation set.

Given this, I was wondering if there might be other factors that could lead to a lower DER, such as a different threshold (e.g., not using the default 0.5) or any other settings that may affect the evaluation results.

I’d greatly appreciate any insight you could share.

[fnlandini]

Hi @Cerbera-odollam
Sorry for the delay. Threshold 0.5 is fine but I realize now that I think you are using simulated mixtures, not simulated conversations as we proposed. The results I shared were on simulated conversations. I do not have results for simulated mixtures.

[Cerbera-odollam]

@fnlandini
Thank you for your support. It’s been a great help in setting up and running my experiments.

I believe this might be one of my last questions. I noticed that the seed is fixed at 3 in the current training setup. Based on the context, I assumed that this seed was chosen because it yielded good performance on the simulated conversation dataset.

I was wondering, when conducting experiments on the simulated mixtures, did you also use a fixed random seed, or did you perform multiple runs with random seeds?

[fnlandini]

Hi,
The choice of 3 was not based on any results but simply one number. We did run the experiments with different seeds to know how much the same training could vary. You can find that in Figure 1